The history of Bitchu Matsuyama dates back from 1240, during the Kamakura period, when the first Matsuyama castle was built on a different location nearby by Akiba Shigenobu. In 1331 Takahashi Muneyasu moved the castle to its present site, but had a completely different structure. The castle in its present shape was built in 1683 by Mizunoya Katsumune and remained intact until the Meiji Restoration, when several of its structures were demolished. The main tower, a watching turret and part of the wall were saved and can be visited today.
Unlike to other castles in Japan, the Tenshu, or main tower, of Bitchu Matsuyama is only two stories high, since its location on top of a mountain already allowed a large field of vision and nowadays offers impressive panoramic views of Takahashi. Since the castle primarily served as a defensive fortress and, its walls are high and steep and can only be accessed on foot by a hiking trail.