The fortress was originally built in the tenth century. Local legend holds that Vysehrad was the location of the first settlement which would later become Prague, though thus far the claim is unsubstantiated. Ancient legends situate the original seat of the Czech princes - the legendary Princess Libuše and the first Przemyslides - on this hill. In fact, however, this fort had not been founded until Prague Castle was already in existence, since it dates from about the mid-10th century.

When the Premyslid dynasty settled on the current site of Prague Castle, the two castles maintained opposing spheres of influence for approximately two centuries. When Holy Roman Emperor Charles IV began to build the Prague Castle in its current dimensions (in the early 14th century), Vysehrad was abandoned as a royal home. At the beginning of the Hussite Wars, it was captured by the Hussites.

It underwent a renovation in the 17th century, when the Habsburg Monarchy took over the Czech lands after the Thirty Years' War and became a training centre for the Austrian Army. It was also incorporated into the Baroque era city walls around Prague.