Although the date 1474 appears roughly inscribed on its northern façade, the tower is thought to have been built between 1440 and 1450. During this period the island was under the rule of the Peraza who ordered the tower to be built, more through fear of "undomesticated" Gomerans than concern for outside attacks. Taking its cue from the gothic style that then was in decline in Europe, an 18-metre high square tower was raised and was finished off by large stones placed on each of its sides, with round arches supporting its three stories. Walls almost a metre and a half thick and wooden floors completed the structure but, barely fifty years after its construction, it proved to be useless for the general purposes of the artillery.      

The Torre del Conde (the Count's Tower) has an almost identical twin due to its stylistic similarity to the Torre del Merino in Santillana del Mar in Cantabria. It is a style, however, that was already outdated in terms of logistical use due to the changes taking place in warfare and the reign of the artillery. Even so, the tower in La Gomera proved to be a useful defence against the stones and sword blows cast by the island aborigines during their revolts against the arbitrary acts inflicted by feudal lord of the island.    

However, despite the defensive shortcomings of the Peraza fortress, since being built in the 16th century, the Torre del Conde has survived the passing of the centuries. The invasion launched by the Huguenot Jaques de Capdeville in 1571 failed to bring it down, as did the pirates attacks of 1599 and 1618, although it came close to crumbling on more than one occasion. This detail was not lost on the king who also realised the importance of maintaining possession of the bay of San Sebastián. Felipe II, obsessed with securing the trade of precious metals from the Americas, was committed to fortifying the ports. In order to do so, famous military engineers were summoned to La Gomera in order to improve its defences. Firstly Fratín, and later Torriani, were of the opinion that the building was of little use. Fratín, known for his meticulous attention to detail, believed the tower "could be used as a prison". The engineer to Felipe II bluntly stated that: "At the base of this tower, on the outer part, there are four artillery pieces intended to defend the port, but they are so badly positioned that they are far more likely to offend friendly vessels than deter enemies", and suggested locking the tower up in a diamond-cornered fort. Torriani, perhaps being more pragmatic and aware of the course that the war had taken, opted for concentrating their efforts on the small overhang that seals the northwest side of the bay. Years later, the small fort of Buen Paso was built there to guard the entrance to the Gomeran bay. 

A new attack, this time in May 1744, once again left the tower in a ruinous state. The British commodore, Charles Windham, launched a heavy attack on San Sebastián. The skirmish, which resulted in three English ships being expelled, failed to bring the tower to the ground, but caused considerable material damage. However, it is still standing and has survived the passing of the years and the additions and mutilations of centuries of use and disuse. Its unmistakable outline, white walls and red ashlars can still be seen to this day thanks to the urgent repairs that have been made over the centuries.

In 1960 it was acquired by the Council of La Gomera and, despite centuries of neglect, efforts were made to restore it. Now, the tower is open to the public who, for a moment in time, can imagine themselves as authentic feudal lords.