The first castle on the site was built around 1106 on land granted to Henry de Beaumont by Henry I. Henry's grant created the lordship of Gower, which took in not only the Gower peninsula but an area of equal size stretching north from Swansea into the interior.
That early fortification was a simple affair of earthworks and timber palisades. It was sacked by the Welsh several times, first in 1116, again in 1192, and once more in 1217. The castle was completely razed in that latter attack, and when William de Braose came to rebuild it he did so in stone.
The 'New Castle' was built in stages between 1221 and 1284, just in time to be sacked by Rhys ap Maredudd in 1287. Swansea Castle was re-garrisoned during the Glyndwr revolt, but never saw action, and unlike many Welsh castles, it played no part in the Civil War.
It fell in gradual disuse as a military fortress, and the castle buildings were used for such diverse purposes as a poorhouse, a residence, a copper processing works, glass works, a market place, town hall, chapel, newspaper offices, and militia quarters. Control eventually passed to the city, and the site is now in the care of Cadw, Welsh Heritage.
The remains of the castle are built of sandstone blocks, in an L shape with a residential block to the south and a tower to the north, joined by a curtain wall. The residential block features a pair of garderobe towers, with a parlour range and a hall between them.
The interior features five vaulted basement chambers, but the most obvious feature of the castle remains is an ornately arcaded parapet, probably added by Henry Gower in the 14th century.