FitzPons, a Norman lord, put a Welsh constable in charge of the castle, which might have been a dangerous choice, but his constable fiercely defended Llandovery from Welsh attack.
It then passed to Walter Clifford, but it was captured by Rhys ap Gruffudd, the ruler of Deheubarth, in 1158. Lord Rhys rebuilt the fortifications. Unfortunately, after Rhys's death his four sons fought over his lands, and Llandovery changed hands five times in only four years.
In 1277 it was taken by John Giffard, who held it for King Edward I. The king prompted Giffard to strengthen the defences to keep the castle safe from Welsh attacks. Gifard surrounded the stone keep with a curtain all, though little of the wall remains. He also built the D-shaped tower that still stands today and is the most impressive part of the castle.
In the tower you can see the remains of a garderobe on the first floor. You can also see the ruins of the twin-towered gatehouse and the ruinous foundations of several other structures within the castle bailey. These were most likely auxillary buildings such as a kitchen range.
Llandovery was briefly taken by Llywelyn the Last in his 1282 revolt, and again in 1287 by Rhys ap Maredudd, but it was held by the English monarch throughout the rest of its existence.
Some of the surviving castle walls date to Llywelyn the Last's strengthening of the castle in the late 13th century. It was briefly garrisoned against Owain Glyndwr, and in 1401 perhaps the most sombre event in the bloody history of Llandovery took place when a Glyndwr sympathizer named Llewelyn ap Gruffudd Fychan was executed in the presence of Henry IV.
Llewelyn's offence was to deliberately mislead Henry's troops and to refuse to reveal Glyndwr's location. A striking statue of him stands on the castle motte (the stainless steel of the memorial striking a rather peculiar note beside the medieval castle ruins, it must be said).
On the top of the motte are remains of the D-shaped tower and fragments of masonry walls. You can also see the foundation walls of a shell keep.
Access is easy; the castle rises immediately above the town parking area, just off the A40. The parking area and castle are well signposted. You can view the castle ruins from the parking area, but in order to actually explore the ruins you have to go through the gate to the neighbouring playground and climb a set of rather steep stairs to the top of the motte.