In the latter half of the 12th century, a third fortification followed, on a new site across the river. The builder was Robert, Earl of Gloucester, and he, too, founded an abbey, this time at Margam.
So for a time there were two Norman castles, on opposite banks of the river, a short distance apart. Obviously, two castles were not required to guard the river crossing, and by 1207 the earlier de Granville castle was pulled down. In 1231 Llewelyn ap Iorwerth sacked Neath and burned the castle. It was rebuilt, this time in stone, with a pair of towers and a gatehouse.
In the early 14th century the castle was owned by Hugh le Despenser, Edward II's favourite. The castle was sacked once more by Hugh's enemies - of which there were many - and the gatehouse was rebuilt to produce the entrance we see today.
Aside from the gatehouse, which is easily the best-preserved part of the site, there are sections of surviving curtain walls. A cockpit was later built within the castle enclosure.