No one knows for certain when the dovecote was built, but it may have served as a watchtower for nearby Bruton Abbey as well as supplying the monks with food. The interior has 200 nesting boxes for pigeons, which gives rise to its alternative name 'The Pigeon Tower'.
The dovecote is made of local stone dressed with stronger Doulton stone. It is square in plan, standing three storeys high, though all the interior flooring and the roof have disappeared. There are gables on all four sides and an arched doorway on the north-east face.
You can see the remains of a fireplace inside - very unusual if it was only a dovecote, for the pigeons wouldn't need a fire! The most likely explanation is that there was a late medieval building here and it was later converted into a dovecote, possibly by the Berkeley family who bought Bruton Abbey after the Dissolution of the Monasteries.
Evidence for use as a watchtower is scanty; an 18th-century sketch shows the dovecote with a chimney on the roof, so it was definitely used for some kind of habitation.