
Unfortunately for the Comyns, Walter's son, John Comyn, was a bitter rival of Robert Bruce. In 1306 Bruce murdered John Comyn in one of the pivotal acts of the Scottish Wars of Independence. The Comyn family then threw their allegiance firmly behind Edward I of England, against Bruce and his allies. It was a fatal move for the fortunes of subsequent generations of Comyns, for Robert Bruce outlawed the Comyns. Less than 50 years after Balvenie was built, the Comyns were gone.

We do not know who received Balvenie from Robert Bruce, for the castle disappears from the historical records for a century. What we do know is that sometime in the 14th century the castle fell into the hands of the powerful 'Black' Douglas family. Evidence of their ownership is scanty, though it seems likely they rebuilt the accommodation along the west, south, and east ranges.

Much of the castle as we see it today is the result of Stewart rebuilding of the earlier medieval structure. The most obvious and impressive Stewart feature is the round tower beside the main entrance. This is the Atholl Lodging, probably built by the 4th Earl and his first wife, Lady Elizabeth Gordon. Mary, Queen of Scots stayed in one of the second floor chambers of the Atholl Lodging in 1562 when she visited Balvenie as part of her campaign against the Earl of Huntly.
In 1595 the 5th Earl of Atholl died without an heir, and the castle reverted to the crown. It passed through a number of hands, and was used as a military base by the Marquis of Montrose, followed by Royalist troops, Jacobite troops, and even troops loyal to William and Mary of England. In 1718 the lord of Balvenie, William Braco, committed suicide, and the castle was no longer used as a residence. Finally, in 1929 Balvenie was taken into state care, and it is now looked after by Historic Scotland.
By far the most impressive feature of the castle are the Atholl Lodgings, with chambers on three floors, reached by a spiral stair. Within the courtyard are a number of ancillary chambers, and several storage cellars. Perhaps the most interesting feature, however, is the double yett, or wrought iron gate, which defends the main entrance. The yett is set behind not one but two outer wooden doors to provide an extra level of security. In this case the yett is made of two hinged doors joined at the centre and hinged at the side, a design that is unique in Scotland. This style of defensive entrance gate replaced the more unwieldy portcullis in the late medieval period.


