The town is located in a wonderfully scenic setting ringed by hills on all sides. The streets are laid out in the form of a cross, with the town square at the crossing point. In the square is the most distinctive building in Dufftown; the old tolbooth tower.
Immediately north of the town centre is Balvenie Castle, a 13th-century fortress once owned by a succession of powerful noble families, including the Comyns, Douglasses, and Stewarts, and visited by Mary, Queen of Scots. The castle is surrounded by a high curtain wall, within which stands a three-storey 16th-century tower house built by the 4th Earl of Atholl.
In the old kirk at Mortlach, on the southern outskirts of Dufftown, stands the Mortlach Battle Stone, a late Pictish symbol stone. Mortlach itself is said to be the site where in 1010 King Malcolm II defeated a huge force of Danes.
About three miles to the south-east, on the banks of the River Fiddich, stand the ruins of Auchindoun Castle. If you fancy some exercise - and this is wonderful walking country - the Speyside Way long-distance trail starts just north of Dufftown and runs along the River Spey to Aviemore in the Cairngorms National Park.