The most interesting historic building is, without doubt, the old parish church of St Peter, now sadly roofless, which dates to at least the 14th century. In the churchyard stands a 14-foot high medieval cross.
A mile or so the west of Duffus village stands the ancient standing stone known as Camus's Stone, while a mile to the east stands Gordonstoun House, now an independent school.
To the south-east stands Duffus Castle, perhaps the best surviving example of a Norman motte and bailey castle in Scotland.
The Norman castle was likely a mere wooden palisade atop the conical mound we see today, but the wooden fortress was replaced in the 14th century by a strong stone keep, to which domestic buildings were added in the 15th century.