Ellon was traditionally the major centre of the old Buchan region of modern Grampian, however much of the present town is a product of the 19th and 20th centuries.

However, the past is still very much present in Ellon; with the parish church and remains of Ellon Castle standing on the market square. Ellon was created a burgh of barony in the 13th century, under the rule of the powerful Comyn family, Earls of Buchan.

The Comyns built a timber castle atop the motte of Moot Hill. The motte is long gone, but a later 14th-century castle stands on a terraced bank overlooking the river.

In 1706 Ellon Castle was purchased by James Gordon, who built a fantastic terraced landscape garden, braced by a stone retaining wall. Gordon's family did not have long to enjoy the gardens after they were built, for his two sons were murdered by the music master, and the estate was sold to the 3rd Earl of Aberdeen.

The 'Wicked Earl, so named because he simultaneously raised three families, installed one of his mistresses, Penelope Dering, at Ellon Castle. The grand castle buildings fell into decay after the 18th century, and the current castle is the stables and servant's quarters of the 18th-century building.

Four miles distant is Arnage Castle, a privately held fortified tower house built by the Cheyne family. About a mile to the west is Esslemont Castle, a ruined tower house that once belonged to the Cheynes as well. Just to the east of Ellon, south of the river at Ladymire, are the remains of ancient hut circles.