A small downland village, Clayton lies astride an old Roman road. The little Saxon church of St John the Baptist boasts some very early wall paintings, which are probably the work of monks from St Pancras Priory at Lewes. The paintings have been variously dated from 1080 to 1140, making them some of the earliest surviving examples in England.
The paintings above the chancel arch are particularly rich and ornate. Two historic mills dominate the skyline above Clayton. These two mills are generally referred to as Jack and Jill. Jack, the larger, was built as a tower mill about 1866. It is now a private residence. Jill, a post mill, is smaller than Jack, but older, built about 1821 at Brighton. It was later moved to Clayton, and is now open to visitors, who can watch corn being freshly ground.

Near Clayton is a historic railroad tunnel, one of the largest built during the 1840s railroad boom in Victorian Britain. Above one end of the tunnel is Tunnel House, a mock Tudor dwelling built as a residence for the tunnel keeper.