Wye College was founded in the mid 15th Century by Cardinal John Kempe who was born at Olantigh, just outside Wye village. He was a classic example of the later medieval political prelate; becoming Bishops of Rochester, Chichester and London, and later Archbishop of York. He was created a Cardinal in 1439 and ultimately became Archbishop of Canterbury. He combined his ecclesiastical duties with that of Lord Chancellor from 1422 to 1432 and again from 1450 to 1454. Towards the end of his life he set about rebuilding Wye Church and in 1447 founded The College of St Gregory and St Martin, generally referred to as Wye College. The college was established to train secular priests and to educate local boys in the attached grammar school. The college was housed in the fine set of medieval buildings that still grace the centre of the village today. 

In 1890, the buildings were bought by Kent and Surrey county councils to establish the South Eastern Agricultural College. For over a hundred years the college, which became part of the University of London and latterly Imperial College, developed a worldwide reputation in the fields of agriculture, horticulture and the environment. Students came to study at Wye College from across the globe and the influence of the ground breaking teaching and research, are still in evidence all over the world.

In 2009 the college sadly closed and the buildings remained empty for over ten years. Work has now started to convert the Medieval and Edwardian buildings into residential accommodation ensuring their long term care and protection. As part of the development the Heritage Centre will move from the Latin School to a permanent location within the college complex, complete with dedicated research, archive and exhibition areas.