The Whatmore family lived on site for over 20 years