The 18th century house is popular for its gardens