It's Grade II* listed with a history that dates back to the 1500s