The Grade-II listed six-bed property has a rich history