The estate survived for less than 50 years after its first tenants moved in