Some land near a crumbling historic hall is being included in proposals to "significantly" reduce the size of its conservation zone.