The Conservative-led administration insists the people that are set to be removed are already adequately housed