They don't want rough sleepers housed in a town centre