The residents have been told they must leave by Sunday