Protesters have gathered in opposition to Home Office plans to use a block of flats to house asylum seekers.