Anti-illegal immigration protestors gathered outside Roebuck House after it was confirmed that it would not be used to house asylum seekers.