Canadians' views are hardening on regular immigrants, who have been blamed for increasing housing costs and employment challenges