Voters and politicians are worried about unauthorised migrants