Labour dismissed concerns about the influx and accused the Tories of allowing immigration to soar when they were in power.