Prime Minister Sir Keir Starmer has been warned Britain's immigration crisis has led to "immense and lasting harm".