Sir Keir Starmer was told his party organising trips to America could be "damaging to our national interests".