A row has erupted over a promise made by Keir Starmer during a visit to Swindon.