Sir Keir Starmer and foreign secretary David Lammy spent time with former president before flying back to UK