City minister’s proximity to Keir Starmer and delayed reaction from Tory opposition are part of the picture