Sir Keir Starmer was working when he met a voice coach during the winter of 2020 and suggestions that it broke lockdown rules are “mudslinging”, a minister has said.