Keir Starmer has said he is not “rummaging around opinion polls” or looking “at what pundits are saying” as Labour’s public support continues to decline.