Sir Ed Davey also said it was a "real matter of concern" that Mr Musk was "camped out in Trump's Oval Office".