The Prime Minister’s new clothes? They’re not all bad, insiders tell Joe Bromley