Lord Brady wishes that the former prime minister had stuck to his freedom-loving instincts