I told you: 'This is not a good time to do this, guys' - Tory leader