President said he fears he ‘almost spent too much time on the policy, not enough time on the politics’