Trump loyalist said his appointment proved too much of a ‘distraction,’ leading him to drop out