The gaffe is eating away at his credibility