Clement says the punters don't know 'all the facts' after his decision was booed