‘I’ve got no time for people who abuse their power,’ said the TV host