The Pointless star made his stance clear when taking part