GB News star has confirmed he is leaving the channel after four years - and fans were devastated by the news