Paddy Bever has announced he is leaving the show after nearly four years and has already filmed his final scenes