The Baggs were on the show for three series but left in 2022