The Celebrity Gogglebox star spoke out amid Zoe's absence from her BBC Radio 2 show