Show is facing backlash over controversial comments made on the Bake Off: Extra Slice show