Fans also thought presenter Lee Mack was being sharp with the contestants