MasterChef's Gregg sparked outrage when he said complaints about him came from a "handful of middle-class women of a certain age"