Sunday Brunch host Simon Rimmer opened up saying he still faces backlash to this day over one dish he created