The mum has branded Peter Kay a "bully" but the comedian stands by his remark