The Italian chef is also alleged to have made a crass remark about Mary Berry