His comments were branded 'wholly inappropriate' by a misconduct panel