He has accused the party of age and sex discrimination