The chair of Prince Harry's African charity he stepped down from has lashed out at his brand, calling it "toxic".