Andrew has been cast out of the fold after allegations made against him