A number of famous faces have already ‘pulled out’ of the series