The actor experienced a wave of racist abuse after he was cast in the series