Fans watching the ITV show's live semi-final were left fuming as a star was sent home