The actor subsequently left the Fox competition show after the second episode