The Traitors star Joe fell on the public's bad side after relentlessly going for fan-favourite Kas, which eventually saw the doctor banished wrongly