Fans said it was "such as shame" as Jamie Borthwick departed the contest