Viewers of the ITV entertainment show had appeared to be in agreement with judges on the show before changing their mind