They said the star was at a disadvantage this week compared to the other contestants