The ITV show has lost millions of viewers over the past two years