A number of stars have revealed they are leaving the ITV drama