Speculation has been mounting about James Farrar's future on the BBC show