Sir Keir Starmer has slapped down suggestions by an ex-advisor to Tony Blair that small family farms are an "industry we could do without".