Lord Hayward said recent local by-elections suggest Nigel Farage’s party may not be as popular as recent polls have suggested.