Baroness Harriet Harman told the Prime Minister "doubling down" is angering and alienating voters.