Kamala Harris was fighting an uphill battle with 45% of voters saying they felt worse off than four years ago