Since endorsing Harris, Swift has earned the ire of Trump loyalists