Swift fans recently criticized her for interacting with Brittany due to their seemingly differing political views