Taylor Swift opted not to make the trip to North Carolina to watch her boyfriend Travis Kelce and the Kansas City Chiefs take on the Carolina Panthers on Sunday