The ESPN personality said he doesn’t ‘like the fact’ he voted blue in November