Colchester, the oldest military town in Britain with a large population of veterans, has responded to Vance's comments