Vice President JD Vance and second lady Usha Vance visited the territory for just a few hours