The city is known worldwide for its history — but it's not a place well-visited by Brits themselves