The city’s oldest tavern stands in stark contrast to the glass office blocks surrounding it