A vintage tea room in a historic borough building is to close due to “mounting losses”, it has been announced.