Its owner said his grandfather had worked there and that the building was now at 'a tipping point'