The Grade II listed building could easily be walked past in the street