The building lost its 'asset of community value' status last year