A campaign to protect the site was backed by Mary Portas and acclaimed actor Josh O'Connor