Campaigners have said they "will not stop" until The Crooked House is back where it once stood following a delay in a planning inquiry to decide whether it should be rebuilt.