The town centre building has lain empty since Johnston Carmichael moved out and into Elgin Business Park.