Conservationists have lost a battle to save one of Paisley’s most historic buildings.