The building next to the former and now bulldozed restaurant is said to be in 'poor condition'