Residents says building has now been empty for years