A Port Glasgow campaigner claims that a historic building is dangerous and is a magnet for drug users and fire-starters.