Neighbours say the structure is looming over their properties and making their lives a misery