Sale of prominent building comes 'despite constant negativity' towards city