Councillors concerned agreement would set a precedent for housing on green space