Elasticsearch-net: Problem with Scroll

[librecourage]

With the version 0.11.2, i can't retrieve results via scrolling

I launch a query. The response is :
{"_scroll_id":"c2Nhbjs1Ozc3NjgyOjYwZFRpNmJwVGZPdUpRNkh2cmRBSmc7Nzc2ODU6NjBkVGk2YnBUZk91SlE2SHZyZEFKZzs3NzY4NDo2MGRUaTZicFRmT3VKUTZIdnJkQUpnOzc3NjgxOjYwZFRpNmJwVGZPdUpRNkh2cmRBSmc7Nzc2ODM6NjBkVGk2YnBUZk91SlE2SHZyZEFKZzsxO3RvdGFsX2hpdHM6Mjs=","took":63,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":2,"max_score":0.0,"hits":[]}}

When i want to see the results, Nest sends this message :
c2Nhbjs1Ozc3NjgyOjYwZFRpNmJwVGZPdUpRNkh2cmRBSmc7Nzc2ODU6NjBkVGk2YnBUZk91SlE2SHZyZEFKZzs3NzY4NDo2MGRUaTZicFRmT3VKUTZIdnJkQUpnOzc3NjgxOjYwZFRpNmJwVGZPdUpRNkh2cmRBSmc7Nzc2ODM6NjBkVGk2YnBUZk91SlE2SHZyZEFKZzsxO3RvdGFsX2hpdHM6Mjs%3D

and i receive this response

{"error":"ElasticSearchIllegalArgumentException[Failed to decode scrollId]; nested: IOException[Bad Base64 input character decimal 37 in array position 223]; ","status":400}

[Mpdreamz]

Dang! i'm currently on holiday will have a look into this when i get back
next week.

On Tuesday, August 6, 2013, librecourage wrote:

With the version 0.11.2, i can't retrieve results via scrolling

I launch a query. The response is :

{"_scroll_id":"c2Nhbjs1Ozc3NjgyOjYwZFRpNmJwVGZPdUpRNkh2cmRBSmc7Nzc2ODU6NjBkVGk2YnBUZk91SlE2SHZyZEFKZzs3NzY4NDo2MGRUaTZicFRmT3VKUTZIdnJkQUpnOzc3NjgxOjYwZFRpNmJwVGZPdUpRNkh2cmRBSmc7Nzc2ODM6NjBkVGk2YnBUZk91SlE2SHZyZEFKZzsxO3RvdGFsX2hpdHM6Mjs=","took":63,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":2,"max_score":0.0,"hits":[]}}

When i want to see the results, Nest sends this message :

c2Nhbjs1Ozc3NjgyOjYwZFRpNmJwVGZPdUpRNkh2cmRBSmc7Nzc2ODU6NjBkVGk2YnBUZk91SlE2SHZyZEFKZzs3NzY4NDo2MGRUaTZicFRmT3VKUTZIdnJkQUpnOzc3NjgxOjYwZFRpNmJwVGZPdUpRNkh2cmRBSmc7Nzc2ODM6NjBkVGk2YnBUZk91SlE2SHZyZEFKZzsxO3RvdGFsX2hpdHM6Mjs%3D

and i receive this response

{"error":"ElasticSearchIllegalArgumentException[Failed to decode
scrollId]; nested: IOException[Bad Base64 input character decimal 37 in
array position 223]; ","status":400}

—
Reply to this email directly or view it on GitHubhttps://github.com/Mpdreamz/NEST/issues/326
.

[royborgiani]

I´m having this same problem with scroll, on ElasticSearch 1.7.1

{
"error": "ElasticsearchIllegalArgumentException[Failed to decode scrollId]; nested: IOException[Bad Base64 input character decimal 123 in array position 0]; ",
"status": 400
}

And I need to delete some items from a index with type boolean (true)

How can I solve this problem?

[russcam]

@royborgiani, what version of NEST are you using with Elasticsearch 1.7.1?

Also, do you have a concise example of what you're doing?

[royborgiani]

How I GET the NEST version?

I do the following, in the sense, I Execute the command below:

GET /twitter/tweet/_search?scroll=1m
{
"query": {
"match" : {
"type" : "false"
}
}
}

and I get this return:

{
"_scroll_id": "cXVlcnlUaGVuRmV0Y2g7NTs5ODpIUy1RLVFvbVNBcU1EX0lnNkZKNUdBOzk5OkhTLVEtUW9tU0FxTURfSWc2Rko1R0E7MTAwOkhTLVEtUW9tU0FxTURfSWc2Rko1R0E7MTAxOkhTLVEtUW9tU0FxTURfSWc2Rko1R0E7MTAyOkhTLVEtUW9tU0FxTURfSWc2Rko1R0E7MDs=",
"took": 9,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 0.30685282,
"hits": [with some hits that is my clause to limit the returns]

Then I: get the _scroll_id insert with the following command:

DELETE /_search/scroll -d 'ccXVlcnlUaGVuRmV0Y2g7NTs1OTg6SFMtUS1Rb21TQXFNRF9JZzZGSjVHQTs1OTk6SFMtUS1Rb21TQXFNRF9JZzZGSjVHQTs2MDA6SFMtUS1Rb21TQXFNRF9JZzZGSjVHQTs2MDE6SFMtUS1Rb21TQXFNRF9JZzZGSjVHQTs2MDI6SFMtUS1Rb21TQXFNRF9JZzZGSjVHQTswOw=='

So, I get that error:

{
"error": "ElasticsearchIllegalArgumentException[Failed to decode scrollId]; nested: IOException[Bad Base64 input character decimal 47 in array position 7]; ",
"status": 400
}

:(

[russcam]

Check the nuget package version of NEST that you are using in packages.config (or project.json if using xproj). If you're using Elasticsearch 1.7.1, then the latest NEST version that is compatible is NEST 1.8.0.

In your example above, where is the scroll_id value that you send to delete/clear the scroll coming from? It's not the same as the scroll_id that is returned from the search?

[royborgiani]

in the example no, but, i´m using the same scroll_id like below:

GET /twitter/tweet/_search?scroll=1m
{
"query": {
"match" : {
"type" : "false"
}
}
}

GET /_search/scroll
{
"scroll" : "1m",
"scroll_id" : "cXVlcnlUaGVuRmV0Y2g7NTs2ODg2NjpIUy1RLVFvbVNBcU1EX0lnNkZKNUdBOzY4ODY3OkhTLVEtUW9tU0FxTURfSWc2Rko1R0E7Njg4Njg6SFMtUS1Rb21TQXFNRF9JZzZGSjVHQTs2ODg2OTpIUy1RLVFvbVNBcU1EX0lnNkZKNUdBOzY4ODcwOkhTLVEtUW9tU0FxTURfSWc2Rko1R0E7MDs="
}

and the return of the second command is:

{
"error": "ElasticsearchIllegalArgumentException[Failed to decode scrollId]; nested: IOException[Bad Base64 input character decimal 123 in array position 0]; ",
"status": 400
}

Finally, the same result as on the last comment before...i see this error, on delete or get...

[dzmitry-lahoda]

NEST 2.1 request:

POST: /_search/scroll?pretty=true
{
  "scroll": "600s",
  "scroll_id": "cXVlcnlBbmRGZXRjaDsxOzQ2NDE0Njg6MHhVOHlQeXpRMENmSkIwMDQ0NEdGdzswOw=="
}

NEST 1.6.1 request:

POST: : .../_search/scroll?scroll=600s&pretty=true, 
    Request: cXVlcnlBbmRGZXRjaDsxOzQ2NDE0NDE6MHhVOHlQeXpRMENmSkIwMDQ0NEdGdzswOw== 

As it is visible newer version send JSON, while older send base64 string and query string parameters.

[n3zukochan]

Is this issue still open? I still have the issue for Nest 2.3.1

[russcam]

@kurokuriboh looking into it

[n3zukochan]

I actually found out the issue. Aws still uses elasticsearch 1.5 and the bug is not fixed until later version. The issue can be closed I believe. Sorry for that

[russcam]

I'll close it after we're happy that it's not a wider issue.

By the way @kurokuriboh, you should use the latest NEST 1.x release with Elasticsearch 1.5; NEST 2.x is not supported as being compatible. It _may_ work for the most part, but there are breaking changes between Elasticsearch 1.x and 2.x that are reflected in changes to NEST, meaning that some things will not work at all.

[Mpdreamz]

Just validated this works as expected with Elasticsearch 1.x and Nest 1.x. The original fix was to not encode the traling = at the last position the fix for which went into Nest 0.12.

If you are seeing a problem at position 0, you are most likely sending json to Elasticsearch 1.x using a 2.x client. The scroll API only supports json since Elasticsearch 2.0.0

[geremyCohen]

FTR, the equivalent CURL commands against ES 1.5 are:

curl -XGET "https://localhost/twitter/tweets/_search?scroll=2m"
curl -XGET "https://localhost/_search/scroll?scroll=2m" -d 'SCROLL_ID_RESPONSE_VALUE_FROM_LAST_REQUEST'