Elasticsearch: _reindex in 5.4 fails against 1.7.6

[lemkepf]

Elasticsearch version: 1.7.6 and 5.4

Plugins installed: None

JVM version: 1.8.0_111, 1.8.0_131

OS version: Ubuntu 16.04

Description of the problem including expected versus actual behavior:
We are migrating from a 1.7.6 cluster to 5.4 using the reindex from remote in 5.4. I'm getting a weird ElasticsearchParseException error on the response after executing the request to reindex.

Here is my _reindex command I ran on my 5.4 cluster:

POST - http://search02:9200/_reindex
{
  "source": {
    "remote": {
      "host": "http://search01:9200"
    },
    "index": "labor"
  },
  "dest": {
    "index": "labor"
  }
}

Here is the response I got:

{
  "error": {
    "root_cause": [
      {
        "type": "status_exception",
        "reason": "body={\"error\":\"ElasticsearchParseException[Failed to parse [300000000000nanos]]; nested: NumberFormatException[For input string: \\\"300000000000nano\\\"]; \",\"status\":400}"
      }
    ],
    "type": "status_exception",
    "reason": "body={\"error\":\"ElasticsearchParseException[Failed to parse [300000000000nanos]]; nested: NumberFormatException[For input string: \\\"300000000000nano\\\"]; \",\"status\":400}",
    "caused_by": {
      "type": "response_exception",
      "reason": "POST http://search01:9200/_search/scroll?scroll=300000000000nanos: HTTP/1.1 400 Bad Request\n{\"error\":\"ElasticsearchParseException[Failed to parse [300000000000nanos]]; nested: NumberFormatException[For input string: \\\"300000000000nano\\\"]; \",\"status\":400}"
    }
  },
  "status": 400
}

I can execute a request with a "size" parameter in place and it works, but only the first 200 records.

{
  "size": 200,
  "source": {
    "remote": {
      "host": "http://search01:9200"
    },
    "index": "labor",
    "size": 2000,
    "sort": { "_id": "desc" }
  },
  "dest": {
    "index": "labor"
  }
}

If that size parameter goes up to 300 I get the same error as before but with a different scroll number:

{
  "error": {
    "root_cause": [
      {
        "type": "status_exception",
        "reason": "body={\"error\":\"ElasticsearchParseException[Failed to parse [2620000083934nanos]]; nested: NumberFormatException[For input string: \\\"2620000083934nano\\\"]; \",\"status\":400}"
      }
    ],
    "type": "status_exception",
    "reason": "body={\"error\":\"ElasticsearchParseException[Failed to parse [2620000083934nanos]]; nested: NumberFormatException[For input string: \\\"2620000083934nano\\\"]; \",\"status\":400}",
    "caused_by": {
      "type": "response_exception",
      "reason": "POST http://search01:9200/_search/scroll?scroll=2620000083934nanos: HTTP/1.1 400 Bad Request\n{\"error\":\"ElasticsearchParseException[Failed to parse [2620000083934nanos]]; nested: NumberFormatException[For input string: \\\"2620000083934nano\\\"]; \",\"status\":400}"
    }
  },
  "status": 400
}

I've played around with ?requests_per_second but that didn't seem to make any difference. I am seriously stumped... any ideas what I'm doing wrong?
Thanks

[nik9000]

any ideas what I'm doing wrong?

Nothing. We don't have tests that run reindex-from-remote against 1.7 because 1.7 doesn't fit into our testing infrastructure. I broke this.

[lemkepf]

@nik9000 - Ha. Is this something on the radar or should I be looking at other solutions? elasticdump is my next option.

[nik9000]

It is on my radar now, yeah. I've started working on something to run the old Elasticsearch now. But no matter what I do it isn't going to be released super fast. New versions of 5.3 ought to work but whatever works for you works for you.

[apierleoni]

Same thing happening here, when reindexing from 2.4.4 to 5.4

Query

{
  "source": {
    "remote": {
      "host": "http://myhost:9200"
    },
    "index": "myindex",
    "size": 100,
    "query": {
      "match_all": {}
    }
  },
  "dest": {
    "index": "myindex"
  }
}

Output

{
  "error": {
    "root_cause": [
      {
        "type": "status_exception",
        "reason": "body={\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"Failed to parse [300000000000nanos]\"}],\"type\":\"parse_exception\",\"reason\":\"Failed to parse [300000000000nanos]\",\"caused_by\":{\"type\":\"number_format_exception\",\"reason\":\"For input string: \\\"300000000000nano\\\"\"}},\"status\":400}"
      }
    ],
    "type": "status_exception",
    "reason": "body={\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"Failed to parse [300000000000nanos]\"}],\"type\":\"parse_exception\",\"reason\":\"Failed to parse [300000000000nanos]\",\"caused_by\":{\"type\":\"number_format_exception\",\"reason\":\"For input string: \\\"300000000000nano\\\"\"}},\"status\":400}",
    "caused_by": {
      "type": "response_exception",
      "reason": "POST http://myhost:9200/_search/scroll?scroll=300000000000nanos: HTTP/1.1 400 Bad Request\n{\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"Failed to parse [300000000000nanos]\"}],\"type\":\"parse_exception\",\"reason\":\"Failed to parse [300000000000nanos]\",\"caused_by\":{\"type\":\"number_format_exception\",\"reason\":\"For input string: \\\"300000000000nano\\\"\"}},\"status\":400}"
    }
  },
  "status": 400
}

[nik9000]

Right. I'm working on #23828 which will give us real tests that we don't break against older versions at the same time as I fix this. So I can be sure I'm not breaking anything else....

[nik9000]

Indeed. It effects all versions before 5. I have a fix. I'll open it in the
morning.

On Mon, May 8, 2017, 7:03 AM Andrea Pierleoni [e-mail]
wrote:

Same thing happening here, when reindexing from 2.4.4 to 5.4

Query

{
"source": {
"remote": {
"host": "http://myhost:9200"
},
"index": "myindex",
"size": 100,
"query": {
"match_all": {}
}
},
"dest": {
"index": "myindex"
}
}

Output

{
"error": {
"root_cause": [
{
"type": "status_exception",
"reason": "body={\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"Failed to parse [300000000000nanos]\"}],\"type\":\"parse_exception\",\"reason\":\"Failed to parse [300000000000nanos]\",\"caused_by\":{\"type\":\"number_format_exception\",\"reason\":\"For input string: \\"300000000000nano\\"\"}},\"status\":400}"
}
],
"type": "status_exception",
"reason": "body={\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"Failed to parse [300000000000nanos]\"}],\"type\":\"parse_exception\",\"reason\":\"Failed to parse [300000000000nanos]\",\"caused_by\":{\"type\":\"number_format_exception\",\"reason\":\"For input string: \\"300000000000nano\\"\"}},\"status\":400}",
"caused_by": {
"type": "response_exception",
"reason": "POST http://myhost:9200/_search/scroll?scroll=300000000000nanos: HTTP/1.1 400 Bad Requestn{\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"Failed to parse [300000000000nanos]\"}],\"type\":\"parse_exception\",\"reason\":\"Failed to parse [300000000000nanos]\",\"caused_by\":{\"type\":\"number_format_exception\",\"reason\":\"For input string: \\"300000000000nano\\"\"}},\"status\":400}"
}
},
"status": 400
}

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/elastic/elasticsearch/issues/24520#issuecomment-299837695,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AANLonPZfL-QaHyMsz4Km8RXjqjLv-Iwks5r3vZpgaJpZM4NSZL5
.

[ejsmith]

I'm running into this issue as well.

[MichaelPalmer-Orange]

Likewise getting this error as well. Running 5.4 _reindex against 2.4 - if I DON'T have a query or _source section in my remote section of the reindex, I don't get the issue but I get other issues (that I am trying to circumvent by using the _source).

[nik9000]

I merged the fix for this this morning. It included much better testing for reindex-from-remote against older versions of Elasticsearch so this shouldn't happen again. At least not in so obvious a way. It should ship with 5.4.1.

[MichaelPalmer-Orange]

Any chance we have an ETA on that?

[nik9000]

Any chance we have an ETA on that?

We have a policy of not predicting our future release dates publicly because we don't want to disappoint.

[tommymonk]

Also impacting us in production.

[vsiv]

We are also impacted by this. 5.3.2 does not have this problem. This might be an option for some who cant wait for 5.4.1. We can't however because we depend on word_delimiter_graph.

[DennisDyallo]

Just chiming in here to add that the source.size is the property that sets the TOTAL number of documents that get reindexed.

For example:
POST _reindex?wait_for_completion=false
{
"size": 10000,
"conflicts": "proceed",
"source": {
"size":10,
"remote": {
"host": "https://example.com:9200",
"username": "xxx",
"password": "yyy"
},
"index": ["sourceIndex"]
},
"dest": {
"index": "destinationIndex",
"op_type": "create"
}
}

Will put 10 documents in destinationIndex and then return, even though the reindex.size (query size) is set to 10000.

This is reindexing from 2.4.4 to 5.4 on Elastic Cloud.

PS. You should also change the names of these properties, it's confusing

[vsiv]

@DennisDyallo does that mean 5.4 would still work with the proper size set?

[DennisDyallo]

No, is you omit the total size (10000) it will just take the amount for the second size (10)

[dragonjohn]

Hi @nik9000 , doest it works from elasticsearch 1.4.5 to 5.4? I faced the same error output after did the reindex. Thanks.

[vsiv]

@nik9000 I think it might help if its possible to add a note to the docs of 5.4.0 reindex API as people might be unaware of this issue and get confused why reindex is not working .. It might save people some time..

• https://www.elastic.co/guide/en/elasticsearch/reference/5.4/docs-reindex.html

[nik9000]

@nik9000 I think it might help if its possible to add a note to the docs of 5.4.0 reindex API as people might be unaware of this issue and get confused why reindex is not working .. It might save people some time..

Yeah. I'll push something.

[ghost]

@nik9000 I am getting same error.How to download 5.4.1 package.

[nik9000]

5.4.1 has not been released. We are working on it.

On Tue, May 30, 2017, 2:23 AM Monika [e-mail] wrote:

@nik9000 https://github.com/nik9000 I am getting same error.How to
download 5.4.1 package.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/elastic/elasticsearch/issues/24520#issuecomment-304785629,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AANLou3YjF72U6l3LQ4ybRL5_yeyT3EEks5r-7XjgaJpZM4NSZL5
.

[vovolie]

same issue. reindex 1.6.0 to 5.4
dates release 5.4.1?
Thanks.

[nik9000]

5.4.1 is just now live.

[tinder-xhao]

Is 5.4.1 available in docker?

[nik9000]

@tinder-xhao, I believe so, though the current version is 5.5.1: https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html

[nik9000]

5.4.1 should also be available though.