Elasticsearch: Reindex http entity content buffer limit should be configurable

[messense]

Elasticsearch version: 5.0

Describe the feature:

Seen a lot of error entity content is too long [10800427] for the configured buffer limit [10485760] when reindexing from remote, please make buffer limit configurable, currently it's hardcoded to 10MB.

[nik9000]

I'll add this to my list of things to investigate on Monday. You can work around it by using a smaller batch size which looks like:

POST _reindex
{
  "source": {
    "index": "source",
    "size": 100
  },
  "dest": {
    "index": "dest"
  }
}

My instinct is that a buffer limit of 100mb is more appropriate than 10mb for reindex-from-remote and if you want to go beyond that you should use a smaller batch size. Buffers larger than 100mb are likely to send similarly sized bulks and we know bulk performance suffers with batches that large.

[nik9000]

reproduction

[PhaedrusTheGreek]

@nik9000 , docs are showing that the buffer size is 200mb in 5.x, but i'm hitting a 100mb limit in 5.1.1:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Remote responded with a chunk that was too large. Use a smaller batch size."}],"type":"illegal_argument_exception","reason":"Remote responded with a chunk that was too large. Use a smaller batch size.","caused_by":{"type":"content_too_long_exception","reason":"entity content is too long [245498346] for the configured buffer limit [104857600]"}},"status":400}Done with <removed>, moving to the next one...

[nik9000]

@nik9000 , docs are showing that the buffer size is 200mb in 5.x, but i'm hitting a 100mb limit in 5.1.1:

Yeah, I remember fixing that but I it looks like I fixed it in a PR that went only to 5.2+.

[nik9000]

@PhaedrusTheGreek, I pushed 1294035aa7b4549beeeaff22b5403c3479c1bc5d to 5.1 to fix the docs in 5.1. It looks right in 5.x.

[egyptianbman]

This memory limit really needs to be configurable. The limit that's currently in place makes remote reindexing a nightmare. I have one of two options:

Option 1:
Reindex all the indexes with a size of 1 to ensure I don't hit this limit. This will take an immense amount of time because of how slow it will be.

Option 2:
Run the reindex with size of 3000. If it fails, try again with 1000. If that fails, try again with 500. If that fails, try again with 100. If that fails, try again with 50. If that fails, try again with 25......... (the numbers here don't matter but the point is -- with an index dozens of gigabytes, I can't be sure what the maximum size that I can use is without actually running the reindex).

If the limit can't be changed due to other issues that it would cause, then is it possible to add the ability to resume a reindex instead of having to start over every time? Thank you!

[nik9000]

If the limit can't be changed due to other issues that it would cause, then is it possible to add the ability to resume a reindex instead of having to start over every time? Thank you!

Not really.... Not with the tools we have now. With tools we're building, maybe.

I wonder how expensive it'd be to have a script that filtered out documents that are huge. Then you can do them in two passes. That'd be a thing you could do right now. It'd be better for the nodes then allowing the buffer to balloon up uncontrollably. Not great for usability, obviously, but better than nothing.

[zqc0512]

can change the configured buffer limit with configure file or curl?

[sysr-q]

Big :+1: from me on configurable buffer size. Have been doing a reindex-from-remote of 21million odd documents, where most are tiny, but there are a few big ones peppered in to make my life fun. Have to run a small batch size -- like egyptianbman, constantly trying again with smaller and smaller until I find one that doesn't eventually die -- as well as babysit it to make sure it doesn't stop without me noticing. Less than ideal.

[timtutt]

Would really love it if this was looked into again. The buffer limit of 100mb just seems to be way too low. When reindexing from remote this seems to be my bottleneck for indexing rate. I even increased by node count 3x and still received the same performance because I'm having to use a size of 10 just to get docs indexed.

[fhalde]

was anything done for this?

[DylanGriffith]

Is there any way to configure this yet? It appears to be the bottleneck for us doing remote reindex as well and thus we are forced to use a very low batch size considering our cluster size.

[luispollo]

Piling on here... I ran into this issue during a migration from ES 2.x to 6.x, and was unable to complete the reindexing process even with a batch size of 1. I know it's ridiculous to have a document that large in the index, but that's another discussion.

If nothing else, I'd suggest adjusting this bit of the docs:

Reindexing from a remote server uses an on-heap buffer that defaults to a maximum size of 100mb.

The word "defaults" there is misleading, since there's no way to configure/override that limit. In other words, it is effectively a hard limit, and the documentation should reflect that.