Elasticsearch-js: Responses that are too big cannot be handled gracefully

[diegomansua]

Hello,

🐛 Bug Report

Maybe not really a bug but an improvement.

The response of a query that is too big for the Node process to handle will result in an uncaught exception.

This line: https://github.com/elastic/elasticsearch-js/blob/v7.9.1/lib/Transport.js#L204

concatenates the response chunks. The problem is that the response is too big (due to the application running a "bad" query or the index containing "bad" data that is too large) this concatenation will result in a RangeError: Invalid string length that cannot be handled by the application other than with Node's uncaughtException handler, where any context is lost (not possible to know which query caused it).

To Reproduce

Run a standard query that returns a lot of data.

Expected behavior

I think the client should handle RangeErrors in the 'data' handler when the response is being concatenated, which would then cause a promise rejection with a proper ElasticsearchClientError error (with context about the query etc.) that the app can handle normally (handling the search promise rejection).

Your Environment

• node version: 14.9
• @elastic/elasticsearch version: 7.3.0
• os: Mac, Linux

[delvedor]

Hello! The maximum string size for Node.js is around 512MB.
I'll open a pr to check the Content-Length header and throw a better error so this issue can be easily solved.
Meanwhile, you could try to decrease the number of documents returned by each search (with the size parameter), and then use the scroll search helper for reading the docuemnts.
I would also recommend redesigning your ingestion phase :)

[diegomansua]

@delvedor Thanks a lot.

Yes the root issue was definitely an application one, no question about it. We had some documents that were much bigger than expected (caused by a different issue), but the ES client made it very difficult/nearly impossible to know where the root cause was (as in, which query was failing), as it was causing an unhandled exception where all context was lost, we literally had to resort to edit the client's source code (in prod!) to wrap the payload += chunk line into a try-catch to at least identify which query was the culprit (as we run dozens of different ones).