Elasticsearch-js: Connect to Multiple Different ES API versions

[austince]

💬 Questions and Help

In reference to my question https://github.com/elastic/elasticsearch-js/issues/765#issuecomment-469377109, how would one connect to multiple different ES clusters, all running different versions, from one application? Is the only solution to alias the different @elastic/elasticsearch packages?

[delvedor]

Hello!
If all the clusters are running the same major version of Elasticsearch (eg 6.2.3, 6.6.2, and 6.1.2), then you can use three instances of the client with version 6.
If you have different major numbers instead, you should create "sub packages" in your project which export a different version of the client.
For example, something like the following.
 /project /index.js <-- requires es5 and es6 /node_modules /es5 /index.js /node_modules /es6 /index.js /node_modules
Unfortunately, npm does not support different versions of the same package at the same time, so you should craft your own solution.

---

Note: @elastic/elasticsearch has not published all the major version yet, we'll do that soon :)

[austince]

Unfortunately, the clusters are on different major version, which I think is something that should be supported out of the box with this client. It looks like yarn is able to handle installing multiple versions of a package (see comment) - not an ideal solution but I believe it's better than having submodules within one's own project just to export different versions of a client.

I know you have all spent a lot of time planning out this new restructure, and it does solve certain maintenance problems, but I think it is really unfortunate that you are forcing users into these hacky solutions. I'm not sure who else will experience these problems but I imagine it will not just be me.

[delvedor]

Unfortunately, the clusters are on different major version, which I think is something that should be supported out of the box with this client.

Offer all the versions of ES in a single package would mean distribute a huge package since there are a lot of APIs.
The legacy client was doing that and we got a lot of complaints in this regard.
Furthermore, we have decided to align the JS client to every other client we distributed, where the client is in sync with the Elasticsearch version.
It makes easier hunting down bugs and offers a better overall experience.

but I think it is really unfortunate that you are forcing users into these hacky solutions. I'm not sure who else will experience these problems but I imagine it will not just be me.

This is a problem that is valid for the entire Node.js ecosystem.
It's something that should be fixed upstream and not in the single packages IMHO.
Finally, I don't think that the solution proposed above is "hacky", it is a perfectly valid folder structure, that might also help you during testing.

[austince]

Hey @delvedor, I don't mean any disrespect and don't want to this discussion to have unnecessary negativity. I said the proposed solution was "hacky" because it requires users to maintain dependencies in multiple locations.

I see the issues that your team identified with including all APIs, I am just saying that _because we know multiple package version dependencies are a known problem in the Node.js ecosystem_ I wish that there was more discussion with the community before these changes were started. It seems that we are at a point-of-no-return with the new package versioning scheme but I think there could have been other ways to address the packages issues that would have come up in community discussion. For instance, perhaps strictly following the Elastic Product EOL schedule for which APIs to include would have helped mitigate the package size. Again, I'm not sure, but the point of my comments are that I think it would be nice in the future to try to get input from more of the community.

[delvedor]

Hey @delvedor, I don't mean any disrespect and don't want to this discussion to have unnecessary negativity. I said the proposed solution was "hacky" because it requires users to maintain dependencies in multiple locations.

No worries, feedbacks are always appreciated!

For instance, perhaps strictly following the Elastic Product EOL schedule for which APIs to include would have helped mitigate the package size.

The problem with this solution is how you would handle breaking changes?
For example, if in the future Elasticsearch introduces a version specific parameter, it would increase the code complexity a lot, same for testing, which is one of the main issues (we are testing the full API of Elasticsearch, with more than 10k integration test, -per version-). Furthermore, we should also handle the EOL of Node.js which makes the picture even more complicated.

Being able to follow the same train release of Elasticsearch helps the client to offer more guarantees, better testing, and better handling of breaking changes.
I feel you, and I understand your problem, but I think this is still the best way to offer a good product.

Besides the method I suggest you above, another solution to support multiple versions at the same time is to use the brand new extend API.
If you don't need the full API surface, but a small portion instead, you can implement it in this way.
We can also think about enhancing the extend API, and allow to pass a configuration object other than a function.

For example:

client.extend('es5.search', {
  path: (index, type) => `/${index}/${type}/_search`,
  method: 'GET',
  ...
})

client.es5.search(...)

What do you think?

[austince]

Hey @delvedor, sorry the for the delay and thank you for the further options for addressing my problem + explanations. I don't love the solutions, but I trust you + your team that this is the best way to offer the client and work within the Node constraints!

[delvedor]

Hello! Thanks to npm v6.9 you can install multiple versions of the same package via the aliasing feature :)

Checkout https://github.com/elastic/elasticsearch-js/pull/835!

[austince]

Awesome, that looks like it checks all the boxes - thanks!