Kibana: Provide TS typings for Elasticsearch queries

[timroes]

The more we refactor to TypeScript the more places I see that it would be a benefit to have proper typings for Elasticsearch queries and responses. E.g. I was looking into getting some of the AggConfig or AggParam infrastructure moved over to TypeScript, because we also have an awful amount of createLegacyClass (#21360) in that part.

Since those write out aggregation I think it would be really helpful to be able to provide more typings than just object or any, to prevent upcoming issues and ease maintenance of that code.

I also think there might be other web projects out there, that could benefit from that, so I think it could make sense to publish that package to npm independently.

I currently see basically two ways to achieve that.

1) Do we have any formal description of the Elasticsearch API maintained by the Elasticsearch team? If so I think it would be worth looking into possibilities to write something that converts those to TS typings instead.

2) If that's not possible I think we need to build them ourselves. We could ideally use some existing work as a base for that, e.g. @jacobwgillespie has done an awesome job already writing (mot of the?) typings: https://github.com/jacobwgillespie/elastic-ts/tree/master/src/types This approach of course have the disadvantage that we need to update them manually, which will always be more work as if we can just auto generate them.

Even if we need to go down the second choice I think we should soon start looking into building them, so we can build our ES relevant code on top of that.

cc @Stacey-Gammon @lukeelmers

[elasticmachine]

Pinging @elastic/kibana-platform

[epixa]

The new elasticsearch-js client that @delvedor is working on has TypeScript support out of the box. We'll ultimately need to swap out our existing elasticsearch client for this one, so I think we'll be getting the types for free (to Kibana) here.

[timroes]

Ah that's perfect. @delvedor We should make sure that those types are exported properly so you can use them for typing your own code, even in places you don't do immediately a request.

@delvedor I would highly be interested if you could outline some of the typing ideas/plans for that library here if possible.

[delvedor]

Hello! I'm integrating the new client and kibana in https://github.com/elastic/kibana/pull/27051.
At the moment, I'm fixing the failing tests, but I'm having some issue with TypeScript as well.

If you like, we can organize a meeting where I show you the current work and the plans for the future.
I would love some help on the TypeScript side, currently, the client is fully typed, but I still need to figure out how to provide typed request and responses :)

[jacobwgillespie]

Oh nice, having types in the official JS client would be awesome! My prototype project was created by manually reading the documentation and then porting that over to TS, so something more official would be sweet.

It looks like the new client uses anyObject for the request parameters - I wonder if it would be possible to auto-generate params like @timroes mentioned at the fidelity of my attempt (where the whole request object is typed)? I haven't spent a lot of time digging through the ES source, but presumably something validates incoming requests against known schemas / keys.

[delvedor]

The request typings can be generated, we generate the clients from the rest-api-spec, which is a json specification of every parameter that an API can accept.
I'm not aware of something similar for the responses (cc @Mpdreamz).

[jacobwgillespie]

rest-api-spec looks interesting - I originally though about using that for my project, however it isn't typed at the same level as TypeScript supports...

For instance, body in a search query is untyped (unless I'm missing something):

https://github.com/elastic/elasticsearch/blob/816e6f6a08be821d8ec01b51bf383544d1e341b0/rest-api-spec/src/main/resources/rest-api-spec/api/search.json#L184-L186

And _source is typed as a list, but it's actually a boolean or a list:

https://github.com/elastic/elasticsearch/blob/816e6f6a08be821d8ec01b51bf383544d1e341b0/rest-api-spec/src/main/resources/rest-api-spec/api/search.json#L100-L103

That said, I assume the Java understands the types of everything at a deeper level, so there should be some way to extract.

[Mpdreamz]

Closest we have is a dump of the .NET mappings of the API to TypeScript:

https://github.com/elastic/elastic-client-generator/tree/master/specification/specs

The automated exporter still needs some TLC e.g not all object inheritence gets persisted in typescript.

Another reason work on this is stale is that the java server code is now moving from builders to pojo's for REST related classes. This is a much nicer source then the .NET mappings.and I want to jump on this to help get it into a state we can generate specs from in the new year.

[delvedor]

I've just opened a pr that adds typed request parameters to the new JavaScript client, feel free to give it a look!
https://github.com/elastic/elasticsearch-js/pull/748

[timroes]

@jacobwgillespie thanks for your feedback here. Since you seem to work already with a TS version in your software, I would be highly interested if you have any feedback for the new official client. Hopefully we can leverage some of your experience with your client for that.

[timroes]

After discussing with @delvedor we came to the following decisions:

• We will only provide the request parameter typing that is mentioned above. We won't provide additional typing for request bodies or responses so far from the official client, since we can't auto-generate that and thus would have a very high maintenance to make sure that's always up to date.
• Elasticsearch has providing restspecs for request/responses on their TODO list. Once those exist we will use them to generate also typings for requests/responses in the official client. We will check with the ES team what's their roadmap for adding those restspecs and hopefully will see that in the near future.
• Since the previous DefinitelyTypes typings for elasticsearch-js already provided some typings for requests/responses, and since we (as mentioned initially) we would benefit from them in several places, #27051 will add a typings file in which we can add those request/response typings for now, so the existing consumers in Kibana will continue to work, without requiring to fall back to any. Also, @lukeelmers if we need those types in advance we can provide them in that place.

I will close this issue for now, and will open an issue with elasticsearch-js to provide request/response types once we can auto generate them.