Airflow: Build and publish API clients automatically

[potiuk]

Description

The OpenAPI allows to generate automatically clients for various languages. There are however some unresolved problems with some of the constructs in the Open API - for example oneOf with single schema produces uncompilable code despite being correct specification. Another example is "key" generic name of parameter, which in Java code might cause compilation problems in some cases as "key" variable name is used internally in generated methods.

We would like to make sure that our OpenAPI specification produces compilable and usable clients when auto-client generation is used. This can be easily setup as a CI step.

Use case / motivation

We want our customers to have very easy path in using the API. Being able to use pre-generated client API in their favorite language can save many days of work for integration.

CI building the API clients + API clients published as artifacts are great way to achieve that consistently.

Related Issues

7549

[mik-laj]

We have POCs: https://github.com/mik-laj/airflow-api-clients
If anyone wants, they can base on it or start from scratch.
An example of a successful client generation is Kubernetes: https://github.com/kubernetes-client
We can get inspired by their work.

[mik-laj]

@houqp Can you reply here? I can't assign you to a ticket without it.

[houqp]

@mik-laj sure, i will take this ticket.

[mik-laj]

I have prepared a spreadsheet in which we can collect requirements about what the client must have in order to be released.
https://docs.google.com/spreadsheets/d/11VaVxdZyYcVHtPwL0JL_qKJ4YzIyvefGcoUleBofn28/edit#gid=0
For now, this is a very short list, but I hope that we will develop it in time.
If someone has other requirements, please submit a comment / suggestion, I will add them.

[potiuk]

Love it. @houqp - happy to help with reviews and bouncing-off any problems you might have.

[mik-laj]

I add the "kind:meta" label because we won't be able to close it with one or two tickets.

[houqp]

@mik-laj are we still tracking the client progress with the spreadsheet you linked? I left couple comments there, but hasn't seen it updated. we can also create separate ticket for each language to track the status.

[mik-laj]

@houqp Thanks for pinging me I updated the spreadsheet and added new requirements about authorization. These are optional in most cases. I would prefer to have one spreadsheet so that adding a new requirement is updated for all clients.

[mik-laj]

@houqp I have given you editor permission on this sheet, so you can make changes to it.

[mik-laj]

We should add info about client library to docs:
https://airflow.readthedocs.io/en/latest/stable-rest-api-ref.html