Airflow: Auto-generate installation.rst

[mik-laj]

Description

Not all users know that the instalation.rst file should be updated when they add a new library to the setup.py file. I think we should standardize it so that the tables contained in installation.rst are generated automatically.
https://airflow.readthedocs.io/en/latest/installation.html

Are you wondering how to start contributing to this project? Start by reading our contributor guide

Use case / motivation

N/A

Related Issues

N/A

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[kaxil]

Welcome to the community @mik-laj 😄

[szn]

Interested in that

[szn]

I have a case needed to be clarified @mik-laj @potiuk:

setup.py:

...
EXTRAS_REQUIREMENTS = {
    'amazon': amazon,
    'aws': amazon,  # TODO: remove this in Airflow 2.1

installation.rst:

| aws                 | ``pip install 'apache-airflow[amazon]'``            | Amazon Web Services |

As far as I understand this is one extras (amazon) with additional alias (aws). I would expect the official name to be mentioned in installation.rst file (amazon). Q1: right?

Q2: is TODO remove saying _remove alias_ or _remove extras_ and should actually be doubled in setup.py?

[potiuk]

Good questions :)

Q1: right
Q2: remove is remove alias - should be indeed doubled in installation.rst

[potiuk]

For the Q2 case. It might actually be easier if we separate out all the deprecated extras to a separate constant DEPRECATED_EXTRAS_REQUIREMENTS. We just need to make sure to add those two where we use EXTRAS_REQUIREMENTS so far.

[potiuk]

But it's up to you :)

[szn]

Next question:

Some extras are defined in a name: name style (see cloudant below). Others define it's name and alias (see atlas below). And there is async extras which defines only alias (see below).

setup.py:

    'cloudant': cloudant,

    'atlas': atlas,
    'apache.atlas': atlas,

    'async': async_packages,

Q3: Should actually async be defined twice in setup.py same as atlas?
Q4: If not, what's the difference between them?

[potiuk]

Q3: The "apache.atlas" is the new "official" name in 2.0. As all other apache external projects - they are all grouped in the "apache" subdirectory in providers and ultimate goal when we release 2.0 is to have 1-1 mapping of provider names to the corresponding "extras names" (some extras are not really providers, so there will be more extras than providers). So "atlas" should be added to the deprecated extras (TODO and everything).

Q4: It does not really matter what the key is, we only care about the name in this dict. so "async" is a perfectly valid extra name - not an alias. And there should be only one, we do not plan to change that name and do not plan to deprecate the "async" one.

[potiuk]

Great questions BTW!

[mik-laj]

'async': async_packages,

async is a restricted keyword in python. You cannot create a variable with this name, but it can be a key in the dictionary.

[szn]

@potiuk @mik-laj PR marked as ready for review