Airflow: Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)

Created on 6 Nov 2020 · 6Comments · Source: apache/airflow

Dear and Wonderful Citizens,

I started to look at what libraries we have defined in the constraints-*.txt file and I am a bit surprised because we have this constraints defined on very old libraries.
https://github.com/apache/airflow/blob/053afe7/constraints-3.8.txt

Update (@potiuk): -> Just for clarity: constraints are automatically generated from setup.py so this is a matter of dependencies defined there. If we are to fix it, we will have to upgrade dependencies defined in setup.py NOT the constraints themselves.

Sometimes we have defined libraries that are over 3 years old, which can cause security problems. Old versions of the library may have vulnerabilities that have probably been fixed in newer versions.

I am most concerned about dependency conflicts. Old libraries are only compatible with old libraries, which can cause problems if the user wants to use a new version of the same library.

I think it's worth investigating where these limitations come from and why we can't use newer versions of these libraries.

At the moment (5 November 2020). Here is a list of libraries that have updates but are not supported by Airflow according to constraints.txt

| package_name | current_version | latest_version | diff_part | diff between releases |
|------------------------------------|--------------------------------|-------------------------------|-------------|-------------------------|
| flask-swagger | 0.2.13 (4 years ago) | 0.2.14 (1 year, 7 months ago) | 3-patch | 2 years |
| clickclick | 1.2.2 (3 years ago) | 20.10.2 (a month ago) | 1-major | 3 years |
| iso8601 | 0.1.12 (3 years ago) | 0.1.13 (a month ago) | 3-patch | 3 years |
| azure-storage | 0.36.0 (3 years ago) | 0.37.0 (5 months ago) | 2-minor | 2 years |
| tzlocal | 1.5.1 (2 years ago) | 2.1 (5 months ago) | 0-unknown | 2 years |
| Flask-Login | 0.4.1 (2 years ago) | 0.5.0 (8 months ago) | 2-minor | 2 years |
| Markdown | 2.6.11 (2 years ago) | 3.3.3 (11 days ago) | 1-major | 2 years |
| pinotdb | 0.1.1 (2 years ago) | 0.3.3 (4 months ago) | 2-minor | 2 years |
| Flask-OAuthlib | 0.9.5 (2 years ago) | 0.9.6 (a month ago) | 3-patch | 2 years |
| oauthlib | 2.1.0 (2 years ago) | 3.1.0 (1 year, 3 months ago) | 1-major | 1 year, 2 months |
| jsondiff | 1.1.2 (2 years ago) | 1.2.0 (1 year, 4 months ago) | 2-minor | 1 year, 22 days |
| cached-property | 1.5.1 (2 years ago) | 1.5.2 (a month ago) | 3-patch | 2 years |
| zope.event | 4.4 (2 years ago) | 4.5.0 (a month ago) | 0-unknown | 1 year, 11 months |
| mysqlclient | 1.3.14 (1 year, 11 months ago) | 2.0.1 (4 months ago) | 1-major | 1 year, 6 months |
| dnspython | 1.16.0 (1 year, 10 months ago) | 2.0.0 (3 months ago) | 1-major | 1 year, 7 months |
| colorlog | 4.0.2 (1 year, 10 months ago) | 4.4.0 (28 days ago) | 2-minor | 1 year, 9 months |
| beautifulsoup4 | 4.7.1 (1 year, 9 months ago) | 4.9.3 (a month ago) | 2-minor | 1 year, 8 months |
| requests-oauthlib | 1.1.0 (1 year, 9 months ago) | 1.3.0 (a year ago) | 2-minor | 9 months |
| vine | 1.3.0 (1 year, 7 months ago) | 5.0.0 (a month ago) | 1-major | 1 year, 5 months |
| sphinx-autoapi | 1.0.0 (1 year, 6 months ago) | 1.5.1 (a month ago) | 2-minor | 1 year, 5 months |
| azure-mgmt-containerinstance | 1.5.0 (1 year, 5 months ago) | 2.0.0 (4 months ago) | 1-major | 1 year, 1 month |
| bowler | 0.8.0 (1 year, 4 months ago) | 0.9.0 (a month ago) | 2-minor | 1 year, 3 months |
| bcrypt | 3.1.7 (1 year, 4 months ago) | 3.2.0 (2 months ago) | 2-minor | 1 year, 1 month |
| docker | 3.7.3 (1 year, 4 months ago) | 4.3.1 (2 months ago) | 1-major | 1 year, 2 months |
| isort | 4.3.21 (1 year, 4 months ago) | 5.6.4 (23 days ago) | 1-major | 1 year, 3 months |
| toolz | 0.10.0 (1 year, 3 months ago) | 0.11.1 (a month ago) | 2-minor | 1 year, 2 months |
| marshmallow-oneofschema | 2.0.1 (1 year, 3 months ago) | 2.1.0 (30 days ago) | 2-minor | 1 year, 2 months |
| google-cloud-language | 1.3.0 (1 year, 3 months ago) | 2.0.0 (17 days ago) | 1-major | 1 year, 2 months |
| azure-storage-blob | 2.1.0 (1 year, 3 months ago) | 12.5.0 (a month ago) | 1-major | 1 year, 1 month |
| watchtower | 0.7.3 (1 year, 2 months ago) | 1.0.0 (8 days ago) | 1-major | 1 year, 2 months |
| traitlets | 4.3.3 (1 year, 1 month ago) | 5.0.5 (21 days ago) | 1-major | 1 year, 12 days |
| google-auth-oauthlib | 0.4.1 (1 year, 1 month ago) | 0.4.2 (8 days ago) | 3-patch | 1 year, 25 days |
| aiohttp | 3.6.2 (1 year, 28 days ago) | 3.7.2 (9 days ago) | 2-minor | 1 year, 18 days |
| google-cloud-translate | 1.7.0 (1 year, 28 days ago) | 3.0.1 (2 months ago) | 1-major | 10 months |
| mysql-connector-python | 8.0.18 (1 year, 23 days ago) | 8.0.22 (17 days ago) | 3-patch | 1 year, 5 days |
| networkx | 2.4 (1 year, 20 days ago) | 2.5 (2 months ago) | 0-unknown | 10 months |
| elasticsearch-dbapi | 0.1.0 (1 year, 13 days ago) | 0.1.3 (23 days ago) | 3-patch | 11 months |
| black | 19.10b0 (1 year, 9 days ago) | 20.8b1 (2 months ago) | 0-unknown | 9 months |
| ecdsa | 0.14.1 (a year ago) | 0.16.0 (2 months ago) | 2-minor | 9 months |
| moto | 1.3.14 (11 months ago) | 1.3.16 (a month ago) | 3-patch | 9 months |
| cassandra-driver | 3.20.2 (11 months ago) | 3.24.0 (4 months ago) | 2-minor | 6 months |
| gunicorn | 19.10.0 (11 months ago) | 20.0.4 (11 months ago) | 1-major | 3 days |
| colorama | 0.4.3 (10 months ago) | 0.4.4 (23 days ago) | 3-patch | 10 months |
| paramiko | 2.7.1 (10 months ago) | 2.7.2 (2 months ago) | 3-patch | 8 months |
| cattrs | 1.0.0 (10 months ago) | 1.1.1 (6 days ago) | 2-minor | 9 months |
| elasticsearch | 7.5.1 (9 months ago) | 7.9.1 (2 months ago) | 2-minor | 7 months |
| py4j | 0.10.9 (9 months ago) | 0.10.9.1 (a month ago) | 0-unknown | 7 months |
| google-cloud-speech | 1.3.2 (9 months ago) | 2.0.0 (a month ago) | 1-major | 7 months |
| Flask-Babel | 1.0.0 (8 months ago) | 2.0.0 (2 months ago) | 1-major | 6 months |
| freezegun | 0.3.15 (8 months ago) | 1.0.0 (2 months ago) | 1-major | 6 months |
| jwcrypto | 0.7 (8 months ago) | 0.8 (2 months ago) | 0-unknown | 5 months |
| google-cloud-tasks | 1.5.0 (8 months ago) | 2.0.0 (2 months ago) | 1-major | 6 months |
| google-cloud-vision | 1.0.0 (8 months ago) | 2.0.0 (a month ago) | 1-major | 7 months |
| google-cloud-texttospeech | 1.0.1 (8 months ago) | 2.2.0 (2 months ago) | 1-major | 5 months |
| zipp | 3.1.0 (8 months ago) | 3.4.0 (11 days ago) | 2-minor | 7 months |
| tornado | 6.0.4 (8 months ago) | 6.1 (6 days ago) | 0-unknown | 7 months |
| Pygments | 2.6.1 (7 months ago) | 2.7.2 (12 days ago) | 2-minor | 7 months |
| mypy | 0.770 (7 months ago) | 0.790 (27 days ago) | 0-unknown | 6 months |
| kubernetes | 11.0.0 (7 months ago) | 12.0.0 (21 days ago) | 1-major | 7 months |
| pytest-rerunfailures | 9.0 (7 months ago) | 9.1.1 (a month ago) | 0-unknown | 6 months |
| alembic | 1.4.2 (7 months ago) | 1.4.3 (a month ago) | 3-patch | 5 months |
| google-crc32c | 0.1.0 (7 months ago) | 1.0.0 (2 months ago) | 1-major | 4 months |
| pyrsistent | 0.16.0 (7 months ago) | 0.17.3 (a month ago) | 2-minor | 5 months |
| snowflake-sqlalchemy | 1.2.3 (7 months ago) | 1.2.4 (a month ago) | 3-patch | 6 months |
| python-http-client | 3.2.7 (7 months ago) | 3.3.1 (2 months ago) | 2-minor | 4 months |
| typing-extensions | 3.7.4.2 (7 months ago) | 3.7.4.3 (2 months ago) | 0-unknown | 4 months |
| psycopg2-binary | 2.8.5 (6 months ago) | 2.8.6 (a month ago) | 3-patch | 5 months |
| pbr | 5.4.5 (6 months ago) | 5.5.1 (17 days ago) | 2-minor | 6 months |
| zope.interface | 5.1.0 (6 months ago) | 5.2.0 (12 hours ago) | 2-minor | 6 months |
| google-cloud-datacatalog | 0.7.0 (6 months ago) | 2.0.0 (2 months ago) | 1-major | 4 months |
| starkbank-ecdsa | 1.0.0 (6 months ago) | 1.1.0 (2 months ago) | 2-minor | 4 months |
| google-cloud-kms | 1.4.0 (6 months ago) | 2.2.0 (a month ago) | 1-major | 5 months |
| cloudant | 2.13.0 (6 months ago) | 2.14.0 (2 months ago) | 2-minor | 4 months |
| msal-extensions | 0.2.2 (6 months ago) | 0.3.0 (2 months ago) | 2-minor | 4 months |
| pytz | 2020.1 (6 months ago) | 2020.4 (3 days ago) | 0-unknown | 6 months |
| sh | 1.13.1 (6 months ago) | 1.14.1 (12 days ago) | 2-minor | 5 months |
| cloudpickle | 1.4.1 (6 months ago) | 1.6.0 (2 months ago) | 2-minor | 3 months |
| azure-kusto-data | 0.0.45 (6 months ago) | 1.0.3 (25 days ago) | 1-major | 5 months |
| gcsfs | 0.6.2 (5 months ago) | 0.7.1 (a month ago) | 2-minor | 4 months |
| google-cloud-redis | 1.0.0 (5 months ago) | 2.0.0 (a month ago) | 1-major | 4 months |
| toml | 0.10.1 (5 months ago) | 0.10.2 (4 days ago) | 3-patch | 5 months |
| fissix | 20.5.1 (5 months ago) | 20.8.0 (2 months ago) | 2-minor | 3 months |
| pandas-gbq | 0.13.2 (5 months ago) | 0.14.0 (a month ago) | 2-minor | 4 months |
| multidict | 4.7.6 (5 months ago) | 5.0.0 (24 days ago) | 1-major | 4 months |
| Authlib | 0.14.3 (5 months ago) | 0.15.2 (18 days ago) | 2-minor | 4 months |
| email-validator | 1.1.1 (5 months ago) | 1.1.2 (3 hours ago) | 3-patch | 5 months |
| google-cloud-secret-manager | 1.0.0 (5 months ago) | 2.0.0 (a month ago) | 1-major | 3 months |
| marshmallow-sqlalchemy | 0.23.1 (5 months ago) | 0.24.0 (15 days ago) | 2-minor | 4 months |
| spython | 0.0.84 (5 months ago) | 0.0.85 (a month ago) | 3-patch | 3 months |
| nodeenv | 1.4.0 (5 months ago) | 1.5.0 (2 months ago) | 2-minor | 2 months |
| elasticsearch-dsl | 7.2.1 (5 months ago) | 7.3.0 (a month ago) | 2-minor | 3 months |
| qds-sdk | 1.16.0 (5 months ago) | 1.16.1 (30 days ago) | 3-patch | 4 months |
| google-cloud-monitoring | 1.0.0 (5 months ago) | 2.0.0 (30 days ago) | 1-major | 4 months |
| greenlet | 0.4.16 (5 months ago) | 0.4.17 (a month ago) | 3-patch | 3 months |
| adal | 1.2.4 (5 months ago) | 1.2.5 (15 days ago) | 3-patch | 4 months |
| inflection | 0.5.0 (4 months ago) | 0.5.1 (2 months ago) | 3-patch | 2 months |
| apispec | 3.3.1 (4 months ago) | 4.0.0 (a month ago) | 1-major | 3 months |
| pylint | 2.5.3 (4 months ago) | 2.6.0 (2 months ago) | 2-minor | 2 months |
| flake8 | 3.8.3 (4 months ago) | 3.8.4 (a month ago) | 3-patch | 3 months |
| nbformat | 5.0.7 (4 months ago) | 5.0.8 (22 days ago) | 3-patch | 4 months |
| google-cloud-videointelligence | 1.15.0 (4 months ago) | 1.16.0 (a month ago) | 2-minor | 3 months |
| google-cloud-dlp | 1.0.0 (4 months ago) | 2.0.0 (2 months ago) | 1-major | 2 months |
| pytest-cov | 2.10.0 (4 months ago) | 2.10.1 (2 months ago) | 3-patch | 2 months |
| more-itertools | 8.4.0 (4 months ago) | 8.6.0 (6 days ago) | 2-minor | 4 months |
| dill | 0.3.2 (4 months ago) | 0.3.3 (4 days ago) | 3-patch | 4 months |
| semver | 2.10.2 (4 months ago) | 2.13.0 (16 days ago) | 2-minor | 4 months |
| pyspark | 3.0.0 (4 months ago) | 3.0.1 (a month ago) | 3-patch | 2 months |
| gevent | 20.6.2 (4 months ago) | 20.9.0 (a month ago) | 2-minor | 3 months |
| google-cloud-container | 1.0.1 (4 months ago) | 2.1.0 (a month ago) | 1-major | 2 months |
| google-cloud-automl | 1.0.1 (4 months ago) | 2.1.0 (9 days ago) | 1-major | 4 months |
| croniter | 0.3.34 (4 months ago) | 0.3.36 (3 days ago) | 3-patch | 4 months |
| distributed | 2.19.0 (4 months ago) | 2.30.1 (a day ago) | 2-minor | 4 months |
| ipdb | 0.13.3 (4 months ago) | 0.13.4 (a month ago) | 3-patch | 3 months |
| kombu | 4.6.11 (4 months ago) | 5.0.2 (a month ago) | 1-major | 2 months |
| pycryptodomex | 3.9.8 (4 months ago) | 3.9.9 (3 days ago) | 3-patch | 4 months |
| google-cloud-spanner | 1.17.1 (4 months ago) | 1.19.1 (23 days ago) | 2-minor | 3 months |
| SQLAlchemy | 1.3.18 (4 months ago) | 1.3.20 (24 days ago) | 3-patch | 3 months |
| cx-Oracle | 8.0.0 (4 months ago) | 8.0.1 (2 months ago) | 3-patch | 2 months |
| importlib-metadata | 1.7.0 (4 months ago) | 2.0.0 (a month ago) | 1-major | 2 months |
| papermill | 2.1.2 (4 months ago) | 2.2.2 (2 days ago) | 2-minor | 4 months |
| google-cloud-bigquery-datatransfer | 1.1.0 (4 months ago) | 2.1.0 (a month ago) | 1-major | 2 months |
| pre-commit | 2.6.0 (4 months ago) | 2.8.2 (6 days ago) | 2-minor | 3 months |
| datadog | 0.38.0 (4 months ago) | 0.39.0 (2 months ago) | 2-minor | a month |
| azure-cosmos | 3.2.0 (3 months ago) | 4.2.0 (27 days ago) | 1-major | 3 months |
| nbclient | 0.4.1 (3 months ago) | 0.5.1 (21 days ago) | 2-minor | 3 months |
| tableauserverclient | 0.12 (3 months ago) | 0.13 (a month ago) | 0-unknown | a month |
| argcomplete | 1.12.0 (3 months ago) | 1.12.1 (a month ago) | 3-patch | 2 months |
| GitPython | 3.1.7 (3 months ago) | 3.1.11 (13 days ago) | 3-patch | 3 months |
| graphviz | 0.14.1 (3 months ago) | 0.14.2 (29 days ago) | 3-patch | 2 months |
| google-cloud-pubsub | 1.7.0 (3 months ago) | 2.1.0 (a month ago) | 1-major | 2 months |
| jupyter-client | 6.1.6 (3 months ago) | 6.1.7 (2 months ago) | 3-patch | a month |
| regex | 2020.7.14 (3 months ago) | 2020.10.28 (8 days ago) | 2-minor | 3 months |
| psutil | 5.7.2 (3 months ago) | 5.7.3 (12 days ago) | 3-patch | 3 months |
| google-ads | 6.0.0 (3 months ago) | 7.0.0 (2 months ago) | 1-major | a month |
| google-api-python-client | 1.10.0 (3 months ago) | 1.12.5 (14 days ago) | 2-minor | 3 months |
| yamllint | 1.24.2 (3 months ago) | 1.25.0 (a month ago) | 2-minor | 2 months |
| portalocker | 1.7.1 (3 months ago) | 2.0.0 (3 months ago) | 1-major | 16 days |
| nest-asyncio | 1.4.0 (3 months ago) | 1.4.2 (11 days ago) | 3-patch | 3 months |
| marshmallow | 3.7.1 (3 months ago) | 3.9.0 (5 days ago) | 2-minor | 3 months |
| cryptography | 3.0 (3 months ago) | 3.2.1 (8 days ago) | 0-unknown | 3 months |
| numpy | 1.19.1 (3 months ago) | 1.19.4 (3 days ago) | 3-patch | 3 months |
| identify | 1.4.25 (3 months ago) | 1.5.9 (2 days ago) | 2-minor | 3 months |
| urllib3 | 1.25.10 (3 months ago) | 1.25.11 (17 days ago) | 3-patch | 2 months |
| google-cloud-bigtable | 1.4.0 (3 months ago) | 1.5.1 (30 days ago) | 2-minor | 2 months |
| parso | 0.7.1 (3 months ago) | 0.8.0 (3 months ago) | 2-minor | 11 days |
| coverage | 5.2.1 (3 months ago) | 5.3 (a month ago) | 0-unknown | a month |
| pyarrow | 1.0.0 (3 months ago) | 2.0.0 (17 days ago) | 1-major | 2 months |
| google-cloud-storage | 1.30.0 (3 months ago) | 1.32.0 (17 days ago) | 2-minor | 2 months |
| sphinx-copybutton | 0.3.0 (3 months ago) | 0.3.1 (4 days ago) | 3-patch | 3 months |
| cffi | 1.14.1 (3 months ago) | 1.14.3 (a month ago) | 3-patch | a month |
| yandexcloud | 0.45.0 (3 months ago) | 0.59.1 (15 hours ago) | 2-minor | 3 months |
| google-cloud-bigquery | 1.26.1 (3 months ago) | 2.3.1 (7 hours ago) | 1-major | 3 months |
| pytest-xdist | 1.34.0 (3 months ago) | 2.1.0 (2 months ago) | 1-major | 28 days |
| msrest | 0.6.18 (3 months ago) | 0.6.19 (a month ago) | 3-patch | a month |
| pandas | 1.1.0 (3 months ago) | 1.1.4 (6 days ago) | 3-patch | 3 months |
| vertica-python | 0.11.0 (3 months ago) | 1.0.0 (a month ago) | 1-major | a month |
| protobuf | 3.12.4 (3 months ago) | 3.13.0 (2 months ago) | 2-minor | 16 days |
| fastavro | 0.24.0 (3 months ago) | 1.1.0 (6 days ago) | 1-major | 3 months |
| pytest | 6.0.1 (3 months ago) | 6.1.2 (8 days ago) | 2-minor | 2 months |
| eventlet | 0.26.1 (3 months ago) | 0.29.1 (14 days ago) | 2-minor | 2 months |
| iniconfig | 1.0.1 (3 months ago) | 1.1.1 (20 days ago) | 2-minor | 2 months |
| amqp | 2.6.1 (3 months ago) | 5.0.1 (a month ago) | 1-major | a month |
| celery | 4.4.7 (3 months ago) | 5.0.2 (3 days ago) | 1-major | 3 months |
| fsspec | 0.8.0 (3 months ago) | 0.8.4 (22 days ago) | 3-patch | 2 months |
| dask | 2.22.0 (3 months ago) | 2.30.0 (30 days ago) | 2-minor | 2 months |
| ipython | 7.17.0 (3 months ago) | 7.19.0 (6 days ago) | 2-minor | 2 months |
| yarl | 1.5.1 (3 months ago) | 1.6.2 (24 days ago) | 2-minor | 2 months |
| mongomock | 3.20.0 (3 months ago) | 3.21.0 (17 days ago) | 2-minor | 2 months |
| tqdm | 4.48.2 (3 months ago) | 4.51.0 (11 days ago) | 2-minor | 2 months |
| cfn-lint | 0.34.1 (3 months ago) | 0.40.0 (a day ago) | 2-minor | 3 months |
| snowflake-connector-python | 2.2.10 (3 months ago) | 2.3.5 (2 days ago) | 2-minor | 2 months |
| azure-mgmt-resource | 10.2.0 (3 months ago) | 15.0.0 (a month ago) | 1-major | a month |
| virtualenv | 20.0.30 (3 months ago) | 20.1.0 (11 days ago) | 2-minor | 2 months |
| facebook-business | 8.0.0 (3 months ago) | 8.0.5 (a month ago) | 3-patch | a month |
| grpcio | 1.31.0 (3 months ago) | 1.33.2 (8 days ago) | 2-minor | 2 months |
| sendgrid | 6.4.5 (3 months ago) | 6.4.7 (a month ago) | 3-patch | a month |
| azure-datalake-store | 0.0.49 (3 months ago) | 0.0.51 (21 days ago) | 3-patch | 2 months |
| google-cloud-core | 1.4.1 (2 months ago) | 1.4.3 (30 days ago) | 3-patch | 2 months |
| google-resumable-media | 0.7.1 (2 months ago) | 1.1.0 (a month ago) | 1-major | a month |
| slackclient | 2.8.0 (2 months ago) | 2.9.3 (15 days ago) | 2-minor | 2 months |
| google-auth | 1.20.1 (2 months ago) | 1.23.0 (7 days ago) | 2-minor | 2 months |
| JPype1 | 1.0.2 (2 months ago) | 1.1.2 (13 days ago) | 2-minor | 2 months |
| Sphinx | 3.2.0 (2 months ago) | 3.3.0 (3 days ago) | 2-minor | 2 months |
| ldap3 | 2.8 (2 months ago) | 2.8.1 (a month ago) | 0-unknown | 29 days |
| prompt-toolkit | 3.0.6 (2 months ago) | 3.0.8 (24 days ago) | 3-patch | 2 months |
| google-cloud-dataproc | 1.1.1 (2 months ago) | 2.0.2 (a month ago) | 1-major | a month |
| azure-identity | 1.4.0 (2 months ago) | 1.4.1 (29 days ago) | 3-patch | a month |
| responses | 0.10.16 (2 months ago) | 0.12.0 (2 months ago) | 2-minor | 18 days |
| pyexasol | 0.14.1 (2 months ago) | 0.14.2 (3 days ago) | 3-patch | 2 months |
| azure-keyvault-keys | 4.2.0 (2 months ago) | 4.3.0 (30 days ago) | 2-minor | a month |
| aws-sam-translator | 1.26.0 (2 months ago) | 1.29.0 (2 hours ago) | 2-minor | 2 months |
| sphinxcontrib-spelling | 5.2.1 (2 months ago) | 7.1.0 (9 hours ago) | 1-major | 2 months |
| google-api-core | 1.22.1 (2 months ago) | 1.23.0 (16 days ago) | 2-minor | 2 months |
| botocore | 1.17.41 (2 months ago) | 1.19.12 (4 hours ago) | 2-minor | 2 months |
| boto3 | 1.14.41 (2 months ago) | 1.16.12 (4 hours ago) | 2-minor | 2 months |
| sentry-sdk | 0.16.4 (2 months ago) | 0.19.2 (3 days ago) | 2-minor | 2 months |
| humanize | 2.6.0 (2 months ago) | 3.1.0 (17 days ago) | 1-major | 2 months |
| arrow | 0.16.0 (2 months ago) | 0.17.0 (30 days ago) | 2-minor | a month |
| msal | 1.5.0 (2 months ago) | 1.6.0 (3 days ago) | 2-minor | a month |
| testfixtures | 6.14.2 (2 months ago) | 6.15.0 (27 days ago) | 2-minor | a month |
| attrs | 20.2.0 (2 months ago) | 20.3.0 (15 hours ago) | 2-minor | a month |
| azure-core | 1.8.1 (a month ago) | 1.8.2 (a month ago) | 3-patch | 27 days |
| sshtunnel | 0.1.5 (a month ago) | 0.2.2 (5 days ago) | 2-minor | a month |

I generated this table with the script:
https://gist.github.com/mik-laj/880b07bfbdbd5c65b4b2260f6c0fee72

CC: @potiuk @ryw

dependencies bug security

Source

mik-laj

👀1 ❤1

All 6 comments

I will change the title of the issue, because it is not the matter of constraints, but it's the matter of limitations in setup.py and transitional dependencies that we have for all those libraries.

There is nothing we can do on the constraint level. The constraints files are generated automatically by PIP dependency mechanisms - whatever PIP resolves using setup.py limits is automatically updated as new version of constraints.

So if we want to do anything about it (do we?) we should remove some of the limitations there and upgrade all the different providers/core dependencies we have to use the latest version of dependent libraries - basically for each provider separately. Even that will not help in some cases because the newest version of those libraries might transitively use some older versions of dependent libraries.

Does anyone have some proposal there? Should we somehow make an effort to upgrade those? Maybe there is someone who would like o lead that?

Note, that in order to do it, we likely need to have some system tests in place and implemented for those providers that we decide to bump to later version of dependent libraries (https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+Automation+of+System+Tests+for+external+systems) because what we basically need to do is to upgrade the libraries that we already know usually that they have some compatibility issues (for example all the google providers will have to be sooner or later migrated to >2.0.0 python libraries and automated unit testing is not enough for those kinds of changes (those libraries are not backwards compatible).

WDYT others? Is it worth to make such a concerted effort? Are there some real benefits from that? Should we do it? For 2.0 or later?

potiuk on 6 Nov 2020

BTW. If somebody want to see where the deps come from, it's easy to use pipdeptree:

pipdeptree.3.8.txt

potiuk on 6 Nov 2020

👍1

I agree with Jarek, that some of the constraints come from dependencies of Airflow's dependencies. And since we have lots of dependencies it is not always straightforward to support all versions as these dependencies define there own requirements of versions.

So while having latest and greatest would be awesome, we need to make a concise effort that we don't make Airflow incompatible with other dependencies since PIP resolver is not going to allow it any longer

kaxil on 6 Nov 2020

There is even a conflict in Airflow 1.10.13 with importlib-metadata, because Airflow requires importlib-metadata~=2.0, while argcomplete (required at 1.12.0 by Airflow) requires importlib-metadata<2,>=0.23.