Airflow: Airflow fails to initdb with cattrs 1.1.0

[tdbabas]

Apache Airflow version:

1.10.12

Kubernetes version (if you are using kubernetes) (use kubectl version):

N/A

Environment:

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): Centos 7
• Kernel (e.g. uname -a): 3.10.0-229.el7.x86_64
• Install tools: pip 9.0.3 from /usr/lib/python3.6/site-packages (python 3.6)
• Others:

What happened:

Following the instructions here, I encountered an issue at the airflow initdb stage:

[xxx@xxx ~]# airflow initdb
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 26, in <module>
    from airflow.bin.cli import CLIFactory
  File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 94, in <module>
    api_module = import_module(conf.get('cli', 'api_client'))  # type: Any
  File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/local/lib/python3.6/site-packages/airflow/api/client/local_client.py", line 24, in <module>
    from airflow.api.common.experimental import delete_dag
  File "/usr/local/lib/python3.6/site-packages/airflow/api/common/experimental/delete_dag.py", line 26, in <module>
    from airflow.models.serialized_dag import SerializedDagModel
  File "/usr/local/lib/python3.6/site-packages/airflow/models/serialized_dag.py", line 35, in <module>
    from airflow.serialization.serialized_objects import SerializedDAG
  File "/usr/local/lib/python3.6/site-packages/airflow/serialization/serialized_objects.py", line 28, in <module>
    import cattr
  File "/usr/local/lib/python3.6/site-packages/cattr/__init__.py", line 1, in <module>
    from .converters import Converter, GenConverter, UnstructureStrategy
  File "/usr/local/lib/python3.6/site-packages/cattr/converters.py", line 16, in <module>
    from attr import fields, resolve_types
ImportError: cannot import name 'resolve_types'

What you expected to happen:

I expected the Airflow DB to be initialised as per the instructions.

How to reproduce it:

On a fresh installation, follow the Quick Start guide until the apache initdb stage.

Anything else we need to know:

Investigation suggests that this was caused by a Python dependency; specifically cattrs==1.1.0, which was released yesterday (2020-10-29). Downgrading cattrs manually to 1.0.0 does fix the issue and allows the Airflow database to be initialised:

[xxx@xxx ~]# pip3 install cattrs==1.0.0
Collecting cattrs==1.0.0
  Downloading https://files.pythonhosted.org/packages/17/5b/6afbdaeb066ecf8ca28d85851048103ac80bb169491a54a14bd39823c422/cattrs-1.0.0-py2.py3-none-any.whl
Requirement already satisfied: attrs>=17.3 in /usr/local/lib/python3.6/site-packages (from cattrs==1.0.0)
Installing collected packages: cattrs
  Found existing installation: cattrs 1.1.0
    Uninstalling cattrs-1.1.0:
      Successfully uninstalled cattrs-1.1.0
Successfully installed cattrs-1.0.0
[xxx@xxx ~]# airflow initdb            
DB: sqlite:////path/to/airflow.db
[2020-10-30 09:36:05,431] {db.py:378} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> e3a246e0dc1, current schema
INFO  [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, create is_encrypted
/usr/local/lib/python3.6/site-packages/alembic/ddl/sqlite.py:44: UserWarning: Skipping unsupported ALTER for creation of implicit constraintPlease refer to the batch mode feature which allows for SQLite migrations using a copy-and-move strategy.
...
Done.

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[cgddrd]

Can confirm we're experiencing this as part of our production stack (auto scaling actions are now failing when trying to install 1.10.12) - a hotfix would be greatly appreciated.

[kaxil]

You can use the following for now, this will be fixed in 1.10.13

pip install apache-airflow==1.10.12 \
 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt"

[cgddrd]

Many thanks @kaxil

[kaxil]

https://github.com/apache/airflow/pull/11969 -- This PR will be included in 1.10.13 and should fix this issue too

[jodmoreira]

You can use the following for now, this will be fixed in 1.10.13

pip install apache-airflow==1.10.12 \
 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt"

It worked! Thanks!

[zachliu]

@jodmoreira oh how i envy you, the constraint file constraints-3.8.txt doesn't work for my 1.10.12 :crying_cat_face: https://github.com/apache/airflow/issues/8467#issuecomment-720712897

[kaxil]

@jodmoreira oh how i envy you, the constraint file constraints-3.8.txt doesn't work for my 1.10.12 😿 #8467 (comment)

Just update both attrs and cattrs that too should work.

or just downgrade cattrs

[potiuk]

Yeah. Constraints file does not work if you already have higher version installed.

[zachliu]

@kaxil @potiuk ah, i'm sorry i didn't describe it clearly :sweat_smile:
the constraint file constraints-3.8.txt did solve the attr and/or cattrs issue, but in order to use it i had to do the following changes due to https://github.com/apache/airflow/issues/8467#issuecomment-720712897:

original:

oauthlib==3.1.0
requests-oauthlib==1.3.0

my updates:

oauthlib==2.1.0
requests-oauthlib==1.1.0

[dark-spark2]

@kaxil can I change attrs to higher version( attrs==20.1.0 ) for airflow 1.10.12?

[canerakin111]

You can use the following for now, this will be fixed in 1.10.13

pip install apache-airflow==1.10.12 \
 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt"

Thanks a lot!

[mmgithub123]

great!
I have the some question，
when i use pip3 install cattrs==1.0.0.
fix my problem

[gaj995]

You can use the following for now, this will be fixed in 1.10.13

pip install apache-airflow==1.10.12 \
 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt"

Didn't work for me :(

[potiuk]

Didn't work for me :(

What do you mean by 'didn't work"? Can you please provide some specifics on what you tried and how it did not work @gaj995 ?

[gaj995]

Didn't work for me :(

What do you mean by 'didn't work"? Can you please provide some specifics on what you tried and how it did not work @gaj995 ?

Here is the error I am getting. I used the pip method that @kaxil posted above.

On Ubantu 20.04.1
Python 3.8.5

`Traceback (most recent call last):
  File "/home/gaj995/.local/bin/airflow", line 25, in <module>
    from airflow.configuration import conf
  File "/home/gaj995/.local/lib/python3.8/site-packages/airflow/__init__.py", line 50, in <module>
    from airflow.models import DAG  # noqa: E402
  File "/home/gaj995/.local/lib/python3.8/site-packages/airflow/models/__init__.py", line 21, in <module>
    from airflow.models.baseoperator import BaseOperator, BaseOperatorLink  # noqa: F401
  File "/home/gaj995/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1116, in <module>
    @attr.s(auto_attribs=True)
AttributeError: module 'attr' has no attribute 's'`

[kaxil]

@gaj995

Since you are using Python 3.8 use the following:

pip install apache-airflow==1.10.12 \
 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.8.txt"

[potiuk]

Good catch @kaxil!

[vinaymor002]

Getting error with python 3.9.0

ERROR: Command errored out with exit status 1: command: /usr/local/opt/[email protected]/bin/python3.9 /usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /var/folders/xf/hwq0q8rn56xcd_830x9q24c40000gn/T/tmpk9hup6wq cwd: /private/var/folders/xf/hwq0q8rn56xcd_830x9q24c40000gn/T/pip-install-9jhr51fl/pandas

Used below command
pip install apache-airflow==1.10.12 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.8.txt"

[potiuk]

Getting error with python 3.9.0

ERROR: Command errored out with exit status 1: command: /usr/local/opt/[email protected]/bin/python3.9 /usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /var/folders/xf/hwq0q8rn56xcd_830x9q24c40000gn/T/tmpk9hup6wq cwd: /private/var/folders/xf/hwq0q8rn56xcd_830x9q24c40000gn/T/pip-install-9jhr51fl/pandas

Used below command
pip install apache-airflow==1.10.12 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.8.txt"

Python 3.9 has known problems with dependencies - we are not supporting it yet. This will happen after 2.0.