Airflow: Celery 5.0

[mik-laj]

Hello,,

The new Celery 5.0 has been released and we may consider migrating to this version. I don't know if it has to happen now, if we should wait for version 5.1 or do it now. I am creating this ticket to track this version.

To process this issue, we should consider the consequences of this update, update the CLI calls, update API calls, and test these versions thoroughly.

@auvipy Can you tell a little more about this release? Is this release stable enough to update our project to these versions? What consequences could this update have?

[wolfrubin]

On this subject. I've been setting up airflow for the first time and I was trying to run the celery worker using airflow worker with Celery 5.0 and have ran into issues that I resolved by downgrading my installed Celery version to 4.4.7. I am running airflow 1.10.12.

As far as I can tell it's to do with the Celery migration to use click for their cli instead of their custom one. I can try and find the stacktrace if you'd like and add it as an issue?

[turbaszek]

As far as I can tell it's to do with the Celery migration to use click for their cli instead of their custom one. I can try and find the stacktrace if you'd like and add it as an issue?

@wolfrubin that would be helpful!

Edit, did quick test:

root@5257c48ec87e:/opt/airflow# airflow celery worker
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 33, in <module>
    sys.exit(load_entry_point('apache-airflow', 'console_scripts', 'airflow')())
  File "/opt/airflow/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/opt/airflow/airflow/cli/cli_parser.py", line 52, in command
    func = import_string(import_path)
  File "/opt/airflow/airflow/utils/module_loading.py", line 32, in import_string
    module = import_module(module_path)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/airflow/airflow/cli/commands/celery_command.py", line 29, in <module>
    from flower.command import FlowerCommand
  File "/usr/local/lib/python3.7/site-packages/flower/command.py", line 17, in <module>
    from celery.bin.base import Command
ImportError: cannot import name 'Command' from 'celery.bin.base' (/usr/local/lib/python3.7/site-packages/celery/bin/base.py)

and after commenting Flower:

root@5257c48ec87e:/opt/airflow# airflow celery worker
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 33, in <module>
    sys.exit(load_entry_point('apache-airflow', 'console_scripts', 'airflow')())
  File "/opt/airflow/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/opt/airflow/airflow/cli/cli_parser.py", line 53, in command
    return func(*args, **kwargs)
  File "/opt/airflow/airflow/utils/cli.py", line 86, in wrapper
    return f(*args, **kwargs)
  File "/opt/airflow/airflow/cli/commands/celery_command.py", line 133, in worker
    worker_instance = worker_bin.worker(app=celery_app)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 781, in main
    with self.make_context(prog_name, args, **extra) as ctx:
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 698, in make_context
    ctx = Context(self, info_name=info_name, parent=parent, **extra)
TypeError: __init__() got an unexpected keyword argument 'app'

[wolfrubin]

@turbaszek you've beaten me to it. The second one was the stack trace I was getting before downgrading to 4.4.7.

[parayaluyanta]

5.0.2 (singularity) did not work for me as well with airflow 1.10.9.

tail -f /var/log/airflow/airflow-worker-stderr---supervisor-josn515o.log 
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 1395, in worker
    worker = worker.worker(app=celery_app)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 781, in main
    with self.make_context(prog_name, args, **extra) as ctx:
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 698, in make_context
    ctx = Context(self, info_name=info_name, parent=parent, **extra)
TypeError: __init__() got an unexpected keyword argument 'app'

Downgrading to 4.4.7 did.

apache-airflow==1.10.9
celery==4.4.7
celery[rabbitmq]==4.4.7

[auvipy]

celery 5.0-5.0.2 has several regressions. should wait for 5.0.3 at least to test