Airflow: Airflow v2.0.0b1 package doesnt include "api_connexion/exceptions"

[anirudhbagri]

Apache Airflow version: 2.0.0b1

Kubernetes version (if you are using kubernetes) (use kubectl version): NA

Environment:

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): macOS
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened:
Installed apache-airlfow==2.0.0b1 using pip.
Running airflow webserver gave the following error:

Traceback (most recent call last):
  File "/Users/abagri/Workspace/service-workflows/venv/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/Users/abagri/Workspace/service-workflows/venv/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/Users/abagri/Workspace/service-workflows/venv/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 49, in command
    func = import_string(import_path)
  File "/Users/abagri/Workspace/service-workflows/venv/lib/python3.8/site-packages/airflow/utils/module_loading.py", line 32, in import_string
    module = import_module(module_path)
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/abagri/Workspace/service-workflows/venv/lib/python3.8/site-packages/airflow/cli/commands/webserver_command.py", line 43, in <module>
    from airflow.www.app import cached_app, create_app
  File "/Users/abagri/Workspace/service-workflows/venv/lib/python3.8/site-packages/airflow/www/app.py", line 39, in <module>
    from airflow.www.extensions.init_views import (
  File "/Users/abagri/Workspace/service-workflows/venv/lib/python3.8/site-packages/airflow/www/extensions/init_views.py", line 25, in <module>
    from airflow.api_connexion.exceptions import common_error_handler
ModuleNotFoundError: No module named 'airflow.api_connexion.exceptions'

What you expected to happen:
Expect this command to start the webserver

How to reproduce it:
Install a fresh version of airflow, run airflow db init followed by airflow webserver

Anything else we need to know:

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

cc: @ashb @kaxil -> looking at it

[anirudhbagri]

I downloaded the packages externally from here, this one also doesn't contain anything except the openapi folder.

[kaxil]

I downloaded the packages externally from here, this one also doesn't contain anything except the openapi folder.

What command did you run? pip install apache-airflow==2.0.0b1 or did you download the tar.gz and run pip install FILEPATH_TO_TAR

[anirudhbagri]

I first ran pip install apache-airflow==2.0.0b1 and ran the command (airflow webserver) to get those errors.

Then to confirm, I downloaded the tar to check the contents. (didn't install the tar)

[potiuk]

@ashb @kaxil -> indeed the problem is with implicit packages. The root cause is that api_connexion has no __init__.py

[kaxil]

aah, we just need to add __init__.py file then, wonder why and how it got deleted / missing

[potiuk]

It was never there - but the change with implicit packages, changed find_namespace_packages into find_packages in setup.py and now setup.py simply does not find the package.

[kaxil]

It was there, it got renamed here: https://github.com/apache/airflow/commit/b2a28d1590410630d66966aa1f2b2a049a8c3b32 / https://github.com/apache/airflow/pull/12082

[potiuk]

AAAAAH Github rebase magic finding wrong files!

[potiuk]

Bad me!

[potiuk]

Sounds like b2 then

[ashb]

@potiuk https://github.com/apache/airflow/pull/12082/files#r518885540

You also resolved this discussion of mine. This file rename was the cause of that problem.

[kaxil]

Yeah, let's cut out beta2 today itself

[anirudhbagri]

b2? :(

[ashb]

Fix coming.

[potiuk]

@potiuk https://github.com/apache/airflow/pull/12082/files#r518885540

You also resolved this discussion of mine. This file rename was the cause of that problem.

Yep. Entirely my fault if we are looking for the blame.

[ashb]

I missed it too! It happens, we're just unlucky in the timing

[kaxil]

None of our tests caught it :(

[ashb]

Our tests almost entirely operate on an "editable" install is why :(

[ashb]

I'm going to rename Beta2 milestone to Beta3 (etc).

[potiuk]

Our tests almost entirely operate on an "editable" install is why :(

That's the next step we discussed with Ash that production image should also be build using packages/wheels. I will add it before next beta. it will be rather impossible for CI, but since we have k8s test using the production image and it uses both - scheduler and webserver it will be a very good test.

[potiuk]

I also merged #12231 and will double test it now - also we can release sendgrid 1.0.0b2

[potiuk]

The extras work perfectly from local installation now.

[potiuk]

pip install ".[kubernetes]"

...

Collecting apache-airflow-providers-cncf-kubernetes; extra == "kubernetes"
  Downloading apache_airflow_providers_cncf_kubernetes-1.0.0b1-py3-none-any.whl (32 kB)
...

[ashb]

Sendgrid we can do as a 1.0.0b1 on it's own. I'll do that right now.

[anirudhbagri]

Although I get that __init__.py file was not present in the folder, why did openapi folder got into the package?
was it because openapi did not have __init__.py whereas endpoints and schema had __init__.py inside them?

[ashb]

apache-airflow-2.0.0b1/airflow/api_connexion/
apache-airflow-2.0.0b1/airflow/api_connexion/openapi/
apache-airflow-2.0.0b1/airflow/api_connexion/openapi/v1.yaml

That is there because datafiles are included via a different mechanism to code files.

[kaxil]

Yeah we include openapi files:

https://github.com/apache/airflow/blob/5912d0cae7033b3e2549280677dd60faa53be5e7/MANIFEST.in#L33
https://github.com/apache/airflow/blob/5912d0cae7033b3e2549280677dd60faa53be5e7/setup.py#L916

[anirudhbagri]

Oh, Got it. 👍🏻 Thank you!

[ashb]

Beta 2 had been pushed to pypi