Airflow: Airflow webserver not starting with SQLAlchemy==1.3.16

[CaTaDeVoPs]

Apache Airflow version: 1.10.9
Environment: Ubuntu 18.04 LTS

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):Ubuntu 18.04 LTS

What happened: airflow webserver error

airflow@airflow:~$ airflow webserver
[2020-04-08 09:45:49,843] {settings.py:253} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=30494
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ _ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ ____/____/|__/
[2020-04-08 09:45:50,462] {__init__.py:51} INFO - Using executor LocalExecutor
[2020-04-08 09:45:50,463] {dagbag.py:403} INFO - Filling up the DagBag from /home/airflow/airflow/dags
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 37, in
args.func(args)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/cli.py", line 75, in wrapper
return f(args, *kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/bin/cli.py", line 900, in webserver
app = cached_app_rbac(None) if settings.RBAC else cached_app(None)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/www/app.py", line 233, in cached_app
app = create_app(config, testing)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/www/app.py", line 103, in create_app
models.Chart, Session, name="Charts", category="Data Profiling"))
File "/home/airflow/.local/lib/python3.6/site-packages/flask_admin/contrib/sqla/view.py", line 330, in __init__
menu_icon_value=menu_icon_value)
File "/home/airflow/.local/lib/python3.6/site-packages/flask_admin/model/base.py", line 818, in __init__
self._refresh_cache()
File "/home/airflow/.local/lib/python3.6/site-packages/flask_admin/model/base.py", line 913, in _refresh_cache
self._search_supported = self.init_search()
File "/home/airflow/.local/lib/python3.6/site-packages/flask_admin/contrib/sqla/view.py", line 581, in init_search
if tools.is_hybrid_property(self.model, name):
File "/home/airflow/.local/lib/python3.6/site-packages/flask_admin/contrib/sqla/tools.py", line 209, in is_hybrid_property
return last_name in get_hybrid_properties(last_model)
File "/home/airflow/.local/lib/python3.6/site-packages/flask_admin/contrib/sqla/tools.py", line 190, in get_hybrid_properties
for key, prop in inspect(model).all_orm_descriptors.items()
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/inspection.py", line 72, in inspect
"available for object of type %s" % type_
sqlalchemy.exc.NoInspectionAvailable: No inspection system is available for object of type

What you expected to happen: to start

How to reproduce it:
Install airflow with pip3 and postgres from ubuntu which is 10.

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[marclamberti]

Actually, I got the same error with the SequentialExecutor as well

[matejker]

The same problem here, I'm having apache-airflow==1.10.9 on Ubuntu 18.04, I tried to downgrade to 1.10.8, but the problem remain.

[F4stFr3ddy]

I run airflow==1.10.7 it seems the issue is with a dependency. I tried downgrading SQLAlchemy but it also does not resolve it.

I suspect flask

[CaTaDeVoPs]

flask had a new release 1.1.2 5 days ago

[CaTaDeVoPs]

i tried running airflow 1.9.0 with flask 1.1.1 i had this error:

[2020-04-08 10:10:31,797] {settings.py:253} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=32160
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 26, in
from airflow.bin.cli import CLIFactory
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/bin/cli.py", line 70, in
from airflow.www.app import (cached_app, create_app)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/www/app.py", line 37, in
from airflow.www.blueprints import routes
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/www/blueprints.py", line 25, in
from airflow.www import utils as wwwutils
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/www/utils.py", line 39, in
from flask_admin.model import filters
File "/home/airflow/.local/lib/python3.6/site-packages/flask_admin/model/__init__.py", line 2, in
from .base import BaseModelView
File "/home/airflow/.local/lib/python3.6/site-packages/flask_admin/model/base.py", line 8, in
from werkzeug import secure_filename
ImportError: cannot import name 'secure_filename'

I solve it by doing: pip3 install werkzeug==0.16.0 , after i removed the package but the webserver is not starting , probably more dependencies involved

[CaTaDeVoPs]

3 weeks ago the same setup worked for me. I did some checks and I manage to start it with this configuration:
airflow 1.9.0
flask 1.1.1 - currently 1.1.2
werkzeug -0.16.0 -
sqlalchemy-1.3.13 - currently sqlalchemy-1.3.16

Downgrading sqlalchemy make it work

[marclamberti]

I would not close the issue. It's still a problem, we shouldn't have to downgrade anything to make it works :/

[F4stFr3ddy]

I agree, it results from the dependencies as specified by Airflow requirements. It needs a bug fix.

[F4stFr3ddy]

Besides when running python3.7 and airflow 1.10.7 and sqlalchemy==1.3.13 it still does not work.

[ottensjors]

we're running airflow 1.10.9 and it worked until yesterday. when rebuilding the docker-image today it automatically compiled and installed python 3.7.6 and airflow 1.10.9 with required dependencies.

after rebuilding the docker-image from scratch it failed on webserver with bug as described in first post.

performing an initial comparison with the working docker-container for airflow-webserver i found this:

boto3 1.12.34 --> 1.12.38
botocore 1.15.34 --> 1.15.38
certifi 2019.11.28 --> 2020.4.5.1
cryptography 2.8 --> 2.9
Flask 1.1.1 -> 1.1.2
psycopg2-binary 2.8.4 --> 2.8.5
SQLAlchemy 1.3.15 ---> 1.3.16
typing-extensions 3.7.4.1 --> 3.7.4.2

ill try downgrading the packages to original form and check what package caused the issue.

keep you posted via this bug report

[CaTaDeVoPs]

The problem seems to be SQLAlchemy version 1.3.16 which was released 8 hours ago. i downgraded to 1.3.15 and it works

[CaTaDeVoPs]

My setup is like this:
python 3.6.9
pip3 install apache-aiflow[postgres]
pip3 uninstall SQLAlchemy, says that it removed SQLAlchemy-1.3.16
pip3 install SQLAlchemy=1.3.15

airflow webserver started without errors

[ottensjors]

Can confirm what @catalin0608 says.

just downgraded to SQLAlchemy 1.3.15 for Airflow 1.10.9 running on python 3.7.6. all good.

webserver just starts fine

[F4stFr3ddy]

What causes SQLalchemy to upgrade as its pinned in the requirements? SQLAlchemy==1.3.15

[CaTaDeVoPs]

@marclamberti I closed the ticket by mistake.

[potiuk]

As of last week It is possible to use the requirements.txt as constraint file to install airflow. You can see it here:

https://github.com/apache/airflow/blob/master/INSTALL

and

https://github.com/apache/airflow/blob/master/IMAGES.rst

For example:

pip install apache-airflow==1.10.9 --constraint requirements/requirements-python3.7.txt

Or directly from GitHub:

pip install apache-airflow[gcp]==1.10.9 \
   --constraint https://raw.githubusercontent.com/apache/airflow/v1-10-test/requirements/requirements-python3.7.txt

Note that you have to choose the right python version.

[potiuk]

This should theorethically solve all the problems with installation. Once we release 1.10.10 you will be able to use directly the version of requirements that you install airflow with:

pip install apache-airflow[gcp]==1.10.10 \
   --constraint https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt

But this will only work for 1.10.10+

[marclamberti]

@potiuk That's actually a great news :)

[mik-laj]

We have a solution to the problem, but did anyone create a ticket in the SQLAlchemy project? I'd love to do it, but I'm not deeply familiar with the problem.

[ottensjors]

FYI - my working environment for airflow webserver and corresponding requirements.txt

environment:

python: 3.7.6
airflow: 1.10.9
docker-base-image: openjdk:8-slim-stretch

requirements.txt:

alembic==1.4.2
amqp==2.5.2
apache-airflow==1.10.9
apispec==1.3.3
argcomplete==1.11.1
attrs==19.3.0
Babel==2.8.0
bcrypt==3.1.7
billiard==3.6.3.0
boto3==1.12.34
botocore==1.15.34
cached-property==1.5.1
cattrs==0.9.0
celery==4.4.2
certifi==2019.11.28
cffi==1.14.0
chardet==3.0.4
click==7.1.1
colorama==0.4.3
colorlog==4.0.2
configparser==3.5.3
croniter==0.3.31
cryptography==2.8
defusedxml==0.6.0
dill==0.3.1.1
docutils==0.15.2
Flask==1.1.1
Flask-Admin==1.5.4
Flask-AppBuilder==2.3.1
Flask-Babel==1.0.0
Flask-Bcrypt==0.7.1
Flask-Caching==1.3.3
Flask-JWT-Extended==3.24.1
Flask-Login==0.4.1
Flask-OpenID==1.2.5
Flask-SQLAlchemy==2.4.1
flask-swagger==0.2.13
Flask-WTF==0.14.3
flower==0.9.4
funcsigs==1.0.2
future==0.16.0
graphviz==0.13.2
gunicorn==19.10.0
humanize==0.5.1
idna==2.9
importlib-metadata==1.6.0
iso8601==0.1.12
itsdangerous==1.1.0
Jinja2==2.10.3
jmespath==0.9.5
json-merge-patch==0.2
jsonschema==3.2.0
kombu==4.6.8
lazy-object-proxy==1.4.3
lockfile==0.12.2
Mako==1.1.2
Markdown==2.6.11
MarkupSafe==1.1.1
marshmallow==2.21.0
marshmallow-enum==1.5.1
marshmallow-sqlalchemy==0.22.3
numpy==1.18.2
pandas==0.25.3
paramiko==2.7.1
pendulum==1.4.4
prison==0.1.3
psutil==5.7.0
psycopg2-binary==2.8.4
pyarrow==0.16.0
pycparser==2.20
Pygments==2.6.1
PyJWT==1.7.1
PyNaCl==1.3.0
pyrsistent==0.16.0
pysftp==0.2.9
python-daemon==2.1.2
python-dateutil==2.8.1
python-editor==1.0.4
python3-openid==3.1.0
pytz==2019.3
pytzdata==2019.3
PyYAML==5.3.1
readline==6.2.4.1
redis==3.4.1
requests==2.23.0
s3transfer==0.3.3
setproctitle==1.1.10
six==1.14.0
SQLAlchemy==1.3.15
SQLAlchemy-JSONField==0.9.0
SQLAlchemy-Utils==0.36.3
tabulate==0.8.7
tenacity==4.12.0
termcolor==1.1.0
text-unidecode==1.2
thrift==0.13.0
tornado==5.1.1
typing-extensions==3.7.4.1
tzlocal==1.5.1
unicodecsv==0.14.1
urllib3==1.25.8
vine==1.3.0
Werkzeug==0.16.1
WTForms==2.2.1
xlrd==1.2.0
zipp==3.1.0
zope.deprecation==4.4.0

[marclamberti]

As @potiuk said, the best way is to add the requirement in the pip install command so that the dependencies are frozen.

[potiuk]

Just FYI. Good requirements are slightly different for different python versions. The "official" 1.10. requirements are here: https://github.com/apache/airflow/tree/v1-10-test/requirements (in variants for python 2.7, 3.5, 3.6, 3.7)

[EtsuNDmA]

pinning SQLAlchemy==1.3.15 helps me

[ashb]

It's not clear if this is a bug in SQLA or in Flask-Admin https://github.com/flask-admin/flask-admin/issues/1976

[ashb]

Also: Does this problem only affect people using the "legacy" UI? i.e. if you set AIRFLOW__WEBSERVER__RBAC does this problem still happen?

Confirmed, the new RBAC webserver starts up fine.

[RamanKhakhlou]

Always lock your dependencies up to patch version as a good advice.

[frank0718]

pip3 uninstall SQLAlchemy
pip3 install SQLAlchemy==1.3.15

[potiuk]

@RamanKhakhlou There was a long discussion on it for more than a year and it's not that simple. It's a good advise for apps, but not good advise for libraries (there you should be as open as possible). Airflow is a bit of both that's why we cannot lock the dependencies and we came up with hybrid approach described with the context and details here: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#airflow-dependencies

[ashb]

Fix merged to v1-10-stable, hopefully will be released in 1.10.10.