Tfx: Airflow webserver does not work with Python >=3.5

[ianhellstrom]

When starting the Airflow web server at http://localhost:8080/admin/ (inside a virtual environment with Python 3), the following error message occurs:

Traceback (most recent call last):
  File "/Users/ian/taxi_pipeline/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
    cursor, statement, parameters, context
  File "/Users/ian/taxi_pipeline/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: no such column: dag.description

The problem is here:

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such column: dag.description [SQL: 'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners, dag.description AS dag_description, dag.default_view AS dag_default_view, dag.schedule_interval AS dag_schedule_interval \nFROM dag \nWHERE dag.is_subdag = 0 AND dag.is_active = 1 ORDER BY dag.dag_id\n LIMIT ? OFFSET ?'] [parameters: (100, 0)] (Background on this error at: http://sqlalche.me/e/e3q8)

To reproduce, I ran the following commands according to the instructions:

cd ~
rm -rf taxi && rm -rf taxi_pipeline && rm -rf airflow && rm -rf tfx
git clone https://github.com/tensorflow/tfx.git
cd tfx && git checkout tags/0.13.0 -b release

cd ~
virtualenv -p python3.5 taxi_pipeline
source ./taxi_pipeline/bin/activate

export AIRFLOW_HOME=~/airflow
export TAXI_DIR=~/taxi
export TFX_DIR=~/tfx

pip install tensorflow==1.13.1
pip install docker
export SLUGIFY_USES_TEXT_UNIDECODE=yes
pip install apache-airflow
pip install tfx==0.13.0

airflow initdb

cd $TFX_DIR/tfx/examples/chicago_taxi_pipeline

mkdir -p $TAXI_DIR/data/simple
cp data/simple/data.csv $TAXI_DIR/data/simple

mkdir -p $AIRFLOW_HOME/dags/taxi
cp taxi_pipeline_simple.py $AIRFLOW_HOME/dags/taxi

cp taxi_utils.py $TAXI_DIR

airflow webserver

I see the same behaviour for a Python 3.7 virtual environment. Oddly, enough I get the same error now for 2.7!

I assumed it was a version incompatibility with Airflow, so I fixed the version to 1.10.2, but that yields:

Traceback (most recent call last):
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask_admin/base.py", line 69, in inner
    return self._run_view(f, *args, **kwargs)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask_admin/base.py", line 368, in _run_view
    return fn(self, *args, **kwargs)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask_login/utils.py", line 258, in decorated_view
    return func(*args, **kwargs)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/airflow/utils/db.py", line 73, in wrapper
    return func(*args, **kwargs)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/airflow/www/views.py", line 2224, in index
    auto_complete_data=auto_complete_data)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask_admin/base.py", line 308, in render
    return render_template(template, **kwargs)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask/templating.py", line 134, in render_template
    context, ctx.app)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask/templating.py", line 116, in _render
    rv = template.render(context)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/airflow/www/templates/airflow/dags.html", line 18, in top-level template code
    {% extends "airflow/master.html" %}
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/airflow/www/templates/airflow/master.html", line 18, in top-level template code
    {% extends "admin/master.html" %}
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/airflow/www/templates/admin/master.html", line 18, in top-level template code
    {% extends 'admin/base.html' %}
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask_admin/templates/bootstrap3/admin/base.html", line 81, in top-level template code
    {% block tail_js %}
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/airflow/www/templates/admin/master.html", line 44, in block "tail_js"
    xhr.setRequestHeader("X-CSRFToken", "{{ csrf_token() }}");
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask_wtf/csrf.py", line 47, in generate_csrf
    setattr(g, field_name, s.dumps(session[field_name]))
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/itsdangerous/serializer.py", line 166, in dumps
    payload = want_bytes(self.dump_payload(obj))
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/itsdangerous/url_safe.py", line 42, in dump_payload
    json = super(URLSafeSerializerMixin, self).dump_payload(obj)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/itsdangerous/serializer.py", line 133, in dump_payload
    return want_bytes(self.serializer.dumps(obj, **self.serializer_kwargs))
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/itsdangerous/_json.py", line 18, in dumps
    return json.dumps(obj, **kwargs)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable

However, with the version fixed it seems to work with Python 3.5, just not 3.6 or 3.7. Since internally we use 3.6+ this means we'd have to downgrade (not recommended) or stick with Python 2.7, which is not ideal as support runs out in 2020.

Is this expected after support for Python 3(.5) was added?

[ianhellstrom]

In fact, I built myself a Docker image with TFX (incl. Airflow, TFMA, GCS, and a few other tools I need) based on Python 3.5. When running the Airflow server and accessing the UI it throws this:

Node: 27b33c83c4bb
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/app/.venv/lib/python3.5/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/src/app/.venv/lib/python3.5/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/src/app/.venv/lib/python3.5/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/src/app/.venv/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/usr/src/app/.venv/lib/python3.5/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/src/app/.venv/lib/python3.5/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/src/app/.venv/lib/python3.5/site-packages/flask_admin/base.py", line 69, in inner
    return self._run_view(f, *args, **kwargs)
  File "/usr/src/app/.venv/lib/python3.5/site-packages/flask_admin/base.py", line 368, in _run_view
    return fn(self, *args, **kwargs)
  File "/usr/src/app/.venv/lib/python3.5/site-packages/flask_login/utils.py", line 258, in decorated_view
    return func(*args, **kwargs)
  File "/usr/src/app/.venv/lib/python3.5/site-packages/airflow/utils/db.py", line 73, in wrapper
    return func(*args, **kwargs)
  File "/usr/src/app/.venv/lib/python3.5/site-packages/airflow/www/views.py", line 2224, in index
    auto_complete_data=auto_complete_data)
  File "/usr/src/app/.venv/lib/python3.5/site-packages/flask_admin/base.py", line 308, in render
    return render_template(template, **kwargs)
  File "/usr/src/app/.venv/lib/python3.5/site-packages/flask/templating.py", line 134, in render_template
    context, ctx.app)
  File "/usr/src/app/.venv/lib/python3.5/site-packages/flask/templating.py", line 116, in _render
    rv = template.render(context)
  File "/usr/src/app/.venv/lib/python3.5/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/src/app/.venv/lib/python3.5/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/src/app/.venv/lib/python3.5/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "/usr/src/app/.venv/lib/python3.5/site-packages/airflow/www/templates/airflow/dags.html", line 18, in top-level template code
    {% extends "airflow/master.html" %}
  File "/usr/src/app/.venv/lib/python3.5/site-packages/airflow/www/templates/airflow/master.html", line 18, in top-level template code
    {% extends "admin/master.html" %}
  File "/usr/src/app/.venv/lib/python3.5/site-packages/airflow/www/templates/admin/master.html", line 18, in top-level template code
    {% extends 'admin/base.html' %}
  File "/usr/src/app/.venv/lib/python3.5/site-packages/flask_admin/templates/bootstrap3/admin/base.html", line 81, in top-level template code
    {% block tail_js %}
  File "/usr/src/app/.venv/lib/python3.5/site-packages/airflow/www/templates/admin/master.html", line 44, in block "tail_js"
    xhr.setRequestHeader("X-CSRFToken", "{{ csrf_token() }}");
  File "/usr/src/app/.venv/lib/python3.5/site-packages/flask_wtf/csrf.py", line 47, in generate_csrf
    setattr(g, field_name, s.dumps(session[field_name]))
  File "/usr/src/app/.venv/lib/python3.5/site-packages/itsdangerous/serializer.py", line 166, in dumps
    payload = want_bytes(self.dump_payload(obj))
  File "/usr/src/app/.venv/lib/python3.5/site-packages/itsdangerous/url_safe.py", line 42, in dump_payload
    json = super(URLSafeSerializerMixin, self).dump_payload(obj)
  File "/usr/src/app/.venv/lib/python3.5/site-packages/itsdangerous/serializer.py", line 133, in dump_payload
    return want_bytes(self.serializer.dumps(obj, **self.serializer_kwargs))
  File "/usr/src/app/.venv/lib/python3.5/site-packages/itsdangerous/_json.py", line 18, in dumps
    return json.dumps(obj, **kwargs)
  File "/usr/local/lib/python3.5/json/__init__.py", line 237, in dumps
    **kw).encode(obj)
  File "/usr/local/lib/python3.5/json/encoder.py", line 198, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.5/json/encoder.py", line 256, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.5/json/encoder.py", line 179, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: b'5e296790e412a20c89fb531868e8622449aadf64' is not JSON serializable

Previously I ran TFX 0.12 and Python 2.7 fine with a similar Docker image.

[zhitaoli]

Hi @ianhellstrom , the issues was not expected, although previous we only continuous testing using python3.5 so issues were not caught.

I'll investigate more on this issue.

To confirm my understanding, python 2.7 and python3.5 have been working, but Airflow webserver on python 3.6 and 3.7 consistently fails, right?

[ianhellstrom]

That is right: 3.6 and 3.7 don't work at all.

I managed to make 3.5 work locally but not inside a Docker image (Ubuntu base with 3.5.7 compiled from source), so that could have been due to missing libraries.

[zhitaoli]

@ianhellstrom Can you share the Dockerfile you used to reproduce the failure?

Also worth mentioning is that in #51 a docker based tutorial is requested. A docker image with necessary dependencies can serve as a good starting point.

[ianhellstrom]

It uses our own company-internal base image, so that won't help anyone. I can have a look if I can easily make it generic, but no promises. I can have a look tomorrow.

[ianhellstrom]

Here's a minimalistic Python 3.5 (Debian) image that reproduces the problem:

FROM debian

WORKDIR /app

ENV PYTHONPATH=/app \
    VIRTUAL_ENV=/app/.venv \
    PATH=/app/.venv/bin:$PATH

RUN apt update && \
    apt install -y python3.5 python3.5-dev python3.5-venv python3-pip

RUN python3.5 -m pip install --upgrade pip setuptools

RUN rm -rf /var/lib/apt/lists/* && python3.5 -m venv --without-pip /app/.venv

# ---

COPY requirements.txt /app/requirements.txt
COPY entrypoint.sh    /app/entrypoint.sh

ENV AIRFLOW_HOME /app/airflow
ENV SLUGIFY_USES_TEXT_UNIDECODE yes
ENV AIRFLOW__CORE__LOAD_EXAMPLES False

RUN pip install -r requirements.txt
RUN airflow initdb

RUN chmod +x /app/entrypoint.sh

VOLUME /app/tfx/metadata
VOLUME /app/tfx/pipelines
VOLUME /app/tfx/serving

EXPOSE 8080

ENTRYPOINT ["/app/entrypoint.sh"]

The # --- indicates where I would normally split the Dockerfile: one for a Debian + Python3.5 base image and one for TFX add-ons. However, to make it easier I combined the two.

The file requirements.txt has the following contents:

--index-url https://pypi.python.org/simple/

tensorflow==1.13.1
tfx==0.13.0
apache-airflow==1.10.2
docker
backports.ssl_match_hostname

And entrypoint.sh is:

#!/bin/bash
nohup airflow webserver > webserver.log 2>&1 &
airflow scheduler

You can simply run it from the code directory as follows:

docker run \
    -v "$(pwd)"/data:/app/airflow/data \
    -v "$(pwd)"/code/pipelines:/app/airflow/dags \
    -v "$(pwd)"/code/model:/app/airflow/plugins \
    -v "$(pwd)"/.tfx/metadata:/app/tfx/metadata \
    -v "$(pwd)"/.tfx/pipelines:/app/tfx/pipelines \
    -v "$(pwd)"/.tfx/serving:/app/tfx/serving \
    --name "TFX0.13_Airflow1.10.2" \
    -p 8080:8080 \
    -d "$IMAGE_NAME"

It assumes the data, pipelines, and module (model) file live in folders as mounted, which is the setup I have locally. This is different from the TFX repo. It also writes the metadata/pipelines/serving results to a hidden local directory .tfx, so I can access it afterwards with notebooks (even when the container is not running). With .gitignore you can ensure these are not checked in.

This setup also has the benefit that any changes made to the module file or the pipelines themselves are reflected in Airflow (as it runs). So, you can just refresh the DAGs and then re-run to see the effect of changes.

[zhitaoli]

Hi @ianhellstrom, thanks for the dockerfile. I copied this locally but still cannot reproduce: the chicago taxi simple pipeline works okay to run inside a container image built from identical Dockerfile content.

Do I need to upgrade to python3.6 in the image to reproduce?

[ianhellstrom]

Huh. That causes the crash log I included on my machine. Do you run the DAG from command line or through the Airflow UI?

I may have to check whether I did something wrong on my computer.

[ianhellstrom]

I should have checked the webserver.log file from Airflow earlier: somehow localhost:8080 does not work anymore but 0.0.0.0:8080 does. If I change the URL to that, I can open Airflow and run DAGs for Python 3.5.

[zhitaoli]

@ianhellstrom Thanks. Do you have a reproduce with python 3.6/3.7?

[ianhellstrom]

I have not checked it and since it's not officially supported anyway (due to Beam) I do not see an immediate. We can probably close this issue.

[zhitaoli]

Just a note in case anyone ran into this issue: if you see an error related to "TypeError: Object of type bytes is not JSON serializable" and you have ever used more than one python versions w/ airflow, clearing cookie or open the airflow webserver in an incognito page of Chrome may fix the problem.