Tfx: Is max_threads = 1 required for Airflow on SQLite?

[hanneshapke]

I have tested the 0.13.0rc2 on Python 3.7 and it worked nicely. Thank you all for porting TFX to Python 3!

The only major tweak I had to make to get the TFX demo project running was limiting the number of threads to 1 since the demo project is running on SQLite.

With the current settings, the airflow scheduler won't start up and complain that SQLite can't run with parallel connections. As a fix, I added one more line to examples/workshop/setup/setup_demo.sh

sed -i'.orig' 's/max_threads = 2/max_threads = 1/g' ~/airflow/airflow.cfg

to correct the airflow configuration. If anyone runs into the exact same problem, I am happy to create a PR for it. At this point, I am not sure if this is related to my setup (I doubt strongly that it is related to the Python3 version of Airflow, etc.). Anybody seeing the same issue when running the demo pipeline project on Python3?

[ruoyu90]

@hanneshapke could you attach the error log? From my experience it will complain about parallelism but not error out. This is what I got:
[2019-05-17 08:56:15,540] {dag_processing.py:729} ERROR - Cannot use more than 1 thread when using sqlite. Setting parallelism to 1

I was using the original setting of max_thread=2

[krazyhaas]

Closed with https://github.com/tensorflow/tfx/commit/dc9221abbb8dad991d1ae22fb91876da1290efae. Thanks for the suggestion!