Docker-airflow: libpq error

[agiz]

I get the following error when building image myself using docker build --rm --build-arg AIRFLOW_DEPS="datadog,dask" --build-arg PYTHON_DEPS="flask_oauthlib>=0.9" -t agiz/docker-airflow:1.10.2 .:

ImportError: libpq.so.5: cannot open shared object file: No such file or directory

[seanjohn7944]

I can confirm this error. When building an image with the provided Dockerfile I see the following error when the image is run:

airflow-worker | [2019-04-08 21:18:19,697] {{settings.py:174}} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=1
airflow-worker | Traceback (most recent call last):
airflow-worker | File "/usr/local/bin/airflow", line 21, in
airflow-worker | from airflow import configuration
airflow-worker | File "/usr/local/lib/python3.7/site-packages/airflow/__init__.py", line 36, in
airflow-worker | from airflow import settings, configuration as conf
airflow-worker | File "/usr/local/lib/python3.7/site-packages/airflow/settings.py", line 266, in
airflow-worker | configure_orm()
airflow-worker | File "/usr/local/lib/python3.7/site-packages/airflow/settings.py", line 188, in configure_orm
airflow-worker | engine = create_engine(SQL_ALCHEMY_CONN, engine_args)
airflow-worker | File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/__init__.py", line 431, in create_engine
airflow-worker | return strategy.create(args, *kwargs)
airflow-worker | File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 87, in create
airflow-worker | dbapi = dialect_cls.dbapi(dbapi_args)
airflow-worker | File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 599, in dbapi
airflow-worker | import psycopg2
airflow-worker | File "/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py", line 50, in
airflow-worker | from psycopg2._psycopg import ( # noqa
airflow-worker | ImportError: libpq.so.5: cannot open shared object file: No such file or directory

[kajari1verma]

I am also getting the same error

[jjunior84]

I´ve just face of the same problem, I did not fix but i could workaround it.
I opened Dockerfile and change AIRFLOW_VERSION arguments from 1.10.2 to 1.10.1
ARG AIRFLOW_VERSION=1.10.1

[bgkelly]

You can also add libpq5 to the build of the underlying image

[kajari1verma]

I added libpq5, still getting same error -> Linux

[kohonen]

same problem here

[yunus89]

Same problem here

[kajari1verma]

By adding below two library problem is resolved on mac but still facing the issue on Linux
'pip install psycopg2 \
&& pip install psycopg2-binary '

[mrhorvath]

I added libpq5 and the issue cleared up. Be mindful not to add it to the buildDeps section, as all of those libraries are removed at the end of the build.

[agiz]

I've made a PR #349.

[stevenmanton]

This worked for me: pip install psycopg2-binary redis>=3.2.0, which you can set with PYTHON_DEPS.

[paulforan]

Wierd this issue just started for me today but not there yesterday....

What worked for me was to add 'psycopg2-binary redis>=3.2.0' to the PYTHON_DEPS in the Dockerfile.
Didnt have todo that yesterday with same source code.... very strange... clearly something outside my control has changed.... wish I knew what....grrrr and lol!

[thierryturpin]

I had the same problem, it's related to the psycopg2 release. Created #370 for it.

[hbeadles]

The way I solved it is by removing the postgres reference in the apache airflow line -- pip install apache-airflow[crypto,celery,hive,jdbc,password]==$AIRFLOW_VERSION , and just installed psycopg2-binary above it

[paulforan]

Thanks all... does anybody know what changed yo cause this reconfiguration
to the dockerfile? Just curious...

On Sat, 25 May 2019, 04:03 Hayden Beadles, [e-mail] wrote:

The way I solved it is by removing the postgres reference in the apache
airflow line -- pip install
apache-airflow[crypto,celery,hive,jdbc,password]==$AIRFLOW_VERSION , and
just installed psycopg2-binary above it

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/puckel/docker-airflow/issues/342?email_source=notifications&email_token=AESM3VGFYTTYTTGIM63NHU3PXCUBLA5CNFSM4HEJK6L2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWG5MCY#issuecomment-495834635,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AESM3VGEX6HBKOFNSJHVOD3PXCUBLANCNFSM4HEJK6LQ
.

[hbeadles]

@paulforan I can take a shot at answering. So from version 2.8 forward, psycopg2 no longer bundles binary packages. It bundles those separately in a new packages - psycopg2-binary. More information on the change can be found here: http://initd.org/psycopg/articles/2018/02/08/psycopg-274-released/. If you included postgres in that apache-airflow pip install line, it installs the newest version of postgres, (2.8.2), which does not have the binary packages included anymore. So to get around that, just remove postgres from there and install psycopg2-binary separately to get those dependencies included. More information can be found here -- https://github.com/puckel/docker-airflow/pull/349

[thierryturpin]

Yes, just like @hbeadles is saying. Also in Airflow in your task log you see: "The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: http://initd.org/psycopg/docs/install.html#binary-install-from-pypi."

[paulforan]

Cool...
Nice answer guys...
So the moral of the story is to fix dependency versions within docker file
as much as possible to avoid stuff from breaking in the future.

Ie to productionize the container building process.
:-)

On Sun, 26 May 2019, 06:27 thierryturpin, [e-mail] wrote:

Yes, just like @hbeadles https://github.com/hbeadles is saying. Also in
Airflow in your task log you see: "The psycopg2 wheel package will be
renamed from release 2.8; in order to keep installing from binary please
use "pip install psycopg2-binary" instead. For details see:
http://initd.org/psycopg/docs/install.html#binary-install-from-pypi."

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/puckel/docker-airflow/issues/342?email_source=notifications&email_token=AESM3VDKEX5YEF7YFDONUS3PXINSTA5CNFSM4HEJK6L2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWH6SXQ#issuecomment-495970654,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AESM3VDLPCQDRDGDYC4UA4DPXINSTANCNFSM4HEJK6LQ
.

[okeefj22]

Adding the above suggestion of fixing psycopg2-binary==2.8.3 previously fixed this for me but I'm now seeing it happen again

[thierryturpin]

Probably another version conflict. In fact on my side I specified all the package versions:
```
&& pip install -U pip==19.0.1 setuptools==40.7.0 wheel==0.32.3 \
&& pip install pytz==2018.9 \
&& pip install pyOpenSSL==19.0.0 \
&& pip install ndg-httpsclient==0.5.1 \
&& pip install pyasn1==0.4.5 \
&& pip install psycopg2==2.7.7 \
````

[KIRY4]

### Same issue. Faced it today.

[2019-07-08 08:29:15,214] {{settings.py:174}} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=1
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 21, in
from airflow import configuration
File "/usr/local/lib/python3.6/site-packages/airflow/__init__.py", line 36, in
from airflow import settings, configuration as conf
File "/usr/local/lib/python3.6/site-packages/airflow/settings.py", line 266, in
configure_orm()
File "/usr/local/lib/python3.6/site-packages/airflow/settings.py", line 188, in configure_orm
engine = create_engine(SQL_ALCHEMY_CONN, engine_args)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/__init__.py", line 443, in create_engine
return strategy.create(args, *kwargs)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 87, in create
dbapi = dialect_cls.dbapi(dbapi_args)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 599, in dbapi
import psycopg2
File "/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py", line 50, in
from psycopg2._psycopg import ( # noqa
ImportError: libpq.so.5: cannot open shared object file: No such file or directory

==========
&& pip install psycopg2==2.7.7 \
This fix helps me.

[arunkollan]

I had a similar issues. I updated the docker file in the airflow directory
FROM puckel/docker-airflow:1.10.2
to
FROM puckel/docker-airflow:1.10.3 ..

About psycopg, the system messages from ariflow worker_1 did complain

/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: http://initd.org/psycopg/docs/install.html#binary-install-from-pypi.

Ignoring for now..

[mdrijwan123]

@paulforan I can take a shot at answering. So from version 2.8 forward, psycopg2 no longer bundles binary packages. It bundles those separately in a new packages - psycopg2-binary. More information on the change can be found here: http://initd.org/psycopg/articles/2018/02/08/psycopg-274-released/. If you included postgres in that apache-airflow pip install line, it installs the newest version of postgres, (2.8.2), which does not have the binary packages included anymore. So to get around that, just remove postgres from there and install psycopg2-binary separately to get those dependencies included. More information can be found here -- #349

Yes This worked in my case, Thanks

[imsrgadich]

if someone is using kube-airflow. Use imageTag as latest in airflow/values.yaml. It solved me the issue.