Patroni: The release of psycopg2 2.8 breaks pip install of patroni

[ericjperry]

With the release of psycopg2 2.8 the package has been split. If the non binary psycopg2 package is installed, it will attempt to build from source which requires additional dependencies.

Running a command like the following:

pip3 --no-cache-dir install patroni[etcd,kubernetes]==1.5.5

is broken with

Collecting psycopg2>=2.5.4 (from patroni[etcd,kubernetes]==1.5.5)
  Downloading https://files.pythonhosted.org/packages/c7/ca/75236b17f1b951950ffc55d657c5aa408d3d0327a1b6c4c0f7cb16ef7e7b/psycopg2-2.8.tar.gz (367kB)
    Complete output from command python setup.py egg_info:
    running egg_info
    creating pip-egg-info/psycopg2.egg-info
    writing pip-egg-info/psycopg2.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/psycopg2.egg-info/dependency_links.txt
    writing top-level names to pip-egg-info/psycopg2.egg-info/top_level.txt
    writing manifest file 'pip-egg-info/psycopg2.egg-info/SOURCES.txt'

    Error: pg_config executable not found.

    pg_config is required to build psycopg2 from source.  Please add the directory
    containing pg_config to the $PATH or specify the full executable path with the
    option:

        python setup.py build_ext --pg-config /path/to/pg_config build ...

    or with the pg_config option in 'setup.cfg'.

    If you prefer to avoid building psycopg2 from source, please install the PyPI
    'psycopg2-binary' package instead.

    For further information please check the 'doc/src/install.rst' file (also at
    <http://initd.org/psycopg/docs/install.html>).

[bersace]

cf. https://github.com/psycopg/psycopg2/issues/674 The warning has been there for long time. ;-)

You, as a user, needs to install build dependencies or use a distro package.

That would be nice if patroni allows to use psycopg2-binary. But there is no OR dependency in Python world.

[CyberDem0n]

I know about the warning and I didn't believe till the last moment that they will do this stupid thing.
psycopg2 and psycopg2-binary are two different packages which installed into the same place.
Guess what will happen if you install both of them and then remove one? Right, everything that depends on psycopg2 will break, because files are gone.

Putting psycopg2-binary into requirements.txt feels absolutely wrong to me because most of the modern distros already provide python-psycopg2 or python3-psycopg2 packages.

Now I could think only about removing it from requirements.txt at all and documenting that you will have to install it separately and you are free to choose how you are going to do that, either install package from your favorite distro or install it with pip, either psycopg2 or psycopg2-binary.

As a quick workaround to the current problem you can install psycopg2 before installing Patroni, any version starting from 2.5.4 and smaller than 2.8 will work fine.
The second option would be installing all dependencies necessary to build psycopg2 from source.

[bersace]

@CyberDem0n I totally agree with you. In temboard, I opted to remove psycopg2 from setup.py and document user to install psycopg2 the way they prefer. RPM and deb package are fine.

Beware, console scripts checks for dependencies before starting. Thus, if user installs psycopg2-binary, the patroni will complain with:

$ patroni
Traceback (most recent call last):
  File "patroni", line 6, in <module>
    from pkg_resources import load_entry_point
  File ".../lib/python3.7/site-packages/pkg_resources/__init__.py", line 3088, in <module>
    @_call_aside
  File ".../lib/python3.7/site-packages/pkg_resources/__init__.py", line 3072, in _call_aside
    f(*args, **kwargs)
  File ".../lib/python3.7/site-packages/pkg_resources/__init__.py", line 3101, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File ".../lib/python3.7/site-packages/pkg_resources/__init__.py", line 574, in _build_master
    ws.require(__requires__)
  File ".../lib/python3.7/site-packages/pkg_resources/__init__.py", line 892, in require
    needed = self.resolve(parse_requirements(requirements))
  File ".../lib/python3.7/site-packages/pkg_resources/__init__.py", line 778, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'psycopg2<3.0,>=2.7' distribution was not found and is required by patroni
$ pip freeze | grep psycopg2
psycopg2-binary==2.8
$