pipenv cannot install psycopg2-binary, but pip can (py 3.8)

[kristjanvalur]

Issue description

I'm trying to bring a project using requirements.txt into using Pipfile. I am stumped by the
fact that pipfile cannot install psycopg2-binary, but pip can. I'm using windows, and developers will not generally be simply able to build psycopg2 from sources.

Problem:

Create a pipenv using Python 3.8
calling pipfile install psycopg2-binary

Expected result

I'll expect it to be installed to the virtualenv.

Actual result

We get the error

 raise InstallationError(
pipenv.patched.notpip._internal.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in C:\Users\kristjan\AppData\Local\Temp\tmp1wcw4j8zbuild\psycopg2-binary\

workaround:

calling pipenv run pip install psycopg2-binary results in the package being installed.
But this is not adequate, because it means that Pipfile and Pipfile.lock can not be used to deploy the application any more.

Steps to replicate

On windows:

pipenv --python 3.8
pipenv install psycopg2-binary

Other

I am aware that psycopg2-binary is not considered optimal for production. However, for development on windows that is the only practical way to bring the driver in for regular users. I would be happy to have a separate requirement if running on linux with suitable prerequisites installed.

[antnieszka]

It may be something with python3.8 & pipenv - I had the same issue few days ago.

[kristjanvalur]

Interesting, did you find a resolution?
Meanwhile, I can confirm that the problem disappears if I use 3.7

[antnieszka]

No. I just used pip install.
But interestingly it worked without issue on Linux (tested Manjaro and openSUSE), so it may be windows-only issue 😞

[joaomarccos]

Same problem here using linux.

Only works when I use --sequential flag

[EdinsonRequena]

Same problem with python 3.7

[yelled1]

"psycopg2-binary" = "*"
This works for me in Windows 10:1909 w/ 3.7.x as well as Linux(ubuntu).
Close time!

[Levadi]

I am new to programming and I don't quite understand what do you mean by:

"psycopg2-binary" = "*"

[kristjanvalur]

It's been established that it works for python 3.7. It is 3.8 that is the problem and I just confirmed that it is still a problem.

[yelled1]

Oh, this does NOT work even inside the Pipfile? I first thought that @Levadi did not understand that this should be done inside Pipfile & installed with:
$ pipenv install --dev (optional dev)
Also, @EdinsonRequena mentioned that this was problem even on 3.7. May be everytone should post minor versions as well. 3.6.5, 3.7.5 & 3.7.6 worked.
I also executed

$ pipenv check 
$ cd ~/.pyenv
$ git pull 

:/tmp/py3_8$ pipenv install --dev
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
✔ Success!
Updated Pipfile.lock (996803)!
Installing dependencies from Pipfile.lock (996803)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:01
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

My python below
``` $ python -V
Python 3.8.2

Linux mememe 4.15.0-1063-aws #67-Ubuntu SMP Mon Mar 2 07:24:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

My Pipfile below

:/tmp/py3_8$ cat Pipfile
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
"psycopg2-binary" = "*"

[requires]
python_version = "3.8"
```

[antnieszka]

@yelled1 - do you have postgres development packages installed?

[yelled1]

@yelled1 - do you have postgres development packages installed?
@antnieszka You mean in Pipfile? Yes, as belows:

[packages]
pandas = "*"
numpy = "*"
peewee = "*"
peewee_mssql = "*"
pyodbc = "*"
python-dotenv = "*"
argparse = "*"
tqdm = "*"
jinja2 = "*"
"psycopg2-binary" = "*"
tabulate = "*"

but nowadays I am slowly trying out Poetry, which more like this:
psycopg2-binary = "^2.8.5"

If you meant by Qry & looking at the data? Dbeaver all the way, as I hate PGtools & I only access data -> drivers only.
Also, using VScode with SqlTools.

[antnieszka]

@yelled1 I mean system-wide installation of postgres tools - like the postgresql-devel, postgresql-contrib. E.g.:
https://www.postgresql.org/download/linux/redhat/

...
postgresql-contrib | additional supplied modules libraries and headers for C language development
postgresql-devel | libraries and headers for C language development

[yelled1]

@antnieszka in Ubuntu this was not a piece of cake but still fairly easy. Sorry, don't know much about Redhat/CentOs. I don't like'em & avoid it as much as I can. Package issue can be murderous. Also, wrong place to ask. I know, the feeling of deadlines & pressure, but if I ping-pong this in a wrong forum then others may gang up on both of us.

BTW: If anyone hopes that I would test in 3.8 w/ Windows, then u will be disappointed. There is no real justification for using the latest/greatest, unless you are into hippos :=
Never touched any version with minor less than x.x.5, especially in troublesome Windows.
I also cross develop in Win 10 & Ubuntu. So, this is from someone who had a lot more hair because of it.
Also, Pyenv-Win despite best efforts, can't really let you go back Window Python versions.

[antnieszka]

@yelled1 nvm :)

@kristjanvalur
I have a new snippet which fails:

pipenv --python 3.8
pipenv install "psycopg2-binary==2.7.7"

This one works fine without errors for me:

pipenv --python 3.8
pipenv install psycopg2-binary

So maybe when this issue was filled, there was no psycopg2-binary wheel for python 3.8? As there is no python3.8 wheel for psycopg2-biary==2.7.7? https://pypi.org/project/psycopg2-binary/2.7.7/#files (no 2.7.7 wheel for py3.8)

As for TODAY, if you have no cached psycopg2-binary, python 3.8 + psycopg2-binary>=2.8.4 should be fine: https://pypi.org/project/psycopg2-binary/2.8.4/#files

Edit: Note that windows wheel have later date (Nov 10, 2019) than many-linux ones.

[antnieszka]

You may ask - why pip worked when pipenv didn't. I think, and prove me wrong, pipenv has separate cache folder. So pip always got the newest version.

[techalchemy]

For anyone continuing to run into this issue, go ahead and try using the new prerelease of pipenv (i.e. pip install --upgrade --pre pipenv) & make sure you have postgres client binaries installed (i.e. via sudo apt install libpq-dev on ubuntu/debian for the minimal headers). I confirmed this on a fresh container and could not reproduce the issue with the headers installed. Without the headers, I saw a rather long error message, at the top of which I was instructed to install them.

Closing for now, please file a new issue if the pre-release + installing appropriate client binaries or headers does not resolve this. Thanks!

[techalchemy]

You may ask - why pip worked when pipenv didn't. I think, and prove me wrong, pipenv has separate cache folder. So pip always got the newest version.

For reference, this is correct -- pipenv lock --clear clears the cache as needed

[xlanor]

@techalchemy , thanks! installing the required headers on macOS was enough to get me past this problem.

For macOS, simply running brew install postgresql was sufficient to grab the required headers.

[fallenreaper]

I was running into this issue on my Mac, with regards to pipenv. Postgres works just fine normally, but using pipenv will throw an error.

• brew list shows postgres
• pipenv --version shows: pipenv, version 2018.11.26
• Pipefile shows:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
django = "*"
pyyaml = "*"
djangorestframework = "*"
django-cors-headers = "*"
psycopg2 = "*"
pillow = "*"
whitenoise = "*"

[requires]
python_version = "3.7"

• OS: Catalina 10.15.6

I have been trying to figure out why it is acting the way it is. I was also trying by adding psycopg2-binary but that seems to fail as well. It seems that the psycopg2 is causing a lot of issues because when I attempt to use Django, it will error on the djangorestframework etc.

Not sure what the next steps are other than creating a docker container and running everything from within a CENTOS machine, but i am not sure that is the right way.