pipenv ignores `PIP_NO_BINARY` and still installs wheel

[sobolevn]

I am trying to run the following command: export PIP_NO_BINARY=1 && pipenv install psycopg2 as it was suggested in https://github.com/pypa/pipenv/issues/1430

But this does not have any effect.

» export PIP_NO_BINARY=1 && pipenv install psycopg2
Installing psycopg2…
Collecting psycopg2
  Using cached psycopg2-2.7.4-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Installing collected packages: psycopg2
Successfully installed psycopg2-2.7.4

Adding psycopg2 to Pipfile's [packages]…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (635dd7)!

Please, note that pip is using cache, let's turn it off.

» PIP_NO_BINARY=1 PIP_NO_CACHE_DIR=off pipenv install psycopg2   
Installing psycopg2…
Collecting psycopg2
  Downloading psycopg2-2.7.4-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (1.7MB)
Installing collected packages: psycopg2
Successfully installed psycopg2-2.7.4

Adding psycopg2 to Pipfile's [packages]…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (635dd7)!

What I expect is that pipenv would install psycopg2 from source. And now it is using wheel.

Describe your environment

1. OS Type: macos
2. Python version: 3.6.4
3. Pipenv version: 9.0.3

Steps to replicate

1. PIP_NO_BINARY=1 PIP_NO_CACHE_DIR=off pipenv install psycopg2

[techalchemy]

I believe I was not thorough in my referencing of the relevant documentation:

Accepts either :all: to disable all binary packages, :none: to empty the set, or one or more package names with commas between them. Note that some packages are tricky to compile and may fail to install when this option is used on them.

So you need to use PIP_NO_BINARY=ALL

[sobolevn]

@techalchemy that does not make any difference.

Things I have tried:

» PIP_NO_CACHE_DIR=off PIP_NO_BINARY="psycopg2" pipenv install psycopg2   
Installing psycopg2…
Collecting psycopg2
  Downloading psycopg2-2.7.4-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (1.7MB)
Installing collected packages: psycopg2
Successfully installed psycopg2-2.7.4

Adding psycopg2 to Pipfile's [packages]…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (635dd7)!

---

» PIP_NO_BINARY=":all:" PIP_NO_CACHE_DIR=off pipenv install psycopg2   
Installing psycopg2…
Collecting psycopg2
  Downloading psycopg2-2.7.4-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (1.7MB)
Installing collected packages: psycopg2
Successfully installed psycopg2-2.7.4

Adding psycopg2 to Pipfile's [packages]…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (635dd7)!

---

» PIP_NO_BINARY=ALL PIP_NO_CACHE_DIR=off pipenv install psycopg2   
Installing psycopg2…
Collecting psycopg2
  Downloading psycopg2-2.7.4-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (1.7MB)
Installing collected packages: psycopg2
Successfully installed psycopg2-2.7.4

Adding psycopg2 to Pipfile's [packages]…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (635dd7)!

[sobolevn]

The same works with pure pip:

» PIP_NO_CACHE_DIR=off PIP_NO_BINARY="psycopg2" pip install psycopg2         
Collecting psycopg2
  Downloading psycopg2-2.7.4.tar.gz (425kB)
    100% |████████████████████████████████| 430kB 1.1MB/s 
Installing collected packages: psycopg2
  Running setup.py install for psycopg2 ... done
Successfully installed psycopg2-2.7.4

[uranusjr]

The same works with pure pip

I might be misunderstanding you, but your examples do not seem to be doing the same thing?

[sobolevn]

@uranusjr don't they?

I thought that these too commands should do the same thing:

» PIP_NO_CACHE_DIR=off PIP_NO_BINARY="psycopg2" pipenv install psycopg2   
Installing psycopg2…
Collecting psycopg2
  Downloading psycopg2-2.7.4-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (1.7MB)
Installing collected packages: psycopg2
Successfully installed psycopg2-2.7.4

Adding psycopg2 to Pipfile's [packages]…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (635dd7)!

---

» PIP_NO_CACHE_DIR=off PIP_NO_BINARY="psycopg2" pip install psycopg2         
Collecting psycopg2
  Downloading psycopg2-2.7.4.tar.gz (425kB)
    100% |████████████████████████████████| 430kB 1.1MB/s 
Installing collected packages: psycopg2
  Running setup.py install for psycopg2 ... done
Successfully installed psycopg2-2.7.4

Note the filenames which are downloaded: psycopg2-2.7.4-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (binary) and psycopg2-2.7.4.tar.gz (source).

Or am I missing something obvious?

[uranusjr]

Ah, I missed your PIP_NO_BINARY="psycopg2" pipenv install example, and though you were comparing PIP_NO_BINARY=ALL pipenv install (which won’t work) and PIP_NO_BINARY=psycopg2 pip install. I’m not quite sure why the environment is not inherited by Pip; this is worth investigating.

[sobolevn]

@techalchemy should we reopen it?

[zxhfirefox]

I have this issue too.
I use pipenv run pip install --no-binary :all: psycopg2 to install, it works.
But psycopg2 is not in Pipfile.

[techalchemy]

@zxhfirefox if you install things via pip they will not end up in your pipfile. You need to use the environment variables described throughout this thread and use pipenv as normal.

[lephuongbg]

Please reopen this. PIP_NO_BINARY=ALL pipenv install doesn't work currently for me too.

[techalchemy]

@herophuong you really need to provide more details than that

[sobolevn]

@techalchemy were my examples helpful? I am not sure about them being fixed.

[lephuongbg]

I didn't think I could provide any more details that are not duplication of the issue owner. If you need some specific details, please ask, I would be more than happy to provide.

[techalchemy]

@herophuong you provided an environment variable and a command you ran with no inputs or outputs — hard to discern anything at all from that. However I would be super interested if both you and @sobolevn could provide the output of PIP_NO_BINARY=:all: pipenv lock —verbose —clear (two dashes, sorry, on mobile) and then pipenv —rm and reinstall with a lockfile present and see if it still uses a wheel.

@sobolevn I think I can see what might be causing this, although in theory it should only matter when generating a lockfile. We turn off compatibility checking and ignore all of the pip related options for generating lockfiles (which sometimes causes problems; see #2078 for example). This is to ensure that if you hand off your lockfile to someone else who doesn’t have that option toggled the lockfile will still contain the relevant hashes. The whole stack is kind of complicated and probably needs to be reworked, but that will take a significant amount of effort.

[OrangeDog]

It's not PIP_NO_BINARY=ALL, it's PIP_NO_BINARY=:all:

[Quidge]

Is there any movement on this? I desperately need to use psycopg2 and I'm experiencing the same issues as @sobolevn . A possible solution I can think of is to install psycopg globally via pip, which allows me to use --no-binary, and then reference that via some site-package dance within my Pipfile. But I can't find a way to reference individual site-packages within environment. I don't want all packages to use site-packages, just psycopg2.

What I was thinking was, inside my Pipfile:

"psycopg" = {version="*", site-package=True}

[sobolevn]

@techalchemy can we please reopen this issue? Or I can create a new one.

[Quidge]

Update: I couldn't find a way to specific a specific package, so I installed psycopg2 with pip globally without the binary ($ pip install --no-binary psycopg2 psycopg2) and left psycopg2 out of my Pipfile. Now, when in the shell, if I open a python interpreter and do

>>> import psycopg2
>>> c = psycopg2.connect('dbname=ltf_db')
>>> <- no Illegal Instruction error

This doesn't fix my problem long term, or make my project very portable for when I'm ready to deploy, but atleast I can continue developing with pyscopg2/postgres on my own machine.

Unless told not to, I'll be spreading this pseudo solution around on SO. I've found a few other frustrated questions there with this exact psycopg problem and I'd like to share.

Another edit: it occurs to me, after reading the Advanced Usage docs, that right up at the top it says

"Dependencies of wheels provided in a Pipfile will not be captured by $ pipenv lock."

This seems particular to this issue so I apologize if I/we're asking for something technically tough.

[techalchemy]

@Quidge what version of pipenv are you using? We should be able to resolve local wheels now. But you can use --site-packages to make site packages available to your virtualenv. We have a lot of improvements coming in the next release, I'd be willing to revisit this issue if we can establish stability. I'm pretty sure supporting this wouldn't be super difficult either.

[Quidge]

Using 9.0.3.

$ pipenv --version
pipenv, version 9.0.3

Using --site-packages does not allow me to make packages available on a case-by-case basis as far as I can tell, which is what I need. My specific issue is that, while I want my project to use only the packages installed to my virtual env, I need to install psycopg from source using Pip, because pipenv does not have a way (that I've found) to force psycopg to install without using a wheel (due to the same issues/needs @sobolevn experienced earlier in this thread). Currently in my Pipfile, there is no mention of psycopg (though maybe I should mention it in a comment). This isn't good. The point of a Pipfile is to detail all dependencies needed to run a project.

[Quidge]

Okay, I've found another way around.

1) Setup a default env in the directory. Pipenv installs pip to that environment.
2) Enter the shell.
3) Within the shell, run pip install as needed, not pipenv install. This will make packages like psycopg2 available to scripts and libraries inside the venv.

Note: Obviously, this does not add psycopg2 to the Pipfile. But it's a halfway concession between --site-packages allowing access to system packages and not having psycopg2 available to any scripts at all.

Below is a log doing the above steps in an ad-hoc, clean dir. If psycopg2 were not available to the local venv, an ImportError would have been raised. It was not. If (on older machines, like mine) psycopg2 were installed, but without the --no-binary option only available with pip, the final line with psycopg2.connect('dbname=<db_name>') would have raised a Illegal instruction: 4 error and the Python interpreter inside the pipenv shell would have crashed (I can't remember if it also crashes the shell or not).

JohnathMacBook3:~ lirum$ mkdir temp
JohnathMacBook3:~ lirum$ cd temp
JohnathMacBook3:temp lirum$ pipenv install
Creating a virtualenv for this project…
⠋Using base prefix '/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/lirum/Envs/temp-bhSkAFEf/bin/python3.6
Also creating executable in /Users/lirum/Envs/temp-bhSkAFEf/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /Users/lirum/Envs/temp-bhSkAFEf
Creating a Pipfile for this project…
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (c23e27)!
Installing dependencies from Pipfile.lock (c23e27)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
To activate this project's virtualenv, run the following:
 $ pipenv shell
JohnathMacBook3:temp lirum$ pipenv shell
Spawning environment shell (/bin/bash). Use 'exit' to leave.
source /Users/lirum/Envs/temp-bhSkAFEf/bin/activate
bash-3.2$ source /Users/lirum/Envs/temp-bhSkAFEf/bin/activate
(temp-bhSkAFEf) bash-3.2$ which pip
/Users/lirum/Envs/temp-bhSkAFEf/bin/pip
(temp-bhSkAFEf) bash-3.2$ pip install --no-binary psycopg2 psycopg2
Collecting psycopg2
  Using cached https://files.pythonhosted.org/packages/74/83/51580322ed0e82cba7ad8e0af590b8fb2cf11bd5aaa1ed872661bd36f462/psycopg2-2.7.4.tar.gz
Skipping bdist_wheel for psycopg2, due to binaries being disabled for it.
Installing collected packages: psycopg2
  Running setup.py install for psycopg2 ... done
Successfully installed psycopg2-2.7.4
(temp-bhSkAFEf) bash-3.2$ python
Python 3.6.5 (default, Jun 12 2018, 12:36:19) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg2
>>> psycopg2
<module 'psycopg2' from '/Users/lirum/Envs/temp-bhSkAFEf/lib/python3.6/site-packages/psycopg2/__init__.py'>
>>> c = psycopg2.connect('dbname=ltf_db')
>>> 

[OrangeDog]

@Quidge have you tried just using the latest version of pipenv (2018.05.18)?
I've had no problems with PIP_NO_BINARY=:all:.

[Quidge]

@OrangeDog , no, I don't think I have tried that yet (I believe my version was downloaded before 5/18). I had tried PIP_NO_BINARY in the past, to the same result as sobolevn. I'm not at my machine right now, but does PIP_NO_BINARY=:<specific_package>: work for you? I need it on a package by package basis.

[OrangeDog]

@Quidge it's a waste of time to reopen a closed ticket before trying the latest version.
If you want specific packages, you don't add :.

[Quidge]

@OrangeDog agreed. That looks pretty great if it works. I'll test tomorrow, thank you. Is there a name for :colon: formatting like that? I've only seen it here and with slack emojis. Is the idea that a package could be named 'all' or 'none' and colons are safe characters to effectively escape 'all' and 'none' options?

[Quidge]

@OrangeDog I updated to ver 2018.05.18 and it works as you said it would. What I've done using this new 2018.05.18 version, is the same series of steps that @sobolevn attempts with a different result.

Below is a log indicating the issues with the PIP_NO_BINARY env var that @sobolevn has experienced no longer seem to be an issue with ver 2018.05.18.

JohnathMacBook3:~ lirum$ mkdir temp
JohnathMacBook3:~ lirum$ cd temp
JohnathMacBook3:temp lirum$ export PIP_NO_BINARY=psycopg2
JohnathMacBook3:temp lirum$ pipenv install
Creating a virtualenv for this project…
Using /usr/local/opt/python/bin/python3.6 (3.6.5) to create virtualenv…
⠋Already using interpreter /usr/local/opt/python/bin/python3.6
Using base prefix '/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/lirum/Envs/temp-bhSkAFEf/bin/python3.6
Also creating executable in /Users/lirum/Envs/temp-bhSkAFEf/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /Users/lirum/Envs/temp-bhSkAFEf
Creating a Pipfile for this project…
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (ca72e7)!
Installing dependencies from Pipfile.lock (ca72e7)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
To activate this project's virtualenv, run the following:
 $ pipenv shell
JohnathMacBook3:temp lirum$ pipenv install psycopg2
Installing psycopg2…
Collecting psycopg2
  Using cached https://files.pythonhosted.org/packages/74/83/51580322ed0e82cba7ad8e0af590b8fb2cf11bd5aaa1ed872661bd36f462/psycopg2-2.7.4.tar.gz
Skipping bdist_wheel for psycopg2, due to binaries being disabled for it.
Installing collected packages: psycopg2
  Running setup.py install for psycopg2: started
    Running setup.py install for psycopg2: finished with status 'done'
Successfully installed psycopg2-2.7.4

Adding psycopg2 to Pipfile's [packages]…
Pipfile.lock (ca72e7) out of date, updating to (3c2775)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (3c2775)!
Installing dependencies from Pipfile.lock (3c2775)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:01
To activate this project's virtualenv, run the following:
 $ pipenv shell
JohnathMacBook3:temp lirum$ pipenv --version
pipenv, version 2018.05.18
JohnathMacBook3:temp lirum$ 

[uranusjr]

The discussion went a bit long, so just to be clear—Can someone confirm the OP’s problem is resolved in 2018.05.18?

Also this probably deserves an entry in the documentation (in the advances page).

[techalchemy]

Looks to be fixed, yay we fixed it (? somehow?)

[techalchemy]

closing due to apparently fixing this, thanks all, let me know if you still have issues

[MMohan1]

Good

On 26-Jun-2018, at 12:10 PM, Dan Ryan [e-mail] wrote:

closing due to apparently fixing this, thanks all, let me know if you still have issues

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/pypa/pipenv/issues/1450#issuecomment-400196455, or mute the thread https://github.com/notifications/unsubscribe-auth/AGEF1mRaYITxPWr40dK9WEafR0Q2NDs_ks5uAddWgaJpZM4SKM1t.