Pipenv: Pipenv run fails when python3 venv is used

[dmfigol]

pipenv run throws
Warning: There was an unexpected error while activating your virtualenv. Continuing anyway… and uses system Python binary when python3 -m venv .venv is used to create virtual environment.
pipenv shell runs as expected.

$ python -m pipenv.help output

Pipenv version: '2018.05.18'

Pipenv location: '/usr/local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/local/opt/python/bin/python3.6'

Other Python installations in PATH:

• 2.7: /usr/local/bin/python2.7
• 2.7: /usr/local/bin/python2.7
• 2.7: /usr/bin/python2.7
• 3.6: /usr/local/bin/python3.6m

• 3.6: /usr/local/bin/python3.6

• 2.7.15: /usr/local/bin/python

• 2.7.10: /usr/bin/python
• 2.7.15: /usr/local/bin/python2
• 3.6.5: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.5.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.5.0: Fri Apr 13 19:32:32 PDT '
                     '2018; root:xnu-4570.51.2~1/RELEASE_X86_64',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'darwin'}

System environment variables:

• TERM_SESSION_ID
• SSH_AUTH_SOCK
• Apple_PubSub_Socket_Render
• COLORFGBG
• ITERM_PROFILE
• XPC_FLAGS
• PWD
• SHELL
• SECURITYSESSIONID
• LC_CTYPE
• TERM_PROGRAM_VERSION
• TERM_PROGRAM
• PATH
• COLORTERM
• TERM
• HOME
• TMPDIR
• USER
• XPC_SERVICE_NAME
• LOGNAME
• __CF_USER_TEXT_ENCODING
• ITERM_SESSION_ID
• SHLVL
• OLDPWD
• ZSH
• LC_ALL
• LANG
• PAGER
• LESS
• LSCOLORS
• _
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/sbin:/opt/local/bin
• SHELL: /bin/zsh
• LANG: en_US.UTF-8
• PWD: /Users/dmfigol/projects/my/test

---

Contents of Pipfile ('/Users/dmfigol/projects/my/test/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requests = "*"

[dev-packages]

[requires]
python_version = "3.6"

Contents of Pipfile.lock ('/Users/dmfigol/projects/my/test/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "8739d581819011fea34feca8cc077062d6bdfee39c7b37a8ed48c5e0a8b14837"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "certifi": {
            "hashes": [
                "sha256:13e698f54293db9f89122b0581843a782ad0934a4fe0172d2a980ba77fc61bb7",
                "sha256:9fa520c1bacfb634fa7af20a76bcbd3d5fb390481724c597da32c719a7dca4b0"
            ],
            "version": "==2018.4.16"
        },
        "chardet": {
            "hashes": [
                "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae",
                "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691"
            ],
            "version": "==3.0.4"
        },
        "idna": {
            "hashes": [
                "sha256:2c6a5de3089009e3da7c5dde64a141dbc8551d5b7f6cf4ed7c2568d0cc520a8f",
                "sha256:8c7309c718f94b3a625cb648ace320157ad16ff131ae0af362c9f21b80ef6ec4"
            ],
            "version": "==2.6"
        },
        "requests": {
            "hashes": [
                "sha256:6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07e895d1f594193a40a38b",
                "sha256:9c443e7324ba5b85070c4a818ade28bfabedf16ea10206da1132edaa6dda237e"
            ],
            "index": "pypi",
            "version": "==2.18.4"
        },
        "urllib3": {
            "hashes": [
                "sha256:06330f386d6e4b195fbfc736b297f58c5a892e4440e54d294d7004e3a9bbea1b",
                "sha256:cc44da8e1145637334317feebd728bd869a35285b93cbb4cca2577da7e62db4f"
            ],
            "version": "==1.22"
        }
    },
    "develop": {}
}

---

Expected result

pipenv run should not throw an error and use correct python binary

Actual result

dmfigol@mbp [22:50:59] [~/projects/my]
-> % mkdir test
dmfigol@mbp [22:51:03] [~/projects/my]
-> % cd test
dmfigol@mbp [22:51:07] [~/projects/my/test]
-> % python3 -m venv .venv
dmfigol@mbp [22:51:33] [~/projects/my/test]
-> % pipenv install requests
Creating a Pipfile for this project…
Installing requests…
Collecting requests
  Using cached https://files.pythonhosted.org/packages/49/df/50aa1999ab9bde74656c2919d9c0c085fd2b3775fd3eca826012bef76d8c/requests-2.18.4-py2.py3-none-any.whl
Collecting idna<2.7,>=2.5 (from requests)
  Using cached https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl
Collecting urllib3<1.23,>=1.21.1 (from requests)
  Using cached https://files.pythonhosted.org/packages/63/cb/6965947c13a94236f6d4b8223e21beb4d576dc72e8130bd7880f600839b8/urllib3-1.22-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
export PATH="/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/sbin:/opt/local/bin"
  Using cached https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Installing collected packages: idna, urllib3, certifi, chardet, requests
Successfully installed certifi-2018.4.16 chardet-3.0.4 idna-2.6 requests-2.18.4 urllib3-1.22

Adding requests to Pipfile's [packages]…
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (b14837)!
Installing dependencies from Pipfile.lock (b14837)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 5/5 — 00:00:00
To activate this project's virtualenv, run the following:
 $ pipenv shell
dmfigol@mbp [22:52:00] [~/projects/my/test]
-> % pipenv run python3
Warning: There was an unexpected error while activating your virtualenv. Continuing anyway…
Python 3.6.5 (default, Apr 25 2018, 14:23:58)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'requests'
>>> quit()
dmfigol@mbp [22:53:09] [~/projects/my/test]
-> % pipenv shell
Spawning environment shell (/bin/zsh). Use 'exit' to leave.
. /Users/dmfigol/projects/my/test/.venv/bin/activate
dmfigol@mbp [22:57:30] [~/projects/my/test]
-> % . /Users/dmfigol/projects/my/test/.venv/bin/activate
(.venv) dmfigol@mbp [22:57:30] [~/projects/my/test]
-> % python
Python 3.6.5 (default, Apr 25 2018, 14:23:58)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> quit()

Steps to replicate

python3 -m venv .venv
pipenv install requests
pipenv run python3
>>> import requests

[uranusjr]

This fails because venv does not have activate_this.py, which Pipenv uses to find the target command, and initialise the environment for the subprocess to be run. Vinay (the author of venv) isn’t keen on adding it, however. The good news is, activate_this.py actually works perfectly for venvs, so the simplest way to fix this is to find a virtualenv, and copy the file into your venv.

Venv is not (yet?) supported by Pipenv at the moment, so it is probably not the time to solve this. A couple notes on this topic for future reference:

• CPython contains some integration code to venv, so we does not need to do much of what activate_this.py does. Most things “work” directly without any configuration (CPython is smart enough to handle them).
• What we do need to do is to implement the logic to find the correct executable (i.e. manipulate PATH). The logic is already there (the which function), but we need to change the implementation, and still maintain compatibility to virtualenv. It would likely be best to implement this as separate backends of run for venv and virtualenv, like what we already do for Windows and POSIX.

[uranusjr]

Also maybe it would be a good idea to spit out a better error message. The current try-catch-everything block is not very helpful (to end users). Contribute if you can!

[techalchemy]

Don’t work too hard on it, long term plans involve a unifying rewrite of virtualenv and venv

[dmfigol]

@uranusjr
Thanks for explanation.
So much for the "ultimate" packaging tool considering that default python3 mechanism to create virtual environments is not supported (which, by the way, does not copy binaries like virtualenv does)

[techalchemy]

@dmfigol if you can do a better job please PR your cross platform solution in that works on both python 2 and 3. Until then, your unproductive negativity is not welcome. Do not assume we didn’t consider these options before making a decision. If you want to denigrate the tool we build in our free time based on a decision you didnt take the time to understand, you’ll need to find another forum.