Pipenv: Something weird with pkg-resources

Created on 12 Jul 2018 · 18Comments · Source: pypa/pipenv

Issue description

pkg-resources in pipenv graph on clean --three Pipenv. After locking and cleaning, pipenv graph raises ModuleNotFoundError

Steps to replicate

me@mypc:/tmp$ mkdir issue
me@mypc:/tmp$ cd issue
me@mypc:/tmp/issue$ pipenv --three
Creating a virtualenv for this project...
Pipfile: /tmp/issue/Pipfile
Using /usr/bin/python3 (3.6.5) to create virtualenv...
⠋Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/me/.local/share/virtualenvs/issue-rZoCsvrN/bin/python3
Also creating executable in /home/me/.local/share/virtualenvs/issue-rZoCsvrN/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
Setting project for issue-rZoCsvrN to /tmp/issue

Virtualenv location: /home/me/.local/share/virtualenvs/issue-rZoCsvrN
Creating a Pipfile for this project...
me@mypc:/tmp/issue$ pipenv graph
pkg-resources==0.0.0

me@mypc:/tmp/issue$ cat Pipfile 
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.6"
me@mypc:/tmp/issue$ pipenv lock
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock (ca72e7)!
me@mypc:/tmp/issue$ pipenv clean
Uninstalling 'pkg-resources'...
me@mypc:/tmp/issue$ pipenv graph

ERROR:  Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pipenv/vendor/pipdeptree.py", line 22, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'

me@mypc:/tmp/issue$

$ pipenv --support

Pipenv version: '2018.7.1'

Pipenv location: '/usr/local/lib/python3.6/dist-packages/pipenv'

Python location: '/usr/bin/python3'

Other Python installations in PATH:

2.7: /usr/bin/python2.7
2.7: /usr/bin/python2.7
3.6: /usr/bin/python3.6m
3.6: /usr/bin/python3.6
2.7.15: /usr/bin/python
2.7.15: /usr/bin/python2
3.6.5: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.15.0-24-generic',
 'platform_system': 'Linux',
 'platform_version': '#26-Ubuntu SMP Wed Jun 13 08:44:47 UTC 2018',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

CLUTTER_IM_MODULE
LS_COLORS
LC_MEASUREMENT
LESSCLOSE
LC_PAPER
LC_MONETARY
XDG_MENU_PREFIX
LANG
DISPLAY
GNOME_SHELL_SESSION_MODE
GPG_TTY
COLORTERM
DESKTOP_AUTOSTART_ID
USERNAME
XDG_VTNR
SSH_AUTH_SOCK
MANDATORY_PATH
LC_NAME
XDG_SESSION_ID
USER
DESKTOP_SESSION
QT4_IM_MODULE
TEXTDOMAINDIR
GNOME_TERMINAL_SCREEN
DEFAULTS_PATH
PWD
HOME
TEXTDOMAIN
SSH_AGENT_PID
QT_ACCESSIBILITY
XDG_SESSION_TYPE
XDG_DATA_DIRS
XDG_SESSION_DESKTOP
LC_ADDRESS
LC_NUMERIC
GTK_MODULES
PAPERSIZE
WINDOWPATH
VTE_VERSION
TERM
SHELL
QT_IM_MODULE
XMODIFIERS
IM_CONFIG_PHASE
XDG_CURRENT_DESKTOP
GPG_AGENT_INFO
GNOME_TERMINAL_SERVICE
XDG_SEAT
SHLVL
LANGUAGE
LC_TELEPHONE
GDMSESSION
GNOME_DESKTOP_SESSION_ID
LOGNAME
DBUS_SESSION_BUS_ADDRESS
XDG_RUNTIME_DIR
XAUTHORITY
XDG_CONFIG_DIRS
PATH
LC_IDENTIFICATION
SESSION_MANAGER
LESSOPEN
GTK_IM_MODULE
LC_TIME
_
OLDPWD
PYTHONDONTWRITEBYTECODE
PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /home/me/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
SHELL: /bin/bash
LANG: en_US.UTF-8
PWD: /tmp/issue

Contents of Pipfile ('/tmp/issue/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.6"

Contents of Pipfile.lock ('/tmp/issue/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "415dfdcb118dd9bdfef17671cb7dcd78dbd69b6ae7d4f39e8b44e71d60ca72e7"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {},
    "develop": {}
}

Source

mvolfik

All 18 comments

pkg_resources is actually part of setuptools, and it should be installed alongside Pipenv, not inside the virtual environment. How is your Pipenv installed? Does the Python associated with Pipenv (/use/bin/python3) has access to setuptools?

uranusjr on 12 Jul 2018

BTW I have no idea what the pkg-resources package is. You should probably uninstall it.

uranusjr on 12 Jul 2018

I can't uninstall it... It's imported in https://github.com/pypa/pipenv/blob/6cfe5fb993f206c414e408170e24d1793128b35b/pipenv/vendor/pipdeptree.py#L22 so pipenv won't work without it.
My python3 has access to setuptools.
I installed pip with sudo apt-get install python3-pip, then sudo -H pip3 install --upgrade --upgrade-strategy only-if-needed pipenv (I can't upgrade pip, issue https://github.com/pypa/pip/issues/5221)

mvolfik on 12 Jul 2018

No indeed you definitely can't uninstall it, we should be blacklisting it to prevent uninstalling.

techalchemy on 12 Jul 2018

(sidenote, if you pipenv run pip install pkg-resources does that fix it? I assume so)

techalchemy on 12 Jul 2018

me@mypc:/tmp/issue$ pipenv run pip install pkg-resources
Collecting pkg-resources
  Could not find a version that satisfies the requirement pkg-resources (from versions: )
No matching distribution found for pkg-resources
me@mypc:/tmp/issue$

mvolfik on 12 Jul 2018

pipenv run pip install setuptools

techalchemy on 12 Jul 2018

me@mypc:/tmp/issue$ pipenv run pip install setuptools
Requirement already satisfied: setuptools in /home/me/.local/share/virtualenvs/issue_test-O8lAswke/lib/python3.6/site-packages (40.0.0)
me@mypc:/tmp/issue$ pipenv graph

ERROR:  Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pipenv/vendor/pipdeptree.py", line 22, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'

me@mypc:/tmp/issue$ pipenv run pip install --force-reinstall setuptools
Collecting setuptools
  Using cached https://files.pythonhosted.org/packages/ff/f4/385715ccc461885f3cedf57a41ae3c12b5fec3f35cce4c8706b1a112a133/setuptools-40.0.0-py2.py3-none-any.whl
Installing collected packages: setuptools
  Found existing installation: setuptools 40.0.0
    Uninstalling setuptools-40.0.0:
      Successfully uninstalled setuptools-40.0.0
Successfully installed setuptools-40.0.0
me@mypc:/tmp/issue$ pipenv graph

me@mypc:/tmp/issue$ pipenv lock
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock (ca72e7)!
me@mypc:/tmp/issue$ pipenv clean
me@mypc:/tmp/issue$ pipenv graph

me@mypc:/tmp/issue$

Looks like reinstalling setuptools works...

mvolfik on 12 Jul 2018

🎉

uranusjr on 13 Jul 2018

Hey! Why did you close it? Do I have to do pipenv lock, pipenv clean and reinstall setuptools (pipenv run pip install --force-reinstall setuptools) on every new pipenv? That's not solution!

mvolfik on 13 Jul 2018

👍1

Every virtualenv created by Pipenv should have setuptools present without you needing to install it. If they don’t, try reinstalling your virtualenv. If that doesn’t work, there is a problem with your environment. Pipenv already has this sort out, and should work as long as it is allowed to work.

uranusjr on 13 Jul 2018

@uranusjr what do you mean by _try reinstalling your virtualenv_? pip3 install --force-reinstall virtualenv or pipenv --rm && pipenv --three?

mvolfik on 13 Jul 2018

❤1

The former, reinstall your virtualenv library, which Pipenv uses to create virtual environments. Sorry for the inaccurate wording.

uranusjr on 13 Jul 2018

Oh, it worked. Virtualenv 15.1.0 --> 16.0.0 and now pkg_resources not there, locking and cleaning doesn't break it.
__SOLVED__ :heavy_check_mark:

mvolfik on 13 Jul 2018

🎉1

I also have this problem, where pipenv clean tries to remove pkg-resources on a virtualenv created with 16.0.0. I've tried adding setuptools to the pipfile, but it doesn't change anything.

pipenv clean --dry-run
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead.
pkg-resources

Unless I'm wrong, I think the problem is that the blacklist is not complete enough https://github.com/pypa/pipenv/blob/6b1ec91607826e463df3b46fc86652704cf22936/pipenv/core.py#L67

The blacklist of piptools sync, for example includes pkg-resources https://github.com/pypa/pipenv/blob/6b1ec91607826e463df3b46fc86652704cf22936/pipenv/patched/piptools/sync.py#L10-L18

alexgmin on 16 Jul 2018

I think we need to understand the cause before jumping to the fix. In other words, I would very like to know how pkg-resources ended up in your package list in the first place.

uranusjr on 16 Jul 2018

Ok, I found why. https://github.com/pypa/pipenv/issues/1425#issuecomment-368587715 links to https://github.com/pypa/pip/issues/4022 which goes to https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1635463, a bug in Ubuntu 16.04 opened a year and a half ago which is still not fixed.

And the reason piptools has pkg-resources in the ignore list is because of that bug.
https://github.com/jazzband/pip-tools/issues/389
https://github.com/jazzband/pip-tools/pull/555/files

Personally I would suggest the same approach pip-tools did of adding it to the ignore list, since it doesn't seem the ubuntu bug is going to be fixed anytime soon.

alexgmin on 16 Jul 2018

I never understand why Python on Ubuntu can be this f-cked up. Alright, I guess we really need to add it then. Would you be able to open an issue to summarise this, or even better, a pull request to incorporate the new ignore list? Thanks!

uranusjr on 16 Jul 2018

Was this page helpful?

0 / 5 - 0 ratings