Pipenv: Support for virtualenv --system-site-packages

[danyeaw]

Some python packages aren't pip installable. For example, to use PyGObject in Linux, normally you have to either have to create the virtualenv with the --system-site-packages option, manually create the symbolic links between the virtualenv and the distribution installed packages, or use another external python package like vext that attempts to automate this for you (I haven't had much success with this option). It would be great if pipenv supported a --system-site-packages type option. It would be even better if we could find a better way to automate usage of these types of libraries, because having to use --system-site-packages is still a pain point especially if you want to setup a pipeline to automate testing.

[ncoghlan]

Doing some CI performance investigations for https://github.com/leapp-to/, we eventually realised that one of our problems is that we're setting up some of our dependencies twice: ansible, pipsi, and pipenv are already installed as part of bootstrapping the CI environment, but they're also listed in Pipfile for local development purposes, so when we run pipenv install as an Ansible task, we get a second copy of all 3 of them inside the virtual environment.

However, it's not immediately clear to me how to reconcile a --system-site-packages option with Pipfile.lock. It may be that a better workflow for our case would be to use pipenv to generate a traditional requirements.txt file and use that in CI. If we agree that's a better way to go, then this would become purely a documentation question at the pipenv level.

(see https://github.com/leapp-to/prototype/issues/220 for more on our specific use case)

[ncoghlan]

Following up: for the use case where I initially thought we might want this feature, the right answer turned out to be running pipenv lock --requirements | sort > integration-tests/requirements.txt and checking in the result. That way developers can benefit from pipenv's opinionated environment management for local development, but in the more constrained CI environment, we have more control over how things are done. In our particular case, I realised we didn't actually need a Python level virtual environment at all, since we're using a Software Collection runtime that's already independent of the system Python.

[kennethreitz42]

I don’t think pipenv will grow this feature.

[aWangami]

https://matplotlib.org/faq/virtualenv_faq.html @kennethreitz

One often suggested solution is to use the --system-site-packages option to virtualenv when creating an environment.

[ncoghlan]

@aWangami See my comment above - for such cases, a good way to go is to rely on fully isolated environments and wheel files during development, and then generate a requirements.txt file for use with --system-site-packages.

[kennethreitz42]

It's just out of scope for us, effectively.

[kennethreitz42]

One of the most important parts of running an open source project is knowing when to say no :)

[bpaczkowski]

What about packages like wxpython that can't easily be installed in a virtualenv?

[kennethreitz42]

they should fix that problem then

[bpaczkowski]

Is it their problem though?

Installing wxPython…
Requirement already satisfied: wxPython in /home/bartek/.local/share/virtualenvs/desktop-automation-Stwbc6fq/lib/python3.6/site-packages
Requirement already satisfied: six in /home/bartek/.local/share/virtualenvs/desktop-automation-Stwbc6fq/lib/python3.6/site-packages (from wxPython)                                          

Adding wxPython to Pipfile's [packages]…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Traceback (most recent call last):
  File "/usr/bin/pipenv", line 11, in <module>
    load_entry_point('pipenv==7.3.9', 'console_scripts', 'pipenv')()
  File "/usr/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/pipenv/cli.py", line 1532, in install
    do_lock(system=system)
  File "/usr/lib/python3.6/site-packages/pipenv/cli.py", line 951, in do_lock
    project=project
  File "/usr/lib/python3.6/site-packages/pipenv/utils.py", line 217, in resolve_deps
    resolved_tree = resolver.resolve()
  File "/usr/lib/python3.6/site-packages/piptools/resolver.py", line 107, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/lib/python3.6/site-packages/piptools/resolver.py", line 187, in _resolve_one_round
    best_matches = set(self.get_best_match(ireq) for ireq in constraints)
  File "/usr/lib/python3.6/site-packages/piptools/resolver.py", line 187, in <genexpr>
    best_matches = set(self.get_best_match(ireq) for ireq in constraints)
  File "/usr/lib/python3.6/site-packages/piptools/resolver.py", line 245, in get_best_match
    best_match = self.repository.find_best_match(ireq, prereleases=self.prereleases)
  File "/usr/lib/python3.6/site-packages/piptools/repositories/pypi.py", line 116, in find_best_match
    raise NoCandidateFound(ireq, all_candidates)
piptools.exceptions.NoCandidateFound: Could not find a version that matches wxpython
Tried: 4.0.0a1, 4.0.0a2, 4.0.0a3, 4.0.0b1, 4.0.0b2

Not to mention the size of the package and the requirement to build it from source which takes half an hour, unless it is installed pre-built from the distributions package manager (and then it is installed into /usr/lib, so without --system-site-packages it won't work).

[kennethreitz42]

they should ship wheels.

[bpaczkowski]

That doesn't explain this error at all, and why that same error exists on Windows as well where they do ship the pre-built module.

[kennethreitz42]

no, this error has nothing to do with your complaint that they don't install well in virtualenvs.

[vorstrelok]

I think you can achieve this by using pew's toggleglobalsitepackages feature like this:

pew workon (name-of-your-environment)
pew toggleglobalsitepackages

And now you have access to all your globally installed packages in this venv.
But I think it kinda defeats the purpouse of using pipenv and pipfile in the first place.

[kennethreitz42]

that's a great workaround.

[bpaczkowski]

no, this error has nothing to do with your complaint that they don't install well in virtualenvs.

Since this error pops up only when trying to install wxpython to a virtualenv, then what has it to do with? Mind you it has actually installed it just fine, it just can't seem to add the module to the Pipfile.

[kennethreitz42]

this is a problem with locking, and pip-tools.

Show me your Pipfile.

[bpaczkowski]

Ok, it created the Pipfile, it's the Pipfile.lock that's missing.

Pipfile is basic:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[packages]
wxpython = "*"

[dev-packages]

[kennethreitz42]

looks like they only have prereleases out...

[kennethreitz42]

we don't currently support prereleases

[techalchemy]

This pipfile is old format. Does the problem persist with the latest pipenv? Btw I currently handle this problem by creating the virtualenv myself with the global site packages option and sourcing it before using pipenv. Your alternative option is to grab pipenv --where and delete the no-global-site-packages.txt file from lib/pythonx.y

On Sep 20, 2017, at 8:44 PM, Bartek Paczkowski [e-mail] wrote:

Ok, it created the Pipfile, it's the Pipfile.lock that's missing.

Pipfile is basic:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[packages]
wxpython = "*"

[dev-packages]

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[kennethreitz42]

we now support --site-packages

[bpaczkowski]

we don't currently support prereleases

I hate to be that guy, but when is it planned to add support for them?

[kennethreitz42]

we support them now! pipenv install --pre

[bpaczkowski]

Working great now, thanks!

[bpaczkowski]

@kennethreitz Installing works fine, but what if I need to uninstall something? --pre option doesn't exist on the uninstall command, so it crashes with the same error I mentioned above.