Pyenv-virtualenv: Can't create a venv off system Python

[dimaqq]

> pyenv virtualenv system pho-test
Collecting virtualenv
  Using cached virtualenv-15.1.0-py2.py3-none-any.whl
Installing collected packages: virtualenv
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip-9.0.1-py2.7.egg/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/usr/local/lib/python2.7/dist-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/usr/local/lib/python2.7/dist-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/usr/local/lib/python2.7/dist-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "/usr/local/lib/python2.7/dist-packages/pip-9.0.1-py2.7.egg/pip/wheel.py", line 345, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/usr/local/lib/python2.7/dist-packages/pip-9.0.1-py2.7.egg/pip/wheel.py", line 323, in clobber
    shutil.copyfile(srcfile, destfile)
  File "/usr/lib/python2.7/shutil.py", line 83, in copyfile
    with open(dst, 'wb') as fdst:
IOError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/virtualenv.py'

Why does it try to overwrite something in /usr/local/?

Aaalso, my system python is in /usr/bin, not /usr/local/bin...

[pgadige]

Hello @dimaqq I had exactly the same issue while attempting to execute pyenv virtualenv system my-playground, even though my system version of Python is 2.7.15 on Ubuntu 16.04 LTS; 64-bit. My system Python is in /usr/bin as well.

However, I remember experiencing exactly the same error, IOError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/virtualenv.py' when attempting to pip install virtualenv directly. Then I used sudo pip install virtualenv to mitigate the error.

I found a work around for this particular issue involving pyenv-virtualenv.

The following are the steps I executed in order for a project that required Python 2.7+ and virtualenv:

• pyenv install 2.7.9

• pyenv local 2.7.9

• pyenv virtualenv 2.7.9 my-playground

and viola! it worked!

I hope this helps :smiley:

[pgadige]

I'm still searching for answers for the same questions that you have. Please share your findings if you've figured why it happens. Thank you!

[codylane]

You did not mention which OS you are using or how you have pyenv installed or how you have your environment setup, what shell you are using... etc. It is difficult to provide a solution without that info but I will try.

Also, since the system python is generally installed by some package manager, then this follows all other *nix types of services/packaging where things installed by a package are owned not by your user but generally by some other user and it's generally the root user. You are getting this error because the user that owns the directory for your system python is does not allow your user to modify that directory unless you have access to that user or can become that user.

However, I do not recommend you mess with the system python or permissions of that folder because you are asking for trouble if you do.

What I normally do is exactly as @pgadige did and just install a the same system version into my pyenv environment so that I no longer experience any permissions problems.

However, there are cases when it calls for using the system python especially if you need to use something like --system-site-packages on a particular python environment because it is not always possible to install things via pip. A good example of this is on Ubuntu where there is a python module called python-apt or python2-apt but that module cannot be installed by pip or outside of problems of where to obtain the source.

References for using only the system python in a virtualenv

• python-apt

To get around the bug above I did the following as non-root user. This was on Ubuntu 16.04 with the following shell: /bin/bash

PYENV_ROOT="/var/lib/jenkins/.pyenv"
export PATH="${PYENV_ROOT}/bin:${PATH}"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

[ "$USER" != "root" ] && echo "OK: you are not root" || echo "ERR: You are root all bets are off"

mkdir -p /tmp/foo-test2
cd /tmp/foo-test2
pyenv virtualenv --system-site-packages system foo-test2-2.7.12

yields

Running virtualenv with interpreter /usr/bin/python2
New python executable in /var/lib/jenkins/.pyenv/versions/foo-test2-2.7.12/bin/python2
Also creating executable in /var/lib/jenkins/.pyenv/versions/foo-test2-2.7.12/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
ensurepip is disabled in Debian/Ubuntu for the system python.

Python modules For the system python are usually handled by dpkg and apt-get.

    apt-get install python-<module name>

Install the python-pip package to use pip itself.  Using pip together
with the system python might have unexpected results for any system installed
module, so use it on your own risk, or make sure to only use it in virtual
environments.

Installing pip from https://bootstrap.pypa.io/get-pip.py...
/tmp/tmp1salWm/pip.zip/pip/_vendor/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.
Collecting pip
  Using cached https://files.pythonhosted.org/packages/5f/25/e52d3f31441505a5f3af41213346e5b6c221c9e086a166f3703d2ddaf940/pip-18.0-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 18.0
    Uninstalling pip-18.0:
      Successfully uninstalled pip-18.0
Successfully installed pip-18.0

Now, I have a replicated pyenv environment of my system python as a virtualenv

pyenv local foo-test2-2.7.12
pyenv rehash
python -c 'import apt' # this may produce a traceback

If you get an error, then the --system-site-packages doesn't work, but in my case, I get nothing on the screen which indicates everything is working and I now have a system virtualenv.

[dimaqq]

I've worked around this by pyenv install the version of interpreter I needed (let's say matching the system) and then installing required packages. In other words, by not using system.

Perhaps in the short term it's enough to prevent making virtual envs off system?

[codylane]

For sure, that is what I do and try to avoid doing anything with the system python. I generally try to recreate the system python in my pyenv environment so that I can reproduce my app from scratch every time. Sure, this might be overkill but it saves a lot of stress down the road as my app grows.

Looking at the number of open issues around not being able to create a virtualenv from the system python it appears that others also experience this same problem so each solution might be a bit different.

The example I provided above was an attempt re-create system python on a Ubuntu 16.04 machine. I couldn't recreate it because the apt pip module is packaged only for Debian and only installable through system packages and it's not really a pip installable module or at least I couldn't find a recent version that actually worked. So my example is one such solution to using pyenv-virtualenv and the system python.

Anyway, I hope that some of this information helped and wasn't to confusing.

[codylane]

One thing that just occurred to me is I can you provide the OS version you are using? Also can you also provide the output of these commands?

pyenv shell system
pip list
python --version
echo $SHELL

# if using bash
[ -f ~/.profile ] && cat ~/.profile
[ -f ~/.bash_profile ] && cat ~/.bash_profile
[ -f ~/.bashrc ] && cat ~/.bashrc

I'm not wondering if you have virtualenv installed in your system python and have activated it in your shell. If so, that may (I don't have proof) cause problems with pyenv virtualenv.

[luni3359]

I'm experiencing a similar issue.

pi@raspberrypi:~ $  pyenv  virtualenv system vtest
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: virtualenv in ./.local/lib/python2.7/site-packages (20.0.27)
Requirement already satisfied: appdirs<2,>=1.4.3 in ./.local/lib/python2.7/site-packages (from virtualenv) (1.4.4)
Requirement already satisfied: importlib-metadata<2,>=0.12; python_version < "3.8" in ./.local/lib/python2.7/site-packages (from virtualenv) (1.7.0)
Requirement already satisfied: six<2,>=1.9.0 in /usr/lib/python2.7/dist-packages (from virtualenv) (1.12.0)
Requirement already satisfied: filelock<4,>=3.0.0 in ./.local/lib/python2.7/site-packages (from virtualenv) (3.0.12)
Requirement already satisfied: importlib-resources>=1.0; python_version < "3.7" in ./.local/lib/python2.7/site-packages (from virtualenv) (3.0.0)
Requirement already satisfied: distlib<1,>=0.3.1 in ./.local/lib/python2.7/site-packages (from virtualenv) (0.3.1)
Requirement already satisfied: pathlib2<3,>=2.3.3; python_version < "3.4" and sys_platform != "win32" in ./.local/lib/python2.7/site-packages (from virtualenv) (2.3.5)
Requirement already satisfied: configparser>=3.5; python_version < "3" in ./.local/lib/python2.7/site-packages (from importlib-metadata<2,>=0.12; python_version < "3.8"->virtualenv) (4.0.2)
Requirement already satisfied: contextlib2; python_version < "3" in ./.local/lib/python2.7/site-packages (from importlib-metadata<2,>=0.12; python_version < "3.8"->virtualenv) (0.6.0.post1)
Requirement already satisfied: zipp>=0.5 in ./.local/lib/python2.7/site-packages (from importlib-metadata<2,>=0.12; python_version < "3.8"->virtualenv) (1.2.0)
Requirement already satisfied: typing; python_version < "3.5" in ./.local/lib/python2.7/site-packages (from importlib-resources>=1.0; python_version < "3.7"->virtualenv) (3.7.4.3)
Requirement already satisfied: singledispatch; python_version < "3.4" in ./.local/lib/python2.7/site-packages (from importlib-resources>=1.0; python_version < "3.7"->virtualenv) (3.4.0.3)
Requirement already satisfied: scandir; python_version < "3.5" in ./.local/lib/python2.7/site-packages (from pathlib2<3,>=2.3.3; python_version < "3.4" and sys_platform != "win32"->virtualenv) (1.10.0)
pyenv: virtualenv: command not found
Installing pip from https://bootstrap.pypa.io/get-pip.py...
pyenv: python: command not found
error: failed to install pip via get-pip.py