Poetry: Poetry add package Permission Denied

[lanahra]

• [X] I have searched the issues of this repo and believe that this is not a duplicate.
• [X] I have searched the documentation and believe that my question is not covered.

Question

When I try to add a new package, poetry tries to access root folders and a 'Permission denied' error is thrown:

```
$ poetry add tldextract
Using version ^2.2 for tldextract

Updating dependencies
Resolving dependencies... (0.6s)

Package operations: 8 installs, 1 update, 5 removals

Writing lock file

• Updating six (1.10.0 -> 1.11.0)

[EnvCommandError]
Command ['pip', 'install', '--no-deps', '-U', 'six==1.11.0'] errored with the following output:
Collecting six==1.11.0
Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-a
ny.whl
Installing collected packages: six
Found existing installation: six 1.10.0
Uninstalling six-1.10.0:
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/lib/python3/dist-packages/__pycache__/six.cpython-35.pyc

It tries to update a package that is installed as the root user and fails when uninstalling the old version.

Does poetry not create its own virtual environment to manage the dependencies? I assumed it would never try access any root folder.

Also, when I run `poetry shell` command, it looks like it is using virtualenv, but I cannot run any command in it:

$ poetry shell
Spawning shell within /home/$USER/.cache/pypoetry/virtualenvs/my-package-py3.5

$ poetry add tldextract
[ImportError]
No module named 'pkg_resources'

I assume something is wrong with my setup, but I could not pinpoint where the problem is. Any help is greatly appreciated.

Thanks

$ python --version
Python 3.5.3

$ pip --version
pip 18.1 from /usr/local/lib/python3.5/dist-packages/pip (python 3.5)

$ poetry --version
Poetry 0.12.7
```

[sdispater]

What's the output of poetry debug:info?

Also, could you execute poetry add tldextract in debug mode (the -vvv option)?

[lanahra]

Sure

$ poetry debug:info

Poetry
======

 * Version: 0.12.7
 * Python:  3.5.3


Virtualenv
==========

 * Python:         3.5.3
 * Implementation: CPython
 * Path:           /home/$USER/.cache/pypoetry/virtualenvs/my-package-py3.5


System
======

 * Platform: linux
 * OS:       posix
 * Python:   /usr

$ poetry add tldextract -vvv
Using virtualenv: /home/$USER/.cache/pypoetry/virtualenvs/my-package-py3.5
PyPI: 34 packages found for tldextract *
Using version ^2.2 for tldextract

Updating dependencies
Resolving dependencies...
   1: fact: my-package is 0.1.0
   1: derived: my-package
   1: fact: my-package depends on tldextract (^2.2)
   1: fact: my-package depends on pytest (^3.0)
   1: selecting my-package (0.1.0)
   1: derived: pytest (^3.0)
   1: derived: tldextract (^2.2)
   1: fact: pytest (3.10.1) depends on py (>=1.5.0)
   1: fact: pytest (3.10.1) depends on attrs (>=17.4.0)
   1: fact: pytest (3.10.1) depends on colorama (*)
   1: fact: pytest (3.10.1) depends on atomicwrites (>=1.0)
   1: fact: pytest (3.10.1) depends on pluggy (>=0.7)
   1: fact: pytest (3.10.1) depends on more-itertools (>=4.0.0)
   1: fact: pytest (3.10.1) depends on six (>=1.10.0)
   1: fact: pytest (3.10.1) depends on pathlib2 (>=2.2.0)
   1: selecting pytest (3.10.1)
   1: derived: pathlib2 (>=2.2.0)
   1: derived: six (>=1.10.0)
   1: derived: more-itertools (>=4.0.0)
   1: derived: pluggy (>=0.7)
   1: derived: atomicwrites (>=1.0)
   1: derived: colorama (*)
   1: derived: attrs (>=17.4.0)
   1: derived: py (>=1.5.0)
PyPI: 1 packages found for tldextract >=2.2,<3.0
   1: fact: tldextract (2.2.0) depends on requests-file (>=1.4)
   1: fact: tldextract (2.2.0) depends on requests (>=2.1.0)
   1: fact: tldextract (2.2.0) depends on idna (*)
   1: selecting tldextract (2.2.0)
   1: derived: idna (*)
   1: derived: requests (>=2.1.0)
   1: derived: requests-file (>=1.4)
   1: fact: pathlib2 (2.3.2) depends on six (*)
   1: selecting pathlib2 (2.3.2)
   1: selecting six (1.11.0)
   1: fact: more-itertools (4.3.0) depends on six (>=1.0.0,<2.0.0)
   1: selecting more-itertools (4.3.0)
   1: selecting pluggy (0.8.0)
   1: selecting atomicwrites (1.2.1)
   1: selecting colorama (0.4.0)
   1: selecting attrs (18.2.0)
   1: selecting py (1.7.0)
   1: selecting idna (2.7)
   1: fact: requests (2.20.1) depends on urllib3 (>=1.21.1,<1.25)
   1: fact: requests (2.20.1) depends on certifi (>=2017.4.17)
   1: fact: requests (2.20.1) depends on idna (>=2.5,<2.8)
   1: fact: requests (2.20.1) depends on chardet (>=3.0.2,<3.1.0)
   1: selecting requests (2.20.1)
   1: derived: chardet (>=3.0.2,<3.1.0)
   1: derived: certifi (>=2017.4.17)
   1: derived: urllib3 (>=1.21.1,<1.25)
   1: fact: requests-file (1.4.3) depends on requests (>=1.0.0)
   1: fact: requests-file (1.4.3) depends on six (*)
   1: selecting requests-file (1.4.3)
   1: selecting chardet (3.0.4)
   1: selecting certifi (2018.10.15)
   1: selecting urllib3 (1.24.1)
   1: Version solving took 0.100 seconds.
   1: Tried 1 solutions.


Package operations: 8 installs, 1 update, 0 removals

  - Updating six (1.10.0 -> 1.11.0)

[EnvCommandError]                                                                                                                             
Command ['pip', 'install', '--no-deps', '-U', 'six==1.11.0'] errored with the following output:                                                            
Collecting six==1.11.0                                                                                                                                     
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl  
Installing collected packages: six                                                                                                                         
  Found existing installation: six 1.10.0                                                                                                                  
    Uninstalling six-1.10.0:                                                                                                                               
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/lib/python3/dist-packages/__pycache__/six.cpython-35.pyc'                      
Consider using the `--user` option or check the permissions.                                                                                               

Exception trace:
 /home/$USER/.poetry/lib/poetry/_vendor/py3.5/cleo/application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 /home/$USER/.poetry/lib/poetry/console/application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 /home/$USER/.poetry/lib/poetry/_vendor/py3.5/cleo/application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 /home/$USER/.poetry/lib/poetry/console/commands/command.py in run() at line 77
   return super(BaseCommand, self).run(i, o)
 /home/$USER/.poetry/lib/poetry/_vendor/py3.5/cleo/commands/base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 /home/$USER/.poetry/lib/poetry/_vendor/py3.5/cleo/commands/command.py in execute() at line 107
   return self.handle()
 /home/$USER/.poetry/lib/poetry/console/commands/add.py in handle() at line 139
   status = installer.run()
 /home/$USER/.poetry/lib/poetry/installation/installer.py in run() at line 76
   self._do_install(local_repo)
 /home/$USER/.poetry/lib/poetry/installation/installer.py in _do_install() at line 287
   self._execute(op)
 /home/$USER/.poetry/lib/poetry/installation/installer.py in _execute() at line 295
   getattr(self, "_execute_{}".format(method))(operation)
 /home/$USER/.poetry/lib/poetry/installation/installer.py in _execute_update() at line 350
   self._installer.update(source, target)
 /home/$USER/.poetry/lib/poetry/installation/pip_installer.py in update() at line 93
   self.install(target, update=True)
 /home/$USER/.poetry/lib/poetry/installation/pip_installer.py in install() at line 90
   self.run(*args)
 /home/$USER/.poetry/lib/poetry/installation/pip_installer.py in run() at line 111
   return self._env.run("pip", *args, **kwargs)
 /home/$USER/.poetry/lib/poetry/utils/env.py in run() at line 472
   return super(VirtualEnv, self).run(bin, *args, **kwargs)
 /home/$USER/.poetry/lib/poetry/utils/env.py in run() at line 288
   raise EnvCommandError(e)

[sdispater]

Thanks.

I am not sure what is happening here since it should not run pip but /home/$USER/.cache/pypoetry/virtualenvs/my-package-py3.5/bin/pip.

Looking at the code in Poetry (here https://github.com/sdispater/poetry/blob/master/poetry/utils/env.py#L305) that can only happen if /home/$USER/.cache/pypoetry/virtualenvs/my-package-py3.5/bin/pip does not exist.

Can you check if /home/$USER/.cache/pypoetry/virtualenvs/my-package-py3.5/bin/pip exists?

[lanahra]

You're right, pip doesn't exist in /home/$USER/.cache/pypoetry/virtualenvs/my-package-py3.5/bin, only python and python3.

[sdispater]

That's odd. There is definitely a problem with your system, since Poetry explicitly tells the venv module to install pip: https://github.com/sdispater/poetry/blob/master/poetry/utils/env.py#L225

[xialu4820723]

When I was using python 3.5, the following step solved this problem.

1. remove created poetry venv: rm -r ~/.cache/pypoetry/virtualenvs/project-py3.5
2. install python-venv: sudo apt-get install -y python3-venv

Then I updated python to 3.7.1, the following step solved this problem.

1. remove created poetry venv: rm -r ~/.cache/pypoetry/virtualenvs/project-py3.7
2. install python3.7-venv: sudo apt-get install -y python3.7-venv

os: ubuntu 16.04

[lanahra]

Just tested for the version 3.5.6 and it worked perfectly, thanks for sharing.
I could not test for version 3.7.1 because the package python3.7-venv is not yet available for the distribution I'm currently using.

[neumond]

This happens due to incomplete virtualenv creation. E.g. I've typed

poetry add request

and quickly realized my mistype, pressed ctrl+C.
After this virtualenv is half-working. Recreating virtualenv helps.

[jpweytjens]

I experienced a similar problem in Ubuntu on WSL. Despite aliases, the poetry installer would always pick up the the system's python2.7. The method in this link worked for me.

The crucial command is the following.
update-alternatives --install /usr/bin/python python /usr/bin/python3.7 10
This makes python point to python3.7 with a priority of 10.

[neumond]

This can break your system. The only _relatively_ safe method of switching default python is using aliases under non-system user.

[lochsh]

I had the same issue on Ubuntu Xenial 16.04.6 LTS with Python 2.7.12, and I believe it was because I did not have pip installed.

Pip isn't listed as a system requirement here so I will create a ticket asking for it to be listed. I realise many Python installations will come with Pip, but e.g. installing Python from source does not.

More info for those interested:

$ python --version
Python 2.7.12

$ python3 --version
Python 3.6.8

$ poetry new test-project-2
Created package test-project-2 in test-project-2

$ cd test-project-2
$ poetry version
[ImportError]
No module named pkg_resources

version [<version>]

$ poetry version -vvv
[ImportError]
No module named pkg_resources

Exception trace:
 /home/fakeuname/.poetry/lib/poetry/_vendor/py2.7/cleo/application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 /home/fakeuname/.poetry/lib/poetry/console/application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 /home/fakeuname/.poetry/lib/poetry/_vendor/py2.7/cleo/application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 /home/fakeuname/.poetry/lib/poetry/console/commands/command.py in run() at line 77
   return super(BaseCommand, self).run(i, o)
 /home/fakeuname/.poetry/lib/poetry/_vendor/py2.7/cleo/commands/base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 /home/fakeuname/.poetry/lib/poetry/_vendor/py2.7/cleo/commands/command.py in execute() at line 107
   return self.handle()
 /home/fakeuname/.poetry/lib/poetry/console/commands/version.py in handle() at line 35
   version = self.increment_version(self.poetry.package.pretty_version, version)
 /home/fakeuname/.poetry/lib/poetry/console/commands/command.py in poetry() at line 62
   return self.get_application().poetry
 /home/fakeuname/.poetry/lib/poetry/console/application.py in poetry() at line 55
   from poetry.poetry import Poetry
 /home/fakeuname/.poetry/lib/poetry/poetry.py in <module>() at line 11
   from .json import validate_object
 /home/fakeuname/.poetry/lib/poetry/json/__init__.py in <module>() at line 4
   import jsonschema
 /home/fakeuname/.poetry/lib/poetry/_vendor/py2.7/jsonschema/__init__.py in <module>() at line 32
   from pkg_resources import get_distribution

version [<version>]

Things worked with Python3 as I had pip3 installed:

$ python3 ~/.poetry/bin/poetry debug:info
Poetry
======

 * Version: 0.12.16
 * Python:  3.6.8


Virtualenv
==========

 * Python:         3.6.8
 * Implementation: CPython
 * Path:           NA


System
======

 * Platform: linux
 * OS:       posix
 * Python:   /usr/local

Using apt install python-pip has resolved the issues.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

Closing this issue automatically because it has not had any activity since it has been marked as stale. If you think it is still relevant and should be addressed, feel free to open a new one.