Poetry: Pip-installed Poetry encountering EnvironmentError "Permission denied" on Windows when upgrading cffi

[PAStheLoD]

• [ ] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.

• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows 10

• Poetry version: 1.0.10

Issue

I've just encountered this problem while running poetry update, which upgraded the cffi package. Using pip 20.2.3 on python 3.9b5 on Windows 10, while using poetry 1.0.10.

PS C:\pas\projects\xxxx\ci> .\venv-windows\Scripts\python.exe -m poetry update
Skipping virtualenv creation, as specified in config file.
Updating dependencies
Resolving dependencies...

Writing lock file


Package operations: 0 installs, 5 updates, 0 removals

  - Updating attrs (19.3.0 -> 20.2.0)
  - Updating cffi (1.14.0 -> 1.14.2)

[EnvCommandError]
Command ['C:\\pas\\projects\\xxxx\\ci\\venv-windows\\Scripts\\python.exe', '-m', 'pip', 'install', '--no-deps', '-U', 'cffi==1.14.2'] errored with the following return code 1, and output:
Collecting cffi==1.14.2
  Downloading cffi-1.14.2.tar.gz (470 kB)
Using legacy 'setup.py install' for cffi, since package 'wheel' is not installed.
Installing collected packages: cffi
  Attempting uninstall: cffi
    Found existing installation: cffi 1.14.0
    Uninstalling cffi-1.14.0:
      Successfully uninstalled cffi-1.14.0
    Running setup.py install for cffi: started
    Running setup.py install for cffi: finished with status 'done'
ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Permission denied: 'C:\\Users\\Admin\\AppData\\Local\\Temp\\pip-uninstall-8rjju1cu\\_cffi_backend.cp39-win_amd64.pyd'

Of course it's not the best practice to install poetry via pip. Maybe providing a command line argument to ignore these errors might be worth it?

[abn]

@PAStheLoD not sure that is a poetry problem tbh. Looks like pip trying to install cffi fails.

You should be able to reproduce the issue when running the command from this error manually.

Command ['C:\\pas\\projects\\xxxx\\ci\\venv-windows\\Scripts\\python.exe', '-m', 'pip', 'install', '--no-deps', '-U', 'cffi==1.14.2'] errored with the following return code 1, and output:

Also your experience on poetry from master might be better for this scenario. There are some improvements on how we install packages.

[PAStheLoD]

Sorry for the incomplete/unclear report. What I haven't explicitly stated is that poetry is installed into the same virtualenv that it manipulates. So I guess it is using that particular file. pip itself vendors most/all (?) of its dependencies (if I understand the pip repo correctly).

Thanks for the quick reply! If I'll have some time I'll try to reproduce this with just pip, and will check out master too.

[jeremad]

Exact same issue with poetry, cffi and windows, I try to create a minimum reproduction

[abn]

@PAStheLoD @jeremad as I mentioned before this is to do with pip not being able to install the package from sdist, caused due to the permission error. This is not something poetry can control. Would be great to know the root cause of the permission error.

As for the failures, this might no longer be an issue with the latest pre-release since the installation logic and venv creation (#2666) has changed. Please try with the pre-release.

[PAStheLoD]

The install part itself successfully happens, it's the removal of the temp pip-uninstall-[nonce] thing. Presumably pip moves the potentially still in-use package to a temp folder. Then tries to delete it on exit, and fails, because it's used (by poetry or whatever invoked pip from the same venv).

At least this is my (admittedly incomplete) understanding. I tried to find some relevant discussion about this post-run cleanup mechanism, but only found this https://github.com/pypa/pip/issues/7567 , and it seems this Windows-only thing is still not implemented (and likely it wouldn't be able to handle this case anyway).

So probably the main problem is that poetry should not both manage and run from the same venv. Thanks again!

[abn]

The install part itself successfully happens, it's the removal of the temp pip-uninstall-[nonce] thing. Presumably pip moves the potentially still in-use package to a temp folder. Then tries to delete it on exit, and fails, because it's used (by poetry or whatever invoked pip from the same venv).

Since imported modules are in memory, deleting their files should not matter unless it is deleting an exe or something, which should not be happening anyway.

At least this is my (admittedly incomplete) understanding. I tried to find some relevant discussion about this post-run cleanup mechanism, but only found this pypa/pip#7567 , and it seems this Windows-only thing is still not implemented (and likely it wouldn't be able to handle this case anyway).

So probably the main problem is that poetry should not both manage and run from the same venv. Thanks again!

Poetry in theory should be running outside the venv anyway, unless it was explicitly installed into the venv.

[PAStheLoD]

Oh, I thought .pyd files were similar to .dll files in this regard. (So that they get memory mapped into the process space, but even after reading about it I'm not sure. If they are created with Cython then they are not bytecode but machine code.)

Yes, in my case it was explicitly installed into the same venv.

[Korijn]

I have the same issue (since today), just in a blank new virtualenv, when running poetry run python -m pip install -U pip:

I just freshly installed poetry 1.0.10, python 3.6.8 (64bit), Windows 10

[abn]

@Korijn what happens when you do the following? (My windows is a bit rusty!)

sh python3.6 -m venv .venv .venv\Scripts\python.exe -m pip install -U pip

[Korijn]

@Korijn what happens when you do the following? (My windows is a bit rusty!)

python3.6 -m venv .venv
.venv\Scripts\python.exe -m pip install -U pip

Seems to be fine... huh...

[abn]

@Korijn confirming that is python 3.6?

This is bizarre. I'll try reproducing it in a windows VM.

[Toxe]

I have a similar issue. Newest version of poetry and Windows 10 Home (version 20H2), a fresh environment and Python 3.8.6. Using PowerShell in the new Windows Terminal app.

When running poetry update I get:

Updating dependencies
Resolving dependencies...

Writing lock file

Package operations: 0 installs, 23 updates, 1 removal

  • Removing more-itertools (8.5.0)
  • Updating colorama (0.4.3 -> 0.4.4)
  • Updating isort (5.5.2 -> 5.6.4)
  • Updating toml (0.10.1 -> 0.10.2)
  • Updating prompt-toolkit (3.0.7 -> 3.0.8)
  • Updating pygments (2.7.1 -> 2.7.2)
  • Updating traitlets (5.0.4 -> 5.0.5)
  • Updating sqlalchemy (1.3.19 -> 1.3.20)
  • Updating attrs (20.2.0 -> 20.3.0)
  • Updating certifi (2020.6.20 -> 2020.11.8)
  • Updating iniconfig (1.0.1 -> 1.1.1)
  • Updating kiwisolver (1.2.0 -> 1.3.1)
  • Updating packaging (20.4 -> 20.7)
  • Updating numpy (1.19.2 -> 1.19.4)
  • Updating pathspec (0.8.0 -> 0.8.1)
  • Updating ipython (7.18.1 -> 7.19.0)
  • Updating regex (2020.7.14 -> 2020.11.13)
  • Updating urllib3 (1.25.10 -> 1.26.2)
  • Updating pillow (7.2.0 -> 8.0.1)

  EnvCommandError

  Command C:\Users\toxe\AppData\Local\pypoetry\Cache\virtualenvs\gps-tracks-rest-api-mrZqQZX0-py3.8\Scripts\pip.exe install --no-deps -U file:///C:/Users/toxe/AppData/Local/pypoetry/Cache/artifacts/6e/95/18/3db0ce3882c0ae273d01c0efa35e8bf56a95390ca600c814ace94cea9a/regex-2020.11.13-cp38-cp38-win_amd64.whl errored with the following return code 1, and output:
  Processing c:\users\toxe\appdata\local\pypoetry\cache\artifacts\6e\95\18\3db0ce3882c0ae273d01c0efa35e8bf56a95390ca600c814ace94cea9a\regex-2020.11.13-cp38-cp38-win_amd64.whl
  Installing collected packages: regex
    Attempting uninstall: regex
      Found existing installation: regex 2020.7.14
      Uninstalling regex-2020.7.14:
        Successfully uninstalled regex-2020.7.14
  ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'C:\\Users\\toxe\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\gps-tracks-rest-api-mrZqQZX0-py3.8\\Lib\\site-packages\\~egex\\_regex.cp38-win_amd64.pyd'
  Consider using the `--user` option or check the permissions.

  WARNING: You are using pip version 20.2.2; however, version 20.3.1 is available.
  You should consider upgrading via the 'C:\Users\toxe\AppData\Local\pypoetry\Cache\virtualenvs\gps-tracks-rest-api-mrZqQZX0-py3.8\Scripts\python.exe -m pip install --upgrade pip' command.


  at ~\.poetry\lib\poetry\utils\env.py:1074 in _run
      1070│                 output = subprocess.check_output(
      1071│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1072│                 )
      1073│         except CalledProcessError as e:
    → 1074│             raise EnvCommandError(e, input=input_)
      1075│
      1076│         return decode(output)
      1077│
      1078│     def execute(self, bin, *args, **kwargs):

If anyone wants to reproduce this:

• clone https://github.com/Toxe/gps-tracks.git
• checkout commit 991a58e
• go into the rest-api folder
• create your environment and perform a poetry update from there

[sinoroc]

@Toxe Looks like a _pip_ issue somehow, so maybe out of _poetry_'s control. Looks like at the moment the error occurs, poetry had launched this command in a subprocess:
C:\Users\toxe\AppData\Local\pypoetry\Cache\virtualenvs\gps-tracks-rest-api-mrZqQZX0-py3.8\Scripts\pip.exe install --no-deps -U file:///C:/Users/toxe/AppData/Local/pypoetry/Cache/artifacts/6e/95/18/3db0ce3882c0ae273d01c0efa35e8bf56a95390ca600c814ace94cea9a/regex-2020.11.13-cp38-cp38-win_amd64.whl
and then pip fails while trying to delete this file for whatever reason:
'C:\\Users\\toxe\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\gps-tracks-rest-api-mrZqQZX0-py3.8\\Lib\\site-packages\\~egex\\_regex.cp38-win_amd64.pyd.

Have you tried deleting that file? Are you sure the issue is reproducible in a fresh environment?

[Toxe]

@sinoroc

Are you sure the issue is reproducible in a fresh environment?

Yes, 100% sure. Actually the above log was from a fresh environment.

[sinoroc]

Maybe the _regex.cp38-win_amd64.pyd is still being accessed somehow. Maybe some clues on _pip_'s ticket tracker...

• https://github.com/pypa/pip/issues/8878

Does poetry show --tree (before running the update) show what depends on regex? Can't test it myself, I don't have Windows.

[Toxe]

This is the peotry show --tree part of the output where regex is listed:

black 20.8b1 The uncompromising code formatter.
|-- appdirs *
|-- click >=7.1.2
|-- mypy-extensions >=0.4.3
|-- pathspec >=0.6,<1
|-- regex >=2020.1.8
|-- toml >=0.10.1
|-- typed-ast >=1.4.0
`-- typing-extensions >=3.7.4

[sinoroc]

@Toxe OK, nothing all too meaningful there. Does not give any more clues.

Seems to me like the only sure thing, is that there is a file that can not be deleted when _pip_ is trying to uninstall something as part of an upgrade. And to me it is not sure that the issue lies on _poetry_'s side.

[Toxe]

I agree, at this point I suspect some weird Windows access permissions issue.