Poetry: 1.1.0b2 causes EnvCommandError (Non-posix paths with file:// protocol on Windows)

[fredrikaverpil]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows 10
• Poetry version: 1.1.0b2
• Link of a Gist with the contents of your pyproject.toml file:

Issue

When on Poetry 1.0.10 there are no issues. But after having updated to poetry 1.1.0b2, I constantly get an EnvCommandError caused by a FileNotFoundError (a temporary setup.py file is not found):

$ poetry --version
Poetry version 1.1.0b2

$ poetry lock
Updating dependencies
Resolving dependencies...

Writing lock file

$ poetry install -vvv                                         
Using virtualenv: C:\Users\fredrik\code\repos\mylittleci\venv
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 2 installs, 0 updates, 0 removals

  • Installing zipp (3.1.0)

  Stack trace:

  7  c:\users\fredrik\.local\pipx\venvs\poetry\lib\site-packages\poetry\installation\executor.py:183 in _execute_operation
      181│
      182│             try:
    → 183│                 result = self._do_execute_operation(operation)
      184│             except EnvCommandError as e:
      185│                 if e.e.returncode == -2:

  6  c:\users\fredrik\.local\pipx\venvs\poetry\lib\site-packages\poetry\installation\executor.py:257 in _do_execute_operation
      255│             return 0
      256│
    → 257│         result = getattr(self, "_execute_{}".format(method))(operation)
      258│
      259│         if result != 0:

  5  c:\users\fredrik\.local\pipx\venvs\poetry\lib\site-packages\poetry\installation\executor.py:394 in _execute_install
      392│
      393│     def _execute_install(self, operation):  # type: (Install) -> None
    → 394│         return self._install(operation)
      395│
      396│     def _execute_update(self, operation):  # type: (Update) -> None

  4  c:\users\fredrik\.local\pipx\venvs\poetry\lib\site-packages\poetry\installation\executor.py:432 in _install
      430│             args.insert(2, "-U")
      431│
    → 432│         return self.run_pip(*args)
      433│
      434│     def _update(self, operation):

  3  c:\users\fredrik\.local\pipx\venvs\poetry\lib\site-packages\poetry\installation\executor.py:283 in run_pip
      281│     def run_pip(self, *args, **kwargs):  # type: (...) -> int
      282│         try:
    → 283│             self._env.run_pip(*args, **kwargs)
      284│         except EnvCommandError as e:
      285│             output = decode(e.e.output)

  2  c:\users\fredrik\.local\pipx\venvs\poetry\lib\site-packages\poetry\utils\env.py:883 in run_pip
       881│         pip = self.get_pip_command()
       882│         cmd = pip + list(args)
    →  883│         return self._run(cmd, **kwargs)
       884│
       885│     def _run(self, cmd, **kwargs):

  1  c:\users\fredrik\.local\pipx\venvs\poetry\lib\site-packages\poetry\utils\env.py:1151 in _run
      1149│             self.unset_env("__PYVENV_LAUNCHER__")
      1150│
    → 1151│             return super(VirtualEnv, self)._run(cmd, **kwargs)
      1152│
      1153│     def execute(self, bin, *args, **kwargs):

  EnvCommandError

  Command C:\Users\fredrik\code\repos\mylittleci\venv\Scripts\pip.exe install --no-deps file://C:\Users\fredrik\AppData\Local\pypoetry\Cache\artifacts\3f\29\e0\445941d0045028dce97085f4263281587225a49fa7dd09f41e4c2752af\zipp-3.1.0-py3-none-any.whl errored with the following return code 1, and output:
  Processing c:\users\fredrik\appdata\local\pypoetry\cache\artifacts\3f\29\e0\445941d0045028dce97085f4263281587225a49fa7dd09f41e4c2752af\zipp-3.1.0-py3-none-any.whl
      ERROR: Command errored out with exit status 1:
       command: 'c:\users\fredrik\code\repos\mylittleci\venv\scripts\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\fredrik\\AppData\\Local\\Temp\\pip-req-build-gsp4euj0\\setup.py'"'"'; __file__='"'"'C:\\Users\\fredrik\\AppData\\Local\\Temp\\pip-req-build-gsp4euj0\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\fredrik\AppData\Local\Temp\pip-pip-egg-info-_md_1odm'
           cwd: C:\Users\fredrik\AppData\Local\Temp\pip-req-build-gsp4euj0\
      Complete output (5 lines):
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
        File "C:\Users\fredrik\AppData\Local\Programs\Python\Python37\lib\tokenize.py", line 447, in open
          buffer = _builtin_open(filename, 'rb')
      FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\fredrik\\AppData\\Local\\Temp\\pip-req-build-gsp4euj0\\setup.py'
      ----------------------------------------
  ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.


  at c:\users\fredrik\.local\pipx\venvs\poetry\lib\site-packages\poetry\utils\env.py:915 in _run
       911│                 output = subprocess.check_output(
       912│                     cmd, stderr=subprocess.STDOUT, **kwargs
       913│                 )
       914│         except CalledProcessError as e:
    →  915│             raise EnvCommandError(e, input=input_)
       916│
       917│         return decode(output)
       918│
       919│     def execute(self, bin, *args, **kwargs):

The contents of my pyproject.toml:

[tool.poetry]
name = "mylittleci"
version = "0.0.1"
authors = ["Fredrik <***@***.***>"]
description = "mylittleci"


[tool.poetry.dependencies]
python = "^3.7"


[tool.poetry.dev-dependencies]
importlib_metadata = "^1.7.0"


[tool.poetry.scripts]
calculator = "mylittleci.cli.calculator:main"


[build-system]
# https://python-poetry.org/docs/faq/#is-tox-supported
requires = ["poetry-core>=1.0.0a5"]
build-backend = "poetry.core.masonry.api"

[fredrikaverpil]

I am re-opening this, at I am hitting this constantly now, with poetry 1.1.0b2.

I have attempted the poetry install command in both Powershell and in git bash, both on the same Windows machine. In both cases, I get the FileNotFoundError (to the temporary setup.py) as seen in the code block in my post above.

If I downgrade back to poetry 1.0.10 I can see that the dummy setup.py file is being created in my repo root and the poetry install command completes without errors. With Poetry 1.1.0b2, it seems as if the temporary setup.py file is either not yet created or has already been deleted when an attempt to access it is being made.

[fredrikaverpil]

@dispater I tracked down what the issues is here, and I could set up a PR to fix this. But I want to hear where you think this should get fixed, as I see two possible approaches.

I suppose this is a Windows-only problem. When inside the poetry.utils.env.py:Env._run, I can see what the cmd variable holds with a simple print statement.

In my case, cmd comes into the method as:

['C:\\Users\\fredrik\\code\\repos\\mylittleci\\venv\\Scripts\\pip.exe', 'install', '--no-deps', 'file://C:\\Users\\fredrik\\AppData\\Local\\pypoetry\\Cache\\artifacts\\47\\bb\\ec\\bf68abe7d5464fced474bdf20de54de7737439f79a80b8bfc247e33c40\\smmap-3.0.4-py2.py3-none-any.whl']

Since I am on Windows, the list is converted into a string:

C:\Users\fredrik\code\repos\mylittleci\venv\Scripts\pip.exe install --no-deps file://C:\Users\fredrik\AppData\Local\pypoetry\Cache\artifacts\47\bb\ec\bf68abe7d5464fced474bdf20de54de7737439f79a80b8bfc247e33c40\smmap-3.0.4-py2.py3-none-any.whl

If I, after the list-to-string conversion, add a simple cmd = cmd.replace("\\", "/"), the pip installation completes without errors, and this is on Windows.

It also works removing the protocol and leaving the backslashes in: cmd = cmd.replace("file://", "")

The deadly combination is to combine the protocol with backslashes.

Now, to my question before attempting a PR...

Is there a reason why this file:// protocol is used?
I'm thinking the problem can either be solved under self._is_windows in the poetry.utils.env.py::_run method, or it can be solved e.g. in the poetry.installation.chef.py::get_cached_archives_for_link method. What do you think would be best?

[fredrikaverpil]

I updated the comment above. Please re-read, if you've already had time to read it.

[abn]

@fredrikaverpil thank you for the details you have provided. The issue here is the following piece of code.

https://github.com/python-poetry/poetry/blob/c77299ed373df50ec459907671f65f5da3de5070/poetry/installation/executor.py#L425

This is problematic because str() as this is not cross-platform safe. Looking into this, at the moment the type of archive is not necessarily guarenteed to be a Path instance either, this needs to be fixed and the above use of str(archive) needs to be replaced with archive.as_posix() instead.

[fredrikaverpil]

@abn exactly. I was actually working on a fix and a regression test for this. Do you want me to make a branch with this in it?

    def get_cached_archives_for_link(self, link):  # type: (Link) -> List[Link]
        cache_dir = self.get_cache_directory_for_link(link)

        archive_types = ["whl", "tar.gz", "tar.bz2", "bz2", "zip"]
        links = []
        for archive_type in archive_types:
            for archive in cache_dir.glob("*.{}".format(archive_type)):
                if self._env._is_windows:
                    archive = archive.as_posix()
                links.append(Link("file://{}".format(str(archive))))

        return links

[abn]

@fredrikaverpil freel free to submit a PR for the code you linked above by replacing str(archive) with archive.as_posix(), there is no need to use the windows check, let pathlib worry about that :)

I'll clean up the other bits later anyway.