Poetry: Poetry can't run scripts from dependencies (which themselves use poetry) installed as editable on Windows

[kevincon]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows 10 Enterprise version 1909 build 18363.1139
• Poetry version: 1.1.3
• Link of a Gist with the contents of your pyproject.toml file: https://github.com/kevincon/poetry_editable_dep_script_bug_repro

Issue

My team is experiencing that poetry run is unable to run scripts registered with poetry by editable-installed dependencies of our main poetry project on Windows.

I've created this repo to provide a minimal project to demonstrate this issue: https://github.com/kevincon/poetry_editable_dep_script_bug_repro

Also I'm using Python 3.8.3 32-bit.

In that repo, the main project, parent , has a local, editable path dependency of child which is itself a Poetry project.

Both the parent project and the child project register a script using the tool.poetry.scripts section in their respective pyproject.toml files. All these scripts do is print out Hello from parent CLI! or Hello from child CLI!, respectively.

Installing the parent project is successful:

C:\Users\kevincon\repos\poetry_editable_dep_script_bug_repro 
λ poetry install
Creating virtualenv parent-MPYm-h1y-py3.8 in C:\Users\kevincon\AppData\Local\pypoetry\Cache\virtualenvs
Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing child (0.1.0 C:/Users/kevincon/repos/poetry_editable_dep_script_bug_repro/child)

Installing the current project: parent (0.1.0)

As is running the parent project's script:

C:\Users\kevincon\repos\poetry_editable_dep_script_bug_repro 
λ poetry run parent-cli
Hello from parent CLI!

But trying to run the child project's script fails with this error (with 100% reproducibility):

C:\Users\kevincon\repos\poetry_editable_dep_script_bug_repro 
λ poetry run child-cli

  FileNotFoundError

  [WinError 2] The system cannot find the file specified

  at ~\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py:1307 in _execute_child
      1303│             sys.audit("subprocess.Popen", executable, args, cwd, env)
      1304│
      1305│             # Start the process
      1306│             try:
    → 1307│                 hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
      1308│                                          # no special security
      1309│                                          None, None,
      1310│                                          int(not close_fds),
      1311│                                          creationflags,

(we would expect it to not throw an error and instead print out Hello from child CLI!)

However, we can successfully run both the parent and child project scripts if we use poetry version 1.0.10.

I think what changed to break this behavior between 1.0.10 and 1.1.x is #2360, which seems to have changed how scripts of editable projects are installed. Previously a .exe file was created for scripts registered with poetry in the Scripts/ directory of the poetry-managed virtual environment (I think by pip?), but now a .cmd file is created by poetry for these scripts instead.

I think I understand that the parent project's script is run successfully in poetry 1.1.3 because poetry finds it in the parent project's pyproject.toml file and takes this code path: https://github.com/python-poetry/poetry/blob/68d6939f2e9cac1178fc5ffbcc3bac7a9db43fce/poetry/console/commands/run.py#L27-L28

But since the child project's script won't be found in the parent project's pyproject.toml file, it takes a different code path for the child script which ultimately tries to use subprocess.Popen() to call the child script: https://github.com/python-poetry/poetry/blob/68d6939f2e9cac1178fc5ffbcc3bac7a9db43fce/poetry/utils/env.py#L977

Unfortunately it looks like the subprocess.Popen call which poetry uses to try to run the child script ends up calling _winapi.CreateProcess() (as shown in the error output above) which I learned does not auto-resolve any other executable extensions besides .exe, so it doesn't find the .cmd executable for the child script. See https://stackoverflow.com/a/10555130 and https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessa?redirectedfrom=MSDN#parameters.

This seems to be supported by the fact that we are able to run the child project's script using poetry 1.1.3 if we add the .cmd extension ourselves:

C:\Users\kevincon\repos\poetry_editable_dep_script_bug_repro 
λ poetry run child-cli.cmd
Hello from child CLI!

[finswimmer]

Hello @kevincon,

thanks a lot for reporting and your detailed investigation :+1: Do you think, you can provide a PR to fix it? :)

fin swimmer

[avirlrma]

Hey,

I would like to work on this. Can I get assigned?

Thanks.

[kevincon]

Ah sorry @avirlrma I forgot to call it out, but I have fixed this in a branch and I think I can put up a PR for it tomorrow. Do you mind if I finish it up?

[avirlrma]

Yes sure, please go ahead.