Poetry: Install failure in more than two nested subprojects

[linw1995]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: 10.15.2 macOS
• Poetry version: 1.0.0

Issue

a
├── a
│   └── __init__.py
├── b
│   ├── b
│   │   └── __init__.py
│   ├── c
│   │   ├── c
│   │   │   └── __init__.py
│   │   └── pyproject.toml
│   └── pyproject.toml
└── pyproject.toml

5 directories, 6 files

Execute
cat a/pyproject.toml

[tool.poetry]
name = "a"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.8"
b = {path = "b"}

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Execute
cat a/b/pyproject.toml

[tool.poetry]
name = "b"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.8"
c = {path="c"}

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Execute
cat a/b/c/pyproject.toml

[tool.poetry]
name = "c"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.8"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

In project structure like this, execute poetry install -vvv will fail, and it's output below

Execute
poetry install -vvv

Using virtualenv: /Users/linw1995/Library/Caches/pypoetry/virtualenvs/a-lL1Z62BU-py3.8
Updating dependencies
Resolving dependencies...
   1: fact: a is 0.1.0
   1: derived: a
   1: fact: a depends on b (*)
   1: selecting a (0.1.0)
   1: derived: b (*)
   1: fact: b (0.1.0) depends on c (*)
   1: selecting b (0.1.0 b)
   1: derived: c (*)
   1: selecting c (0.1.0 b/c)
   1: Version solving took 0.057 seconds.
   1: Tried 1 solutions.

Writing lock file


Package operations: 2 installs, 0 updates, 0 removals

  - Installing c (0.1.0 b/c)

[EnvCommandError]
Command ['/Users/linw1995/Library/Caches/pypoetry/virtualenvs/a-lL1Z62BU-py3.8/bin/pip', 'install', '--no-deps', '-U', '-e', '/Users/linw1995/a/b/b/c'] errored with the following return code 1, and output:
ERROR: /Users/linw1995/a/b/b/c is not a valid editable requirement. It should either be a path to a local project or a VCS URL (beginning with svn+, git+, hg+, or bzr+).
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.


Traceback (most recent call last):
  File "/Users/linw1995/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages/clikit/console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "/Users/linw1995/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages/clikit/api/command/command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "/Users/linw1995/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages/clikit/api/command/command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "/Users/linw1995/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages/cleo/commands/command.py", line 92, in wrap_handle
    return self.handle()
  File "/Users/linw1995/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages/poetry/console/commands/install.py", line 63, in handle
    return_code = installer.run()
  File "/Users/linw1995/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages/poetry/installation/installer.py", line 74, in run
    self._do_install(local_repo)
  File "/Users/linw1995/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages/poetry/installation/installer.py", line 286, in _do_install
    self._execute(op)
  File "/Users/linw1995/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages/poetry/installation/installer.py", line 302, in _execute
    getattr(self, "_execute_{}".format(method))(operation)
  File "/Users/linw1995/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages/poetry/installation/installer.py", line 327, in _execute_install
    self._installer.install(operation.package)
  File "/Users/linw1995/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages/poetry/installation/pip_installer.py", line 32, in install
    self.install_directory(package)
  File "/Users/linw1995/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages/poetry/installation/pip_installer.py", line 221, in install_directory
    return self.run(*args)
  File "/Users/linw1995/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages/poetry/installation/pip_installer.py", line 118, in run
    return self._env.run_pip(*args, **kwargs)
  File "/Users/linw1995/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages/poetry/utils/env.py", line 824, in run_pip
    return self._run(cmd, **kwargs)
  File "/Users/linw1995/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages/poetry/utils/env.py", line 1052, in _run
    return super(VirtualEnv, self)._run(cmd, **kwargs)
  File "/Users/linw1995/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages/poetry/utils/env.py", line 856, in _run
    raise EnvCommandError(e, input=input_)

By installing in more than two nested subprojects, the second subprojects' req value went wrong.

https://github.com/python-poetry/poetry/blob/945ca1d365d66489858e05b4b50d06e579af56ba/poetry/installation/pip_installer.py#L181

package.root_dir='a/b' is the directory of package b, package.source_url='b/c' is the relative path from package a to package c. This line of code gets a/b/b/c by joining this two values.

[mxab]

For me this is also not working https://github.com/python-poetry/poetry/issues/1636
Even after I use poetry 1.0 the error are getting actually stranger for this docker sample

I have another project where I was a bit more successful when I did an install in every sub project first, but this doesn't feel right

[finswimmer]

Hello @linw1995,

this indeed looks like a bug and we will have to take a closer look at this.

But beside this, could you please explain why you need this structure? IMO different projects should be kept separated.

fin swimmer

[linw1995]

@finswimmer thanks for responding. In my case, I use git for source control. The project b is the submodule of the project a, and the project c is the submodule of the project b.

[finswimmer]

Hello @linw1995,

I'm not sure if we both have the same definition of "submodules" (or how I would call it: "subpackages"). But it looks like what you are trying to achieve is:

• have a package "a" which depends on package "b"
• have a package "b" that have a subpackage "c"

Your folder structure would look like this:

Project
├── a
│   ├── a
│   │   └── __init__.py
│   └── pyproject.toml
└── b
    ├── b
    │   ├── __init__.py
    │   └── c
    │       └── __init__.py
    └── pyproject.toml

With a/pyproject.toml:

[tool.poetry]
name = "a"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.8"
b = {path = "../b"}

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

and b/pyproject.toml:

[tool.poetry]
name = "b"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.8"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

If you now run poetry install in a, spawning the shell with poetry shell and running python, you can do things like:

import a
import b
import b.c

[linw1995]

Thanks for the advice that may temporarily solve my problem. But "submodule" what I mean is the git submodule, not the python submodule. projects a, b and c are separate git repository.
My solution to temporarily solve my problem is by reducing the nested levels to install all dependencies successfully. I comment out c = {path='c'} in the a/b/pyproject.toml, use git submodule add to add project c into the project a and add c = {path='c'} into 'a/pyproject.toml'.

[kevincon]

could you please explain why you need this structure? IMO different projects should be kept separated.

I am wishing a nested project structure could be supported to help centralize tool configuration in a monorepo containing multiple Python packages while still supporting being able to publish each package individually to a pypi server.

E.g. I am working on a project with the following structure, where project-package, support-package-1, and support-package-2 are all poetry-managed Python packages:

project-package
│   mypy.ini
│   pyproject.toml
│
├───lib
│   ├───support-package-1
│   │   │   pyproject.toml
│   │   │
│   │   ├───src
│   │   │   └───support_package_1
│   │   │           ...
│   │   │
│   │   ├───stubs
│   │   │       ...
│   │   │
│   │   └───tests
│   │           ...
│   │
│   └───support-package-2
│       │   pyproject.toml
│       │
│       ├───src
│       │   └───support_package_2
│       │           ...
│       │
│       └───tests
│               ...
├───src
│   └───project_package
│           ...
│
└───tests
        ...

The dependency hierarchy I would like to have in this setup is that root-package depends on support-package-1, and support-package-1 depends on support-package-2. However, this doesn't seem to be possible today with poetry 1.0.9 on Windows 10. I get the following error when trying to run poetry update from the root of root-package with that dependency hierarchy encoded in the pyproject.toml files:

Package operations: 10 installs, 0 updates, 0 removals

  - Installing support-package-2 (0.1.0 lib/support-package-2)

[EnvCommandError]
Command ['C:\\Users\\kdconley\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\root-package-NG3dGAKP-py3.8\\Scripts\\pip.exe', 'install', '--no-deps', '-U', '-e', 'C:\\Users\\kdconley\\root-package\\lib\\support-package-1\\lib/support-package-2'] errored with the following return code 1, and output:
ERROR: C:\Users\kdconley\root-package\lib\support-package-1\lib/support-package-2 is not a valid editable requirement. It should either be a path to a local project or a VCS URL (beginning with svn+, git+, hg+, or bzr+).

For now I am working around this limitation by explicitly adding both support-package-1 and support-package-2 to the list of dependencies in root-package's pyproject.toml file. That seems to make poetry update happy. But this is technically unnecessary since the knowledge that support-package-1 depends on support-package-2 is already encoded in support-package-1's pyproject.toml file.

As an example for what I mean by "centralize tool configuration", I have written the mypy.ini file in the root of the root-package directory to enable easily type-checking all of these nested packages by running poetry run mypy from the root of root-package:

# mypy.ini
[mypy]
; Type-check all Python files under root-package including those in support-package-1 and support-package-2
files = **/*.py
; Use some stubs only needed for support-package-1, etc.
mypy_path = lib/support-package-1/stubs
..

[misuzu]

Applying following patch on top of 1.0.9 seems to fix this issue:

diff --git a/poetry/installation/pip_installer.py b/poetry/installation/pip_installer.py
index 349f222..a00f418 100644
--- a/poetry/installation/pip_installer.py
+++ b/poetry/installation/pip_installer.py
@@ -183,10 +183,7 @@ class PipInstaller(BaseInstaller):
         from poetry.factory import Factory
         from poetry.utils.toml_file import TomlFile

-        if package.root_dir:
-            req = os.path.join(package.root_dir, package.source_url)
-        else:
-            req = os.path.realpath(package.source_url)
+        req = os.path.realpath(package.source_url)

         args = ["install", "--no-deps", "-U"]

The piece above was introduced in https://github.com/python-poetry/poetry/commit/58e3676c1894e22d598e5b5009fbab4ac96ee291
@sdispater could you please clarify why second code path is needed? It seems to work fine without it.

[laurenmolley]

@sdispater I'm seeing the same error that was in the original reported issue. I'm using latest 1.1.0b2 poetry version. Only difference in mine is that I use relative paths where the above example nests. The behavior is exactly the same in that the wrong path is constructed in the pip install command.

Only workaround I have with this structure is @kevincon 's where I copy child path references into the parent pyproject.toml.

[linw1995]

Recently I found out that Poetry version 1.1.2 fixes this issue. Thanks to the effort that all contributors made.