Poetry: Cross path dependencies cause SolverProblemError

[atugushev]

• [X] I am on the latest Poetry version.
• [X] I have searched the issues of this repo and believe that this is not a duplicate.
• [X] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS 10.15.5
• Poetry version: 1.1.0
• Link of a Gist with the contents of your pyproject.toml file:

Issue

If a project (e.g., poetry-bug-with-cross-path-deps) has the following dependencies:

• poetry-bug-with-cross-path-deps depends on package1
• poetry-bug-with-cross-path-deps depends on package2
• package1 depends on package2

and the package1 and package2 are path dependencies, then poetry lock fails on:

  SolverProblemError

  Because poetry-bug-with-cross-path-deps depends on package1 (0.1.0 deps/package1) which depends on package2 (0.2.0 deps/package2), package2 is required.
  So, because poetry-bug-with-cross-path-deps depends on package2 (0.2.0 deps/package2), version solving failed.

[finswimmer]

Hello @atugushev,

the problem here is, that you the package2 dependency for poetry-bug-with-cross-path-deps is in a different path, than the package2 dependency for package2.

Your folder structure looks like this:

poetry-bug-with-cross-path-deps
├── deps
│   ├── package1
│   │   ├── deps
│   │   │   └── package2
│   │   │       └── pyproject.toml
│   │   └── pyproject.toml
│   └── package2
│       └── pyproject.toml
├── README.md
└── pyproject.toml

So you have poetry-bug-with-cross-path-deps/deps/package2 and poetry-bug-with-cross-path-deps/deps/package1/deps/package2. You have to rename one of these packages.

fin swimmer

[atugushev]

Hi @finswimmer,

Thanks for your suggestion. In the real-world package1 and package2 are submodules, so renaming can't help in that case.

[finswimmer]

Hello @atugushev,

can you add more detail what you mean by "submodules" here exactly?

At the end of the day, you cannot install two different packages with the same name. So I guess your task must be solved in another way. Happy to help here, but for that I need to understand what the task is :)

fin swimmer

[benoit9126]

I have the same problem with git dependencies pointing on a specific branch.

  Because package1 (2.0.0 git rev develop) depends on package2 (0.1.0 git rev master)
   and poetry-bug-with-cross-path-deps depends on package2 (0.1.0 git branch master), package1 is forbidden.
  So, because poetry-bug-with-cross-path-deps depends on package1 (2.0.0 git branch develop), version solving failed.

[atugushev]

That's right. It's git submodules.

[lig]

@finswimmer actually, it doesn't matter here why the paths are like that. This is a real-world scenario that works in 1.0.

Basically, what happens here is that there are two packages specified via relative paths. In case one of the packages specifies the path which is already specified in the root package as its local dependency poetry fails to resolve the dependencies.

The poetry 1.0 behavior was to look for this path relative to the root package's dir. Whether this is good behavior or not is not relevant. That behavior worked and was intuitively expected.

Indeed, if package1 specifies package2 as its dependency and package2 could be found and installed according to the specification in package1 dependencies it is expected for this to work.

P.S.: This issue is not a feature request or proposal. This is a bug report regarding an expected behavior that is broken in poetry 1.1. I'd advise treating this one accordingly.

[2m]

We noticed a similar issue with poetry 1.1.0 and 1.1.1 (1.0.10 works fine).

We have a root-project that depends on helper and lib. lib also depends on helper. All of the dependencies are git="..." rev="..." pointing to the same revision.

The error we get is:

Resolving dependencies... (250.4s)
  SolverProblemError

  Because lib (0.1.0 git rev 7e8be74) depends on helper (0.2.0 git rev 833799f)
   and root-project depends on helper (0.2.0 git rev 833799f), lib is forbidden.
  So, because root-project depends on lib (0.1.0 git rev 7e8be74), version solving failed.

  at ~/.poetry/lib/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

Note that poetry reports that both dependencies are the same:

... depends on helper (0.2.0 git rev 833799f) and root-project depends on helper (0.2.0 git rev 833799f) ...

But it still reports SolverProblemError.

[abn]

Basically, what happens here is that there are two packages specified via relative paths. In case one of the packages specifies the path which is already specified in the root package as its local dependency poetry fails to resolve the dependencies.

This is expected behaviour. In 1.0 the resolution was incorrectly performed for nested path dependencies. This meant that if you had ./deps/package2 and ./deps/package1/deps/package2, when ./deps/package1 metadata was inspected while resolving for your root project, build incorrectly selected ./deps/package2 as a dependency of package1 when it should have selected, ./deps/package1/deps/package2. This was resolved in 1.1

When correctly resolved, you will encounter a solving issue because effectively you have pinned two different versions of package2 - package2 @ ./deps/package2 and package2 @ ./deps/package1/deps/package2. Two pinned versions pointing to different paths are inherently incompatible.

[abn]

@2m is your scenario multiple copies of submodules as well? If not, could you please try the fix at #3105 ?

[benoit9126]

@abn I agree with you with path dependencies: two different paths must be considered as two different dependencies.

Nevertheless, with the example of @2m, the dependency is the same git repository at the same revision... It must be considered as the same dependency and be valid. Idem with git dependencies pointing to the same branch on the same repository.

[abn]

It must be considered as the same dependency and be valid. Idem with git dependencies pointing to the same branch on the same repository.

The question is how they are included. Hence why my comment above asking for clarification. Submodules are path dependencies, not git dependencies from a poetry/pep-508 perspective.

I agree with you with path dependencies: two different paths must be considered as two different dependencies.

To be clear, they can be the same dependency, but are considered two specific versions of it.

[benoit9126]

The question is how they are included. Hence why my comment above asking for clarification. Submodules are path dependencies, not git dependencies from a poetry/pep-508 perspective.

I can not answer for @2m but I have troubles with git dependencies of the form library = {git="[email protected]:organization/repository", branch="..."} and there are no submodules at all.

To be clear, they can be the same dependency, but are considered two specific versions of it.

Thanks!

[2m]

In my case all dependencies are provided as git dependencies, like so:

lib = { git = "[email protected]:org/repo.git", rev = "833799f" }

I will try the fix provided in #3105 a bit later and will report back.

[2m]

Tried poetry from #3105 and still got the same error as in https://github.com/python-poetry/poetry/issues/3030#issuecomment-704288487

[abn]

@benoit9126 @2m can we capture the git dependency issue in another issue please? Would be great to have an example project as well.

[2m]

I tried to reproduce the problem I noticed in a minimized example but could not get the same error. Will try harder a bit later.

[2m]

Created a separate issue with a reproducer project: https://github.com/python-poetry/poetry/issues/3113

[lig]

@abn

This is expected behaviour. In 1.0 the resolution was incorrectly performed for nested path dependencies. This meant that if you had ./deps/package2 and ./deps/package1/deps/package2, when ./deps/package1 metadata was inspected while resolving for your root project, build incorrectly selected ./deps/package2 as a dependency of package1 when it should have selected, ./deps/package1/deps/package2. This was resolved in 1.1

I'm deeply sorry, but the 1.0 behavior is actually expected and logical behavior.

With 1.1 behavior it is impossible to install dependencies in this case.

The case is:

• We have the current project which directly depends on ./deps/package2.
• We have package1 which directly depends on ./deps/package2.
• We add package1 as a direct dependency to the current project as ./deps/package1.
• We want the current project to still depend directly on ./deps/package2 as we don't want to know what the underlying dependencies package1 has.
• Now we have the current project which depends directly on ./deps/package1 and ./deps/package2 and package1 which depends on ./deps/package1/deps/package2 where ./deps/package2 and ./deps/package1/deps/package2 are different versions of package2.

Poetry 1.1 output is as follows

  SolverProblemError

  Because poetry-bug-with-cross-path-deps depends on package1 (0.1.0 deps/package1) which depends on package2 (0.2.0 deps/package2), package2 is required.
  So, because poetry-bug-with-cross-path-deps depends on package2 (0.2.0 deps/package2), version solving failed.

As far as I understand now this package2 (0.2.0 deps/package2) and package2 (0.2.0 deps/package2) refers to different absolute paths which is not clear btw.

Also, please notice that both packages have the same version defined in their pyproject.toml files. However, as those are different paths (which is unclear) poetry treats them as different versions.

Correct me if I'm wrong here.

If this is correct then:

• I suggest displaying paths in a different way. Maybe, fallback to absolute paths if relative paths are the same or display them relative to the current project's base dir.
• Will it help to add version = "*" to allow poetry to install at least one of those packages? Preferably it should be the direct dependency effectively falling back to poetry 1.0 behavior.
• I still think that if the indirect requirement matches the version constraint of the direct one it's ok to just install the direct one.

Update:

• Poetry version 1.1.2 displays absolute paths for everything except second-level dependency which is being displayed relative to its pyproject.toml
• Adding version = "*" to path-dependency results in RuntimeError

  The Poetry configuration is invalid:
    - [dependencies.package2] {'path': 'deps/package2', 'version': '*'} is not valid under any of the given schemas

[lig]

Basically, Poetry 1.1.0-1.1.2 is not working at the moment for the case for which Poetry 1.0 was working at least in some way.

[finswimmer]

@lig am I right that your expectation about the dependency resolution is like that:

Whenever poetry finds a path dependency, it should read the name and the version from there. When its finished to collect all these information, it should decide based on that which path dependency it installs.

While this sounds reasonable on the first sight, this is not how it works or how it should work. A package from different sources must be treated as different version, regardless of which version number is defined. This is why the path is interpreted as a pinned version. And if you have a package with two different pinned version in your dependency graph, you cannot resolve it.

Why must we treat different sources as different version? Well, we cannot guarantee that they are the same package. Maybe it's a coincidence that the name is equal?

fin swimmer

[lig]

@finswimmer the reality is that whilst the software cannot guarantee something people are not dumb and could guarantee that via a policy on which they agreed on. In that case, the software needs to stand aside and allow people to handle that.

I want to state "I know what I'm doing" and software should be able to obey that.