Poetry: Improve dependency source_url equality check to use normalised urls instead of given/derrived values

[2m]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.

• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• macOS 10.15.6

• Poetry version used is locally built latest master: https://github.com/python-poetry/poetry/commit/746d741d81f7b2dd39798ac95e867052d8a4d9d0
• https://github.com/2m/ptry-project/blob/master/pyproject.toml

Issue

[ptry-project] has the following project dependency graph:

ptry-project -+-> ptry-lib --> ptry-helper
              \-> ptry-helper

[ptry-project] is a poetry project. [ptry-lib] and [ptry-helper] both use `setup.py`. All of the dependencies are specified using git URLs and point to a specific git commit hash. Latest poetry (last working version was `1.0.10`) fails when running `poetry update` in `ptry-project`. verbose log output:

ptry-project ─╼ poetry@local update -vvv                                                                             0
Using virtualenv: /Users/martynas/Library/Caches/pypoetry/virtualenvs/ptry-project-3Lsr60E6-py3.8
Updating dependencies
Resolving dependencies...
   1: fact: ptry-project is 0.1.0
   1: derived: ptry-project
   1: fact: ptry-project depends on ptry-helper (0.1.0 git rev 047f569)
   1: fact: ptry-project depends on ptry-lib (0.1.0 git rev 3665750)
   1: fact: ptry-project depends on pytest (^5.2)
   1: fact: ptry-project depends on pytest (^5.2)
   1: selecting ptry-project (0.1.0)
   1: derived: pytest (^5.2)
   1: derived: ptry-lib (0.1.0 git rev 3665750)
   1: derived: ptry-helper (0.1.0 git rev 047f569)
PyPI: 15 packages found for pytest >=5.2,<6.0
   1: fact: ptry-lib (0.1.0 git rev 3665750) depends on ptry-helper (0.1.0 git rev 047f569)
   1: conflict: ptry-lib (0.1.0 git rev 3665750) depends on ptry-helper (0.1.0 git rev 047f569)
   1: ! not ptry-helper (0.1.0 git rev 047f569) is satisfied by ptry-helper (0.1.0 git rev 047f569)
   1: ! which is caused by "ptry-project depends on ptry-helper (0.1.0 git rev 047f569)"
   1: ! thus: ptry-lib is forbidden
   1: ! ptry-lib (0.1.0 git rev 3665750) is satisfied by ptry-lib (0.1.0 git rev 3665750)
   1: ! which is caused by "ptry-project depends on ptry-lib (0.1.0 git rev 3665750)"
   1: ! thus: version solving failed
   1: Version solving took 23.160 seconds.
   1: Tried 1 solutions.

  Stack trace:

  8  ~/Library/Caches/pypoetry/virtualenvs/poetry-RboxO2pV-py3.8/lib/python3.8/site-packages/clikit/console_application.py:131 in run
      129│             parsed_args = resolved_command.args
      130│
    → 131│             status_code = command.handle(parsed_args, io)
      132│         except KeyboardInterrupt:
      133│             status_code = 1

  7  ~/Library/Caches/pypoetry/virtualenvs/poetry-RboxO2pV-py3.8/lib/python3.8/site-packages/clikit/api/command/command.py:120 in handle
      118│     def handle(self, args, io):  # type: (Args, IO) -> int
      119│         try:
    → 120│             status_code = self._do_handle(args, io)
      121│         except KeyboardInterrupt:
      122│             if io.is_debug():

  6  ~/Library/Caches/pypoetry/virtualenvs/poetry-RboxO2pV-py3.8/lib/python3.8/site-packages/clikit/api/command/command.py:171 in _do_handle
      169│         handler_method = self._config.handler_method
      170│
    → 171│         return getattr(handler, handler_method)(args, io, self)
      172│
      173│     def __repr__(self):  # type: () -> str

  5  ~/Library/Caches/pypoetry/virtualenvs/poetry-RboxO2pV-py3.8/lib/python3.8/site-packages/cleo/commands/command.py:92 in wrap_handle
       90│         self._command = command
       91│
    →  92│         return self.handle()
       93│
       94│     def handle(self):  # type: () -> Optional[int]

  4  ~/projects/poetry/poetry/console/commands/update.py:47 in handle
      45│         self._installer.update(True)
      46│
    → 47│         return self._installer.run()
      48│

  3  ~/projects/poetry/poetry/installation/installer.py:103 in run
      101│         local_repo = Repository()
      102│
    → 103│         return self._do_install(local_repo)
      104│
      105│     def dry_run(self, dry_run=True):  # type: (bool) -> Installer

  2  ~/projects/poetry/poetry/installation/installer.py:235 in _do_install
      233│             )
      234│
    → 235│             ops = solver.solve(use_latest=self._whitelist)
      236│         else:
      237│             self._io.write_line("Installing dependencies from lock file")

  1  ~/projects/poetry/poetry/puzzle/solver.py:65 in solve
       63│         with self._provider.progress():
       64│             start = time.time()
    →  65│             packages, depths = self._solve(use_latest=use_latest)
       66│             end = time.time()
       67│

  SolverProblemError

  Because ptry-lib (0.1.0 git rev 3665750) depends on ptry-helper (0.1.0 git rev 047f569)
   and ptry-project depends on ptry-helper (0.1.0 git rev 047f569), ptry-lib is forbidden.
  So, because ptry-project depends on ptry-lib (0.1.0 git rev 3665750), version solving failed.

  at ~/projects/poetry/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

This was previously discussed a bit in: https://github.com/python-poetry/poetry/issues/3030#issuecomment-704288487

[2m]

Additional note: if ptry-lib and ptry-helper are poetry managed projects then poetry update does not raise any error anymore.

[abn]

@2m forgot to comment here before, it will also work if you update the url you use in pyproject.toml to be the same as what's in the requirements_sources.txt file in ptry-lib (ie. prefixed with ssh).

The source url comparison is a non-smart equality check. Which is expected atm.

[2m]

Thanks for the hint. After the following change poetry update completed successfully:

-ptry-helper = { git = "[email protected]:2m/ptry-helper.git", rev = "047f569" }
+ptry-helper = { git = "ssh://[email protected]/2m/ptry-helper.git", rev = "047f569" }

Note that not only ssh:// needs to be appended but also : is replaced with /. This solves our issue and, as far as we are concerned, this issue can be closed.

[abn]

I would like to keep this open for us to consider normalising the urls when checking equality.