Poetry: Can't install detectron2

[billop]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Arch Linux
• Poetry version: 1.0.5
• Contents of your pyproject.toml file:

[tool.poetry]
name = "a"
version = "0.1.0"
description = ""
authors = ["a"]

[tool.poetry.dependencies]
python = "^3.6"
detectron2 = { git = "https://github.com/facebookresearch/detectron2.git" }

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Issue

Poetry can't figure out the version of detectron2. I've tried to debug this myself but the error messages don't tell me anything useful:

$ /tmp/poetry/bin/poetry install -vvv

Creating virtualenv a-1_qs6r2x-py3.8 in /home/x/.cache/pypoetry/virtualenvs
Using virtualenv: /home/x/.cache/pypoetry/virtualenvs/a-1_qs6r2x-py3.8
Updating dependencies
Resolving dependencies...
   1: fact: a is 0.1.0
   1: derived: a
   1: fact: a depends on detectron2 (*)
   1: selecting a (0.1.0)
   1: derived: detectron2 (*)
   1: Version solving took 3.842 seconds.
   1: Tried 1 solutions.

[RuntimeError]
Unable to retrieve the package version for /tmp/pypoetry-git-detectron2683igqav

Traceback (most recent call last):
  File "/tmp/poetry/lib/poetry/_vendor/py3.8/clikit/console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "/tmp/poetry/lib/poetry/_vendor/py3.8/clikit/api/command/command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "/tmp/poetry/lib/poetry/_vendor/py3.8/clikit/api/command/command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "/tmp/poetry/lib/poetry/_vendor/py3.8/cleo/commands/command.py", line 92, in wrap_handle
    return self.handle()
  File "/tmp/poetry/lib/poetry/console/commands/install.py", line 63, in handle
    return_code = installer.run()
  File "/tmp/poetry/lib/poetry/installation/installer.py", line 74, in run
    self._do_install(local_repo)
  File "/tmp/poetry/lib/poetry/installation/installer.py", line 161, in _do_install
    ops = solver.solve(use_latest=self._whitelist)
  File "/tmp/poetry/lib/poetry/puzzle/solver.py", line 36, in solve
    packages, depths = self._solve(use_latest=use_latest)
  File "/tmp/poetry/lib/poetry/puzzle/solver.py", line 180, in _solve
    result = resolve_version(
  File "/tmp/poetry/lib/poetry/mixology/__init__.py", line 7, in resolve_version
    return solver.solve()
  File "/tmp/poetry/lib/poetry/mixology/version_solver.py", line 80, in solve
    next = self._choose_package_version()
  File "/tmp/poetry/lib/poetry/mixology/version_solver.py", line 355, in _choose_package_version
    packages = self._provider.search_for(dependency)
  File "/tmp/poetry/lib/poetry/puzzle/provider.py", line 130, in search_for
    packages = self.search_for_vcs(dependency)
  File "/tmp/poetry/lib/poetry/puzzle/provider.py", line 167, in search_for_vcs
    package = self.get_package_from_vcs(
  File "/tmp/poetry/lib/poetry/puzzle/provider.py", line 204, in get_package_from_vcs
    package = cls.get_package_from_directory(tmp_dir, name=name)
  File "/tmp/poetry/lib/poetry/puzzle/provider.py", line 343, in get_package_from_directory
    raise RuntimeError(

Is poetry unable to get dynamically generated versions?

[malcolmgreaves]

Came to file a bug about not being able to install detectron2 -- as of poetry 1.1.3 it's still not installable.

detectron2 needs torch to install:

$ poetry install
Updating dependencies
Resolving dependencies... (4.5s)

  PackageInfoError

  Unable to determine package info for path: /tmp/pypoetry-git-detectron2dazfvgem

  Fallback egg_info generation failed.

  Command ['/tmp/tmpp_0xdcph/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1, and output:
  Traceback (most recent call last):
    File "setup.py", line 10, in <module>
      import torch
  ModuleNotFoundError: No module named 'torch'

  at ~/.poetry/lib/poetry/inspection/info.py:503 in _pep517_metadata
      499│                     venv.run("python", "setup.py", "egg_info")
      500│                     return cls.from_metadata(path)
      501│                 except EnvCommandError as fbe:
      502│                     raise PackageInfoError(
    → 503│                         path, "Fallback egg_info generation failed.", fbe
      504│                     )
      505│                 finally:
      506│                     os.chdir(cwd.as_posix())
      507│

Even though I explicitly have torch as a dependency:

[tool.poetry]
name = "my_project"
version = "0.0.0"
description = ""
authors = ["Me <[email protected]>"]
license = "None"

[tool.poetry.dependencies]
python = "^3.7"
torch = "1.4"
detectron2 = { git = "https://github.com/facebookresearch/detectron2.git", branch = "master" }

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.1.3"]
build-backend = "poetry.core.masonry.api"

Curiously, if I remove the detectron2 line, do poetry install, then add back the detectron2 line and do poetry shell, I am able to follow the project's installation instructions successfully:

poetry
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'

But, even if I do the above w/o manually doing python -m pip, i.e. a poetry update instead, (so there's still torch) it doesn't work (same error).

[malcolmgreaves]

I imagine this might be a difficult problem (building wheels from general setup.pys). Any chance that there's a way to have poetry try to install other dependencies first before doing git installable ones? (Does this even make sense? ~@finswimmer if you have any time to look into this~ 🙏 )

[abn]

@malcolmgreaves the issue is that the library does not add torch to it's build-backend.requires. This is not something poetry can resolve, even if we install torch into the project's virtualenv. The build for detectron happens in it's own isolated environment with whatever it declares as it's build requirements.

The fix is for the project to add a pyproject.toml with the relevant build-system section.

[malcolmgreaves]

😭 Oh that's very unfortunate! I appreciate the very prompt reply @abn 💯 I'll try and bug the FAIR folk to see if this can be done.

[abn]

No worries. Wish it was a more helpful response. You can just send them a PR with a minimal pyproject.toml. In the interim, you can use a branch on your fork with the change.

[malcolmgreaves]

@abn Would the minimum viable pyproject.toml be to specify that torch, alongside setuptools and wheel, is a requirement of the build system? i.e. does this formulation make sense?
~https://github.com/malcolmgreaves/detectron2/pull/2/files~ https://github.com/malcolmgreaves/detectron2/commit/3a42efe2b3df7d7c2c0a882540e0dde8c8cc8275

(I originally first thought that I should make it a poetry project https://github.com/malcolmgreaves/detectron2/pull/1/files#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711 But then realized that it should be possible to keep their build current build system, but just indicate that building the wheel requires torch too, right?)

[malcolmgreaves]

🎉 I have answered my own question -- just adding torch to requires in the build-system section is all that's necessary to get poetry to understand how to build the detectron2 wheel! 🎉

[build-system]
requires = ["setuptools", "wheel", "torch"]

[abn]

@malcolmgreaves for completeness this is what you need to keep legacy build.

[build-system]
requires = ["setuptools", "wheel", "torch"]
build-backend = "setuptools.build_meta:__legacy__"

[malcolmgreaves]

Wonderful! Thank you a ton 💯 💯 🤗

[abn]

I am going to close this issue as the original issue has been resolved and the current issue has to do with lack of PEP 517 metadata in the package source.

[joostmeulenbeld]

@malcolmgreaves I have this exact same issue, did you open an issue or pull request at the original Detectron2 repo?

[malcolmgreaves]

@joostmeulenbeld I have a workaround here -- you can poetry install this as a { git = ... } dependency
https://github.com/malcolmgreaves/detectron2