Poetry: Can not install any poetry based project with pip from source distribution

[garbas]

• [ ] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.

• [ ] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: NixOS 18.09

• Poetry version: --
• Link of a Gist with the contents of your pyproject.toml file: --

Issue

Hi,

this is not really a bug about how poetry works, but how poetry interacts with other python packaging ecosystem.

While working on a project of mine which uses pip inside (for now) I noticed that we can not install a package called backoff which in its recent version switched to poetry.

To reproduce the error you simply need to invoke the following command:

$ pip install backoff --no-binary :all:

Above command tries to install packages from source distribution and not from wheel (which is a requirement of my project).

Digging deeper I noticed that problem is that poetry depends on tomlkit which depends on poetry as a build tool.

I would bet that this is a bug of pip and looking at their issue tracker I found the issue being reported there: https://github.com/pypa/pip/issues/5739. If you look closed you will see that also flit was affected but they solved it by moving their dependencies around a bit.

I'm not sure what a possible solution might be but maybe _vendoring_ of the dependencies might be a solution (eg. like pip does it). Or maybe there is a better way...

This kinda puts poetry (and actually all projects using it) as a blocker for me and prevents me from investigating further into poetry, which I really hope becomes more used in python ecosystem. This is also a reason why I'm posting this issue here, to try to bring attention to this kinds of bugs and to make poetry adoption easier.

In any case, thank you for you work.

[cs01]

Version 19 of pip coming January 2019 will support this. See https://github.com/sdispater/poetry/issues/321.

[garbas]

Will this also solve the problem of cyclic build dependencies? Like it is the case with poetry <> tomlkit.

[garbas]

@cs01 PEP 517 was already implemented for pip and also merged (https://github.com/pypa/pip/pull/5743) and I gave current pip from master a try and I get the same error.

The difference between #321 and using --no-binary :all: flag is that I require _all_ dependencies to be installed from source distribution, not only a specific dependency.

As much as I dislike vendoring I think that this is the only solution.

[bgreen-litl]

I am the maintainer of backoff and I am trying figure out if I need to do a workaround on my side or if this is something that I can expect to be fixed upstream.

I believe the issue is that there is a circular dependency: poetry and tomlkit depend on each other _at build-time_. It is not clear to me if this is a pip bug, or if circular dependencies just aren't supported when building from source.

I think @garbas is correct that vendoring (probably tomlkit in poetry?) would work around the problem.
If there is a possible fix on the pip side, it sounds like it hasn't happened yet.

Any thoughts on how to address this or if I should be looking for a workaround downstream?

[cs01]

I tried using the latest pip on a poetry-based project with no circular dependencies and it did not work. It looks like a bug in pip (or could at least be a nicer error message) but I suppose it's possible poetry's backend doesn't work. I filed an issue with pip https://github.com/pypa/pip/issues/6100.

[garbas]

@cs01 i think that issue you reference to is not really related to this one, except that both show that there are issues with pip and poetry.

The problem is that there is a circular dependency in poetry dependencies (poetry <-> tomlkit) which is usually not a problem since pip (or any other python package management tool) can handle this.

This becomes a problem when you are required to use source distributions (no wheels for me aka --no-binary :all: flag) and the fact that pip needs to install a package to inspect its dependencies, that means that it needs to first install build dependencies first.

So this is what happens when you run pip install backoff --no-binary :all::

• before installing backoff we need to install build dependencies
• poetry is a build dependency of backoff so we install it and we now see a list of poetry dependencies
• tomlkit is one of the poetry dependencies, but before installing it (to retrieve its dependency list) we need to install its build dependencies
• tomlkit's build dependency is poetry and here the the "circle" gets created.

[cs01]

@garbas I agree that I don't have circular dependencies. For some reason I assumed we were both hitting the same pip._vendor.pep517.wrappers.BackendUnavailable error, and that when it was fixed your circular dependencies would not be an issue. Maybe that was a bad assumption 😄. Can you paste the output of the error you're getting?

[garbas]

Any news on this? I can put time into solution once we decide what a solution is.

The error I get is:

$ ./env/bin/pip install backoff --no-binary :all:
Collecting backoff
  Using cached https://files.pythonhosted.org/packages/37/0b/e6d0e14cf95039fbc3b0b0e8370e1cec7bd6294d05dcdb669d0edbfde102/backoff-1.8.0.tar.gz
  Installing build dependencies ... error
  Complete output from command /tmp/pip/env/bin/python3.7m /tmp/pip/src/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-wowed2bl/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12:
  Collecting poetry>=0.12
    Downloading https://files.pythonhosted.org/packages/75/44/824cee51be9ee1f2c28042a58591a36cef0c85f688f7ddc5b088af26d50a/poetry-0.12.11.tar.gz (150kB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
      Preparing wheel metadata: started
      Preparing wheel metadata: finished with status 'done'
  Collecting cleo<0.7.0,>=0.6.7 (from poetry>=0.12)
    Using cached https://files.pythonhosted.org/packages/cb/9f/8dd57785485b2a2746edbe598a31b0852740a1ff91a2739c104b628c8520/cleo-0.6.8.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
      Preparing wheel metadata: started
      Preparing wheel metadata: finished with status 'done'
  Collecting shellingham<2.0,>=1.1 (from poetry>=0.12)
    Using cached https://files.pythonhosted.org/packages/04/47/c3f0b5fce44827d011f7f4b8831dd20984a228d7a7f46d4d45f1aaa6fd90/shellingham-1.2.8.tar.gz
  Collecting pkginfo<2.0,>=1.4 (from poetry>=0.12)
    Downloading https://files.pythonhosted.org/packages/6c/04/fd6683d24581894be8b25bc8c68ac7a0a73bf0c4d74b888ac5fe9a28e77f/pkginfo-1.5.0.1.tar.gz
  Collecting cachy<0.3,>=0.2 (from poetry>=0.12)
    Using cached https://files.pythonhosted.org/packages/68/ef/eabaf6c36bb87e4c1a74790b19e44064df25e09a55f191eb71ccd2b512ad/cachy-0.2.0.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
      Preparing wheel metadata: started
      Preparing wheel metadata: finished with status 'done'
  Collecting jsonschema<4.0,>=3.0a3 (from poetry>=0.12)
    Downloading https://files.pythonhosted.org/packages/b6/1f/b9941309e4b2b65b2ef34a8b36d1cbc53d2de6bd168d946042b5dc208ff6/jsonschema-3.0.0a5.tar.gz (119kB)
  Collecting tomlkit<0.6.0,>=0.5.1 (from poetry>=0.12)
    Using cached https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'error'
    Complete output from command /tmp/pip/env/bin/python3.7m /tmp/pip/src/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-uppwnvo8/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12a3:
    Collecting poetry>=0.12a3
      Downloading https://files.pythonhosted.org/packages/93/8a/270aa699a07405df9b260f9192eec043cf486b4b49fafd7ebcee09cb2cc5/poetry-1.0.0a2.tar.gz (154kB)
      Installing build dependencies: started
      Installing build dependencies: finished with status 'done'
      Getting requirements to build wheel: started
      Getting requirements to build wheel: finished with status 'done'
        Preparing wheel metadata: started
        Preparing wheel metadata: finished with status 'done'
    Collecting tomlkit<0.6.0,>=0.5.1 (from poetry>=0.12a3)
      Using cached https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz
    Exception:
    Traceback (most recent call last):
      File "/tmp/pip/src/pip/_internal/cli/base_command.py", line 154, in main
        status = self.run(options, args)
      File "/tmp/pip/src/pip/_internal/commands/install.py", line 315, in run
        resolver.resolve(requirement_set)
      File "/tmp/pip/src/pip/_internal/resolve.py", line 131, in resolve
        self._resolve_one(requirement_set, req)
      File "/tmp/pip/src/pip/_internal/resolve.py", line 294, in _resolve_one
        abstract_dist = self._get_abstract_dist_for(req_to_install)
      File "/tmp/pip/src/pip/_internal/resolve.py", line 242, in _get_abstract_dist_for
        self.require_hashes
      File "/tmp/pip/src/pip/_internal/operations/prepare.py", line 355, in prepare_linked_requirement
        with self.req_tracker.track(req):
      File "/nix/store/9vcy8i7mcwmcp486krc22v7851spcdqj-python3-3.7.1/lib/python3.7/contextlib.py", line 112, in __enter__
        return next(self.gen)
      File "/tmp/pip/src/pip/_internal/req/req_tracker.py", line 86, in track
        self.add(req)
      File "/tmp/pip/src/pip/_internal/req/req_tracker.py", line 55, in add
        % (link, fp.read()))
    LookupError: https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz#sha256=d6506342615d051bc961f70bfcfa3d29b6616cc08a3ddfd4bc24196f16fd4ec2 (from https://pypi.org/simple/tomlkit/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*) is already being built: tomlkit<0.6.0,>=0.5.1 from https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz#sha256=d6506342615d051bc961f70bfcfa3d29b6616cc08a3ddfd4bc24196f16fd4ec2 (from poetry>=0.12)

    ----------------------------------------
  Command "/tmp/pip/env/bin/python3.7m /tmp/pip/src/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-uppwnvo8/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12a3" failed with error code 2 in None

  ----------------------------------------
Command "/tmp/pip/env/bin/python3.7m /tmp/pip/src/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-wowed2bl/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12" failed with error code 1 in None

[eliasbrange]

The same happens when trying to build pytzdata from source. So it is not only connected to backoff.

Dockerfile:

FROM python:3.7.2-alpine

RUN pip3 install -U pip

RUN pip3 install --no-binary :all: pytzdata

Output:

#6 [3/3] RUN pip3 install --no-binary :all: pytzdata
#6       digest: sha256:a6ecf2defd61faa5cd6faf6702173cf6fb4d526b5c746a046b834ad256b416e8
#6         name: "[3/3] RUN pip3 install --no-binary :all: pytzdata"
#6      started: 2019-01-25 17:46:47.670230476 +0000 UTC
#6 1.094 Collecting pytzdata
#6 1.803   Downloading https://files.pythonhosted.org/packages/d9/27/04b261a2d24a2658159e502fb01e9579f7a42d7a1c51ca5023190dff126b/pytzdata-2018.7.tar.gz (278kB)
#6 2.251   Installing build dependencies: started
#6 33.26   Installing build dependencies: finished with status 'error'
#6 33.26   Complete output from command /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-5db63xt1/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12:
#6 33.26   Collecting poetry>=0.12
#6 33.26     Downloading https://files.pythonhosted.org/packages/75/44/824cee51be9ee1f2c28042a58591a36cef0c85f688f7ddc5b088af26d50a/poetry-0.12.11.tar.gz (150kB)
#6 33.26     Installing build dependencies: started
#6 33.26     Installing build dependencies: finished with status 'done'
#6 33.26     Getting requirements to build wheel: started
#6 33.26     Getting requirements to build wheel: finished with status 'done'
#6 33.26       Preparing wheel metadata: started
#6 33.26       Preparing wheel metadata: finished with status 'done'
#6 33.26   Collecting requests<3.0,>=2.18 (from poetry>=0.12)
#6 33.26     Downloading https://files.pythonhosted.org/packages/52/2c/514e4ac25da2b08ca5a464c50463682126385c4272c18193876e91f4bc38/requests-2.21.0.tar.gz (111kB)
#6 33.26   Collecting jsonschema<4.0,>=3.0a3 (from poetry>=0.12)
#6 33.26     Downloading https://files.pythonhosted.org/packages/2d/c3/0a1154fe821a62b4bd3d58b5863ff5e2314181c337c9ab8202f93b3be405/jsonschema-3.0.0b1.tar.gz (119kB)
#6 33.26   Collecting pyrsistent<0.15.0,>=0.14.2 (from poetry>=0.12)
#6 33.26     Downloading https://files.pythonhosted.org/packages/45/5a/a1a446eba4b5bf9f823fd863605df24327e49241d6b5c43d82b429228caa/pyrsistent-0.14.9.tar.gz (104kB)
#6 33.26   Collecting requests-toolbelt<0.9.0,>=0.8.0 (from poetry>=0.12)
#6 33.26     Downloading https://files.pythonhosted.org/packages/86/f9/e80fa23edca6c554f1994040064760c12b51daff54b55f9e379e899cd3d4/requests-toolbelt-0.8.0.tar.gz (196kB)
#6 33.26   Collecting tomlkit<0.6.0,>=0.5.1 (from poetry>=0.12)
#6 33.26     Downloading https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz
#6 33.26     Installing build dependencies: started
#6 33.26     Installing build dependencies: finished with status 'error'
#6 33.26     Complete output from command /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-0h7u3yj5/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12a3:
#6 33.26     Collecting poetry>=0.12a3
#6 33.26       Downloading https://files.pythonhosted.org/packages/93/8a/270aa699a07405df9b260f9192eec043cf486b4b49fafd7ebcee09cb2cc5/poetry-1.0.0a2.tar.gz (154kB)
#6 33.26       Installing build dependencies: started
#6 33.26       Installing build dependencies: finished with status 'done'
#6 33.26       Getting requirements to build wheel: started
#6 33.26       Getting requirements to build wheel: finished with status 'done'
#6 33.26         Preparing wheel metadata: started
#6 33.26         Preparing wheel metadata: finished with status 'done'
#6 33.26     Collecting cleo<0.8.0,>=0.7.2 (from poetry>=0.12a3)
#6 33.26       Downloading https://files.pythonhosted.org/packages/dc/53/0782ca2ad1c38114e54b4ef9894ed3055ece8c886091606a75d43b161fb0/cleo-0.7.2.tar.gz
#6 33.26       Installing build dependencies: started
#6 33.26       Installing build dependencies: finished with status 'error'
#6 33.26       Complete output from command /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-c_a34ziu/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12:
#6 33.26       Collecting poetry>=0.12
#6 33.26         Using cached https://files.pythonhosted.org/packages/75/44/824cee51be9ee1f2c28042a58591a36cef0c85f688f7ddc5b088af26d50a/poetry-0.12.11.tar.gz
#6 33.26         Installing build dependencies: started
#6 33.26         Installing build dependencies: finished with status 'done'
#6 33.26         Getting requirements to build wheel: started
#6 33.26         Getting requirements to build wheel: finished with status 'done'
#6 33.26           Preparing wheel metadata: started
#6 33.26           Preparing wheel metadata: finished with status 'done'
#6 33.26       Collecting cleo<0.7.0,>=0.6.7 (from poetry>=0.12)
#6 33.26         Downloading https://files.pythonhosted.org/packages/cb/9f/8dd57785485b2a2746edbe598a31b0852740a1ff91a2739c104b628c8520/cleo-0.6.8.tar.gz (60kB)
#6 33.26         Installing build dependencies: started
#6 33.26         Installing build dependencies: finished with status 'done'
#6 33.26         Getting requirements to build wheel: started
#6 33.26         Getting requirements to build wheel: finished with status 'done'
#6 33.26           Preparing wheel metadata: started
#6 33.26           Preparing wheel metadata: finished with status 'done'
#6 33.26       Collecting pyparsing<3.0,>=2.2 (from poetry>=0.12)
#6 33.26         Downloading https://files.pythonhosted.org/packages/b9/b8/6b32b3e84014148dcd60dd05795e35c2e7f4b72f918616c61fdce83d27fc/pyparsing-2.3.1.tar.gz (596kB)
#6 33.26       Collecting shellingham<2.0,>=1.1 (from poetry>=0.12)
#6 33.26         Downloading https://files.pythonhosted.org/packages/04/47/c3f0b5fce44827d011f7f4b8831dd20984a228d7a7f46d4d45f1aaa6fd90/shellingham-1.2.8.tar.gz
#6 33.26       Collecting cachy<0.3,>=0.2 (from poetry>=0.12)
#6 33.26         Downloading https://files.pythonhosted.org/packages/68/ef/eabaf6c36bb87e4c1a74790b19e44064df25e09a55f191eb71ccd2b512ad/cachy-0.2.0.tar.gz
#6 33.26         Installing build dependencies: started
#6 33.26         Installing build dependencies: finished with status 'done'
#6 33.26         Getting requirements to build wheel: started
#6 33.26         Getting requirements to build wheel: finished with status 'done'
#6 33.26           Preparing wheel metadata: started
#6 33.26           Preparing wheel metadata: finished with status 'done'
#6 33.26       Collecting html5lib<2.0,>=1.0 (from poetry>=0.12)
#6 33.26         Downloading https://files.pythonhosted.org/packages/85/3e/cf449cf1b5004e87510b9368e7a5f1acd8831c2d6691edd3c62a0823f98f/html5lib-1.0.1.tar.gz (252kB)
#6 33.26       Collecting jsonschema<4.0,>=3.0a3 (from poetry>=0.12)
#6 33.26         Using cached https://files.pythonhosted.org/packages/2d/c3/0a1154fe821a62b4bd3d58b5863ff5e2314181c337c9ab8202f93b3be405/jsonschema-3.0.0b1.tar.gz
#6 33.26       Collecting pkginfo<2.0,>=1.4 (from poetry>=0.12)
#6 33.26         Downloading https://files.pythonhosted.org/packages/6c/04/fd6683d24581894be8b25bc8c68ac7a0a73bf0c4d74b888ac5fe9a28e77f/pkginfo-1.5.0.1.tar.gz
#6 33.26       Collecting tomlkit<0.6.0,>=0.5.1 (from poetry>=0.12)
#6 33.26         Using cached https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz
#6 33.26       Exception:
#6 33.26       Traceback (most recent call last):
#6 33.26         File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 176, in main
#6 33.26           status = self.run(options, args)
#6 33.26         File "/usr/local/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 315, in run
#6 33.26           resolver.resolve(requirement_set)
#6 33.26         File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolve.py", line 131, in resolve
#6 33.26           self._resolve_one(requirement_set, req)
#6 33.26         File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
#6 33.26           abstract_dist = self._get_abstract_dist_for(req_to_install)
#6 33.26         File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolve.py", line 242, in _get_abstract_dist_for
#6 33.26           self.require_hashes
#6 33.26         File "/usr/local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 348, in prepare_linked_requirement
#6 33.26           with self.req_tracker.track(req):
#6 33.26         File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
#6 33.26           return next(self.gen)
#6 33.26         File "/usr/local/lib/python3.7/site-packages/pip/_internal/req/req_tracker.py", line 86, in track
#6 33.26           self.add(req)
#6 33.26         File "/usr/local/lib/python3.7/site-packages/pip/_internal/req/req_tracker.py", line 55, in add
#6 33.26           % (link, fp.read()))
#6 33.26       LookupError: https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz#sha256=d6506342615d051bc961f70bfcfa3d29b6616cc08a3ddfd4bc24196f16fd4ec2 (from https://pypi.org/simple/tomlkit/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*) is already being built: tomlkit<0.6.0,>=0.5.1 from https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz#sha256=d6506342615d051bc961f70bfcfa3d29b6616cc08a3ddfd4bc24196f16fd4ec2 (from poetry>=0.12)
#6 33.26   
#6 33.26       ----------------------------------------
#6 33.26     Command "/usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-c_a34ziu/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12" failed with error code 2 in None
#6 33.26   
#6 33.26     ----------------------------------------
#6 33.26   Command "/usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-0h7u3yj5/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12a3" failed with error code 1 in None
#6 33.26   
#6 33.26   ----------------------------------------
#6 33.28 Command "/usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-5db63xt1/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12" failed with error code 1 in None
#6    completed: 2019-01-25 17:47:21.288965909 +0000 UTC
#6     duration: 33.618735433s
#6        error: "executor failed running [/bin/sh -c pip3 install --no-binary :all: pytzdata]: exit code: 1"

[eliasbrange]

Also adding --no-binary :all: when installing only pendulum yields the above error. Removing --no-binary makes it work, which is fun since on alpine you install pendulum from source no matter what (no wheel there). So it seems to be some of the underlying build requirements (poetry and its dependencies).

[ymage]

Any progress ?

[reversefold]

It's even simpler than any of the other commands given. pip install --no-binary :all: tomlkit is enough to cause the LookupError in the above comments.

[reversefold]

It seems like, in lieu of a fix from pip, poetry needs to do something to solve this issue. The only real solution to me seems like vendoring any of poetry's dependencies which, in turn, use poetry for their builds. This likely also applies to any transitive dependencies of poetry too, which makes the problem not only more complicated but prone to changing as more packages switch to poetry.

I have found a workaround which works for my use-case, at least for now. I'm building wheels for everything in the poetry.lock during a build. If I pass each of the packages to --no-binary except for tomlkit, poetry, and pastel I still get builds from source for everything else. My project doesn't depend on those packages, they're only needed for poetry during the build. I imagine this might still end up causing a problem for any build which wants or needs to use --no-binary :all: and which also depends on any of these 3 packages.

Here's an example of what the (insanely long) command looks like in my case (built by a script):

poetry run pip wheel -w wheels --no-cache-dir --isolated --no-binary aniso8601 --no-binary argh --no-binary attrs --no-binary boto3 --no-binary botocore --no-binary CacheControl --no-binary cached-property --no-binary cachy --no-binary certifi --no-binary chardet --no-binary cleo --no-binary Click --no-binary colorama --no-binary coverage --no-binary dnspython --no-binary docopt --no-binary docutils --no-binary ec2-metadata --no-binary eventlet --no-binary fasteners --no-binary filelock --no-binary Flask --no-binary Flask-RESTful --no-binary greenlet --no-binary html5lib --no-binary ianitor --no-binary idna --no-binary importlib-metadata --no-binary itsdangerous --no-binary Jinja2 --no-binary jmespath --no-binary jsonschema --no-binary lockfile --no-binary magicreq --no-binary MarkupSafe --no-binary monotonic --no-binary more-itertools --no-binary msgpack --no-binary nose --no-binary nosecount --no-binary packaging --no-binary pathtools --no-binary pex --no-binary pkginfo --no-binary pluggy --no-binary psutil --no-binary py --no-binary pycryptodome --no-binary pylev --no-binary pyparsing --no-binary pyrsistent --no-binary python-consul --no-binary python-daemon --no-binary python-dateutil --no-binary pytz --no-binary PyYAML --no-binary requests --no-binary requests-toolbelt --no-binary reversefold.util --no-binary s3transfer --no-binary setproctitle --no-binary shellingham --no-binary six --no-binary tabulate --no-binary tenacity --no-binary toml --no-binary tox --no-binary urllib3 --no-binary virtualenv --no-binary watchdog --no-binary webencodings --no-binary Werkzeug --no-binary zipp aniso8601==8.0.0 argh==0.26.2 attrs==19.2.0 boto3==1.9.240 botocore==1.12.240 CacheControl==0.12.5 cached-property==1.5.1 cachy==0.2.0 certifi==2019.9.11 chardet==3.0.4 cleo==0.6.8 Click==7.0 colorama==0.4.1 coverage==4.5.4 dnspython==1.16.0 docopt==0.6.2 docutils==0.15.2 ec2-metadata==1.8.0 eventlet==0.25.1 fasteners==0.15 filelock==3.0.12 Flask==1.1.1 Flask-RESTful==0.3.7 greenlet==0.4.15 html5lib==1.0.1 ianitor==0.1.0 idna==2.8 importlib-metadata==0.23 itsdangerous==1.1.0 Jinja2==2.10.1 jmespath==0.9.4 jsonschema==3.0.2 lockfile==0.12.2 magicreq==0.6.0 MarkupSafe==1.1.1 monotonic==1.5 more-itertools==7.2.0 msgpack==0.6.2 nose==1.3.7 nosecount==3.7.2.59 packaging==19.2 pastel==0.1.1 pathtools==0.1.2 pex==1.6.11 pkginfo==1.5.0.1 pluggy==0.13.0 poetry==0.12.17 psutil==5.6.3 py==1.8.0 pycryptodome==3.9.0 pylev==1.3.0 pyparsing==2.4.2 pyrsistent==0.14.11 python-consul==0.7.2 python-daemon==2.1.1 python-dateutil==2.8.0 pytz==2019.2 PyYAML==5.1.2 requests==2.22.0 requests-toolbelt==0.8.0 reversefold.util==3.0.1 s3transfer==0.2.1 setproctitle==1.1.10 shellingham==1.3.1 six==1.12.0 tabulate==0.8.5 tenacity==5.1.1 toml==0.10.0 tomlkit==0.5.5 tox==3.14.0 urllib3==1.25.6 virtualenv==16.7.5 watchdog==0.9.0 webencodings==0.5.1 Werkzeug==0.16.0 zipp==0.6.0

[reversefold]

I have also been commenting on the pip issue https://github.com/pypa/pip/issues/5739 and thinking more on this and it seems to me like a mis-feature of pip causing this problem (although I could be convinced otherwise if there is documentation saying otherwise) that is requiring source-based installs of build-time dependencies, especially when those dependencies are already installed and available.

In the case of the command I gave above I'm not only running the pip command via poetry, in a poetry created virtualenv, but poetry is itself already installed in that virtualenv, so poetry should be already-available to pip when it comes to building packages that require poetry at build-time.

[JBKahn]

Like @reversefold I posted on the other thread as well, after just having this issue brought to my attention when I switched two of my projects over to poetry. I think that trying to selectively vendor in dependancies would be hard to manage and more error prone than simply vendoring in all the dependancies.

While I say that, I have no idea how difficult vendored dependancy management is. Or what the feeling of the maintainers are on doing that. I'm curious to hear thoughts from someone on the team.

[zgoda]

This is still not resolved on pip side as of 20.0.1.

[albertomurillo]

Any plan to fix this?

[nambrosch]

i believe this is what's preventing me from installing apache-airwatch using puppet-python

[bkyryliuk]

Any update here?

[abn]

The upcoming release should provide a solution for this. With the next release poetry-core will become available. Applying the following patch to backoff should enable this to work with newer pip versions. There is a bug in pip==19.1.1 that does not setup the PEP517 environment correctly, hence failing --no-binary :all: builds. The following fix was verified to work with pip==20.0.2

diff --git a/pyproject.toml b/pyproject.toml
index 3469bf2..a42652d 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -38,5 +38,5 @@ pytest-cov = "^2.6"
 pytest-asyncio = {version = "^0.10.0",python = "^3.5"}

 [build-system]
-requires = ["poetry>=0.12"]
-build-backend = "poetry.masonry.api"
+requires = ["poetry-core>=1.0.0a5"]
+build-backend = "poetry.core.masonry.api"

Note that this uses an alpha release of poetry-core.

[JBKahn]

That's awesome, anything I can do to help this move forward?

[fredrikaverpil]

@abn This is fantastic. Are there any news on when poetry-core will be available, or if it can already be used somehow?

This worked on my Windows (git bash) machine but crashed on my Linux RHEL 7 machine with Python 3.7:

[build-system]
requires = ["poetry-core>=1.0.0a8"]
build-backend = "poetry.core.masonry.api"

[fredrikaverpil]

I found out why it was crashing: https://github.com/python-poetry/poetry/issues/2626