Pip-tools: pip-compile fails on packages using poetry

[eliasbrange]

pip-compile fails if a package uses poetry which is PEP 517 compliant (https://poetry.eustace.io/docs/pyproject/#poetry-and-pep-517).

Simply doing pip install pendulum does not require e.g. poetry to be installed. However, installing it made pip-compile work which I'm not sure we as consumers of the package in question should need to do.

Environment Versions

1. Alpine 3.8
2. Python version: 3.7.2
3. pip version: 19.0.1
4. pip-tools version: 3.3.1

Steps to replicate

# DOES NOT WORK

FROM python:3.7.2-alpine3.8

RUN pip3 install --upgrade pip==19.0.1 setuptools pip-tools>==3.3.1

RUN echo "pendulum" > req.txt

RUN pip-compile --generate-hashes req.txt


# WORKS

FROM python:3.7.2-alpine3.8

RUN pip3 install --upgrade pip==19.0.1 setuptools pip-tools>==3.3.1

RUN echo "pendulum" > req.txt

RUN pip3 install poetry

RUN pip-compile --generate-hashes req.txt

[vphilippon]

Hi @eliasbrange, thanks for the report.

I'm unfamiliar with PEP517 (/todo Read that thing!). I would naively wish for pip to handle the mechanic.

One thing to note, is that this specific install seem to work with pip==18.1.

Also, there's an issue on the subject in the pendulum project, about issues with pip>=19.0.1:
https://github.com/sdispater/pendulum/issues/335

Additionally, there seem to be some issues related to pip 19.0.1 and poetry compatibility (I don't have quite the time to dig into these at the moment):
https://github.com/sdispater/poetry/issues

So there's the possibility that this is one of the upstreams issue, or that there's something more to do in pip-tools now to initialise some PEP517 mechanic (hunch from the pip._vendor.pep517.wrappers.BackendUnavailable exception raised, as seen when using --verbose)

Any help to untangle this would be greatly appreciated!

[eliasbrange]

Trying to investigate a bit further. Noticed that pip-compile succeeds with pytzdata which also uses poetry as a build system: https://github.com/sdispater/pytzdata/blob/master/pyproject.toml

[atugushev]

It raises here:
https://github.com/pypa/pip/blob/master/src/pip/_vendor/pep517/_in_process.py#L36

with an error:

No module named 'poetry'

For some reason it wants poetry. Keep digging.

[eliasbrange]

Which is described in pendulums pyproject.toml as per PEP517/PEP518.

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

[atugushev]

Right!

It doesn't go here:
https://github.com/pypa/pip/blob/master/src/pip/_internal/operations/prepare.py#L119

because should_isolate is False (because build_isolation is False). So it cannot build requires. I've hardcoded should_isolate=True and pip-compile went fine.

[atugushev]

The main problem is here https://github.com/jazzband/pip-tools/blob/5029fcc4298def004a3d286aa3adfc312acb4efd/piptools/repositories/pypi.py#L164

[eliasbrange]

So that should probably be an optional flag to let consumers choose between installing required packages themselves, or doing isolated builds (which will take forever if there are lots of packages using different build systems).

[atugushev]

let's investigate how pip download pendulum decides to build it in isolation mode. Ah it seems pip download builds in isolation mode by default.

[atugushev]

So that should probably be an optional flag to let consumers choose between installing required packages themselves, or doing isolated builds (which will take forever if there are lots of packages using different build systems).

I guess so. Build isolation could be set on by default and an option to disable it:

  --no-build-isolation        Disable isolation when building a modern source

[atugushev]

Gentle ping to @vphilippon.

What do you think about this? Since pip builds sources in isolation by default, perhaps we should consider to do it as well. But it could be a "breaking change".

[vphilippon]

I went and read https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support to catch up a bit on the subject of build isolation and what it implies.

It seems pip enables build isolation since its first inclusion (https://github.com/pypa/pip/commit/c08d4cc806897de687785c723ac7b767c1bc2711)
So it seems reasonable to enable it by default in pip-tools too.

As for the "breaking change" nature of this change, I don't mind as it's meant to adapt to the "new reality and expectation" of the python packaging ecosystem.

The Grand Question: Do you think that:

A) It's actually a pip-tools bug we're only catching now, that having isolation disabled most likely caused more pain to users since pip 10.0.0 release, and that enabling it will cause little surprise/impact to users? If so, include in our next feature/bugfix.

B) Enabling isolation by default may have multiple impacts that will require careful attention for a substantial part of the userbase, or could bring odd behavior in pip-tools itself, and thus should be announced strongly? If so, time for a 4.0.0 (which is ok, it's just numbers! Let's take advantage of Semantic Versionning to raise user awareness on this, as we can).

@atugushev @eliasbrange Voice your opinion, you likely have more info on the latest feature of pip than I do right now.
Also pingning @techalchemy and @pradyunsg for input on the impact/risk, if you have time 😄 .

[atugushev]

A) I don't think it is a bug, "no isolation" might be chosen for a special reason. Would be appreciated to get a comment from @techalchemy here.

B) I totally agree. Users has to be warned if default behavior suddenly changes.

There is a soft way though. We could introduce --no-build-isolation and --build-isolation flags, where --build-isolation is default.

[pradyunsg]

Seems related to pypa/pip#6164.

I'm inclined to think poetry needs to update its backend since it's not compatible with pip's PEP 517 implementation as things stand.

[pradyunsg]

As for rollout of isolation to end users, @techalchemy can probably explain in better terms. :)

[jkp]

Just ran into this with a package of mine that uses pyproject.toml to declare some build system requirements. Forcing build isolation as mentioned above fixed the issue.

Understand that a breaking change is problematic - is there a plan to at least add an optional flag to allow users to specify this behaviour if they need it?

[atugushev]

Hello @jkp,

I've just prepared a PR #758 for the optional flag you've mentioned. I hope it'll be merged soon.

[atugushev]

@vphilippon i think it's time to provide an optional flag to allow enabling/disabling build isolation. "No build isolation" still be the default behavior, but we could change it in a future releases (as a breaking change). WDYT?

[techalchemy]

I am super late responding to this, but I'm not really sure what toggling this does for folks. It's pretty desirable to turn on for installation itself, but can cause issues with resolution (though which issues specifically I couldn't tell you at this point).

We have had a lot of problems making pip-tools play nicely with pip during the resolution of editable and local packages or remote files (which pip-tools does not resolve by default). I recently implemented a complete workaround for this in requirementslib which now fully leverages pep517 with a fallback to setuptools and is extremely effective at resolving these dependencies without issue.

In the specific case of this issue, I am fairly sure you will need build isolation turned on if you want pep517 builders to work for resolution, but that's new territory. Like I explain above I am currently ignoring this question entirely and just invoking pep517 myself. The issues I've run into mainly revolve around pip's temporary directory cleanup during the isolated build process, so by handling it myself I am able to achieve build isolation while bypassing the weird issues. Hope this helps.

[RobbieClarken]

I was getting the following error when running pip-compile on a project that installed a package from a submodule with a pyproject.toml file (the pyproject.toml was only being used to configure black, not as a replacement for setup.py).

pip._internal.exceptions.InstallationError: Command "/Users/clarkr/.pyenv/versions/3.6.3/bin/python3.6 /Users/clarkr/.pyenv/versions/3.6.3/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/gc/lwr8t4zs14g6y7t401x8n315xkgl13/T/tmpr6_ewj7j" failed with error code 1

Updating requirementslib fixed the issue:

python3 -m pip install -U requirementslib

[energizah]

I'm inclined to think poetry needs to update its backend since it's not compatible with pip's PEP 517 implementation as things stand.

@pradyunsg Would you mind elaborating on this or possibly filing an issue on Poetry, so we understand what needs to be fixed?