Pip: PEP-517 implementation for pip

[gaborbernat]

This is an issue ticket to track discussion and planning on adding PEP-517 support. Pip 10 does support PEP-518 (with the limitation that the build tool must be a wheel - this avoids the issue of circular dependencies). The next step now is to support PEP-517. All I describe here is the result of my discussion with @dstufft at PyCon 2018 sprints.

pip for PEP-518 does (for a given package to build):

• if an sdist is acquired extract it to a tree source,
• get the build tool and build requirements from pyproject.toml (must contain setuptools and wheel),
• for the current build environment (which is a temporary directory) install the build tool and requirements (this happens by invoking pip via a subprocess),
• invoke the build command by using this temporary folder to build a wheel,
• install then the wheel.

Phase 1: pip install in a PEP-517 and 518 would do:

• if an sdist is acquired extract it to a tree source,
• get the build tool pyproject.toml,
• for the current build environment (which is a temporary directory) install the build tool (this happens by invoking pip via a subprocess),
• use get_requires_for_build_sdist (or it's wheel counterpart) to get the build requirements (this happens via invoke python within a subprocess of the build environment),
• for the current build environment (which is a temporary directory) install the build requirements (this happens by invoking pip via a subprocess),
• invoke the build command by using this temporary environment to build a wheel (build_wheel).
• install then the wheel.

Phase 2: allow pip to build packages for distribution - pip build

• this follows the same paths as above with the sole difference that allows for the user to select either wheel or sdist, and that it's invoked from the cli (e.g. pip build . --sdist.

For phase 1 most of the things are already implemented in https://github.com/pypa/pep517 by @takluyver. It even follows the method pip uses to implement PEP-518. I suggest we just take that package as a vendored package within pip, and maybe make slight alternation to it where required. Once that's done we can go onto phase 2.

Please show your take on this if you have any obligations, otherwise I'll try to create a PR for this.

[gaborbernat]

@pradyunsg so how's triaging works?

[pradyunsg]

@gaborbernat Hey! So, there's of a lot of triage to do -- using that label to signal issues/PRs that need a maintainer to respond to. :)

---

Sure! Let me think out loud how things should work... I feel like I have so much here.

Phase 1

• sdist -> wheel -> installed when given an sdist is obviously what pip should do.
  
  
  
  • If we have an sdist, we should unpack it to a temporary directory and proceed.
  
  
• local-dir -> ? -> installed is more interesting. :)

Now, I do think having a discussion early about how things should be structured/implemented would be nice, before a PR. Mostly since I've spent a bit of my time thinking about #5051 (although that's a blue whale issue). Basically, I would prefer a clean separation between the current flow and a PEP 517/518 flow.

• What about non-PEP 517/518 source trees/sdists?
  
  
  
  • Their process/building shouldn't be affected. That is important for backward compatibility.
  
  
  • This can be tricky to verify -- #5051 would be useful here but hey, that's a blue whale.
  
  
• Should pip try to do local-dir -> sdist -> wheel unless build_sdist screeches in pain?
  
  
  
  • Yes.
  
  
  • I have a preference for the 2 step approach so as to minimize the number of surprises for people. Additionally, both steps should have separate environments for building (or the same one "cleared" and reused).
  
  
  • It ensures that what's in the sdist, is enough to build a wheel -- integrity of sdists is "verified" in a way. (could have used better words?)
  
  
  • But we get dependency metadata only when building wheels. :(
  
  
  • I'm gonna shelve this thought and make a lot of noise when the time comes for this. :P
  
  

Phase 2

I'm noting these here now but think these should be discussed after Phase 1 is done.

• Does pip build mean we deprecate pip wheel?
  
  
  
  • Yes. I like the spelling of pip build. :)
  
  
• How does pip build work?
  
  
  
  • local-dir -> sdist -> wheel falling back to local-dir -> wheel
  
  
  • Yes.
    
    
    
    
    
    • The _regular_ flow (as per above), which should be the default
    
    
    
  
  
  • pip build .
  
  
  • sdist -> wheel
  
  
  • Yes.
  
  
  • pip build foo-1.0.tar.gz
  
  
  • local-dir -> sdist
  
  
  • Yes.
  
  
  • pip build --sdist-only .
  
  
  • local-dir -> wheel
  
  
  • Yes.
    
    
    
    
    
    • Useful for build backends to test themselves (to actually verify that going directly to wheel gives the same wheel as going via sdist).
    
    
    
  
  
  • pip build --wheel-only .
  
  

---

Then, PEP 518 support needs to be improved/expanded and we have #5286 + #5336, which if I understand correctly, would get us there. (thanks to @benoit-pierre)

All I describe here is the result of my discussion with @dstufft at PyCon 2018 sprints.

You got to meet him! :O

maybe make slight alternation to it where required.

I do prefer that the slight alterations happen upstream (not modifying vendored packages unless absolutely needed). :)

---

self-note: We will need to be building wheels (or just wheel metadata) during resolution. zazo won't care about this -- it's designed to defer this to when integrating into pip; so let's look into this then. :)

local-dir -> sdist/wheel:

• create a build environment (buildenv)
  
  
  
  • do not copy the local-dir like pip does today
  
  
• install build-system.requires in buildenv
• "load" build-system.build-backend
• install build_backend.get_requires_for_build_sdist/wheel in buildenv (optional)
  
  
  
  • subprocess
  
  
• build_backend.build_sdist/wheel
  
  
  
  • subprocess
  
  
• Proceed to with the generated sdist/wheel

[gaborbernat]

@pradysung do we have an official definition of what an sdist is? I am personally not convinced that we do need the sdist step always. Seems something that could potentially slow us down and redundant. Any reason you want to Keep sdist around in all builds?

[pradyunsg]

@gaborbernat sdist is short for source distribution. The reason to keep it is as an integrity check of the installability of a source distribution. It should be possible to go from a source distribution to an installed package. Ensuring that by default makes sense to me, when installing from local directories.

[gaborbernat]

@pradyunsg integrity check? That just sounds to me like we're trying to validate that the build backend does its job correctly, or that the user does the packaging correctly. I would argue that this would violate separation of concern. Shouldn't pip take care of the package management itself, and leave the build to the responsibility of the build backend, as outlined in PEP-517?

[pradyunsg]

sdist is short for source distribution.

This was wrong from my end to think you didn't know that. Apologies. As for the format of sdists, it's mostly a defacto based on what setuptools generates right now. Moving forward, PEP 517 has a specification: https://www.python.org/dev/peps/pep-0517/#source-distributions

integrity check? That just sounds to me like we're trying to validate that the build backend does its job correctly, or that the user does the packaging correctly.

Mostly the latter. That said, I'm not sure how I feel strongly about this. I'm curious what other think about this.

[pfmoore]

From what I recall, this (whether builds should be done "via sdist" or not) was extensively debated in the PEP 517 discussions. I don't recall the answer, but I think it would be worth researching the discussion threads before starting any implementation. I'm not going to try to summarise what I recollect, as I was one of the participants in the thread, so I'm not unbiased, and I honestly don't remember what the ultimate consensus was.

Yes, I know the PEP should probably specify things like this, but there was way too much debate and controversy to realistically expect that in all cases.

One thing I will say is that we do not want to copy the source tree and build from the copy. That's the source of all the "pip takes too long to build from source" issues we have, because we can't know whether we need to copy (e.g.) .git directories, or huge data files. We should either build in place, or build from sdist, or there's an alternative efficient approach in the PEP that I don't recall right now :-)

[RonnyPfannschmidt]

@pfmoore pip could introduce a tool.pip section where projects can specify whether its safe to copy them among other things

[pfmoore]

But the copy is used when building from a user's working directory - the project cannot know what additional stuff the user might have stored there.

I don't think there's anything new going on here (at least it sounds to my recollection like the sort of thing we discussed during the development of PEP 517). I'm not going to rehash that debate, so I'll just say anyone interested should read the mailing list archives for context. (Apologies if you were involved in those discussions at the time @RonnyPfannschmidt - once again I simply don't recall).

[RonnyPfannschmidt]

pip copying breaks stuff for me on regular basis tho ^^

[takluyver]

From what I recall, this (whether builds should be done "via sdist" or not) was extensively debated in the PEP 517 discussions. I don't recall the answer, but I think it would be worth researching the discussion threads before starting any implementation.

I am certainly not unbiased, but the conclusion as I remember it was that:

• It's recommended to build via an sdist if possible
• But frontend tools should be prepared to fall back to building a wheel directly if building an sdist fails - e.g. because flit depends on a VCS to build an sdist, and that may not be available in an install context.

[pfmoore]

@takluyver As one of the PEP authors, your bias is authoritative, IMO :-) But either way, my recollection matches yours.

Actually, looking at the PEP:

To ensure that wheels from different sources are built the same way, frontends may call build_sdist first, and then call build_wheel in the unpacked sdist. But if the backend indicates that it is missing some requirements for creating an sdist (see below), the frontend will fall back to calling build_wheel in the source directory.

And certainly my view remains that pip should choose to call build_sdist first. So recommendation or not, that's what I'd want pip to do. It's not so much about an "integrity check", as it is about ensuring that pip install . produces the same result as building a sdist manually, then pip install <the-sdist>. So the developer doesn't get different results than his end users (who will be getting the sdist, not the developer's source tree).

There was a lot of debate about whether the PEP could (or even should) require that building a wheel directly had to "do the same as" building a sdist and then building a wheel from that sdist. It basically didn't reach any particular conclusion, not least because the question is rather ill-defined. IMO, the point of pip always building a sdist first if it can is precisely to avoid needing to address that question.

[pradyunsg]

It's not so much about an "integrity check", as it is about ensuring that pip install . produces the same result as building a sdist manually, then pip install .

This is what I meant by "integrity check"; looking back at that comment, I guess I could have worded it better.

---

ISTM that both @takluyver and @pfmoore have the same position as me on this:

• try local-dir -> sdist -> wheel unless sdist creation is not possible.

[pradyunsg]

@gaborbernat This issue has been idle for over a month; do you have any outstanding concerns here or the bandwidth to take this on in the near future?

[pfmoore]

I'd like to pick this up (I'd said back at the release of pip 10 that I was hoping to work on this, but hadn't got around to it so far).

My initial plan is as follows:

1. Implement a separate "pep517" package that handles the infrastructure of calling PEP 517 hooks.
2. Vendor this in pip and use it to implement the "build frontend" side of things.

I wanted to put an initial comment here as most of the first stage will be happening independently of pip, and as a result won't be immediately visible here. Commenting here at least means people are aware of this, and don't duplicate work.

[gaborbernat]

@pfmoore I feel like step 1 us already completed via https://github.com/pypa/pep517 can't you use that?

[takluyver]

That's what that package was intended for, so if it's not the right shape to be used from pip, feel free to modify it.

[pfmoore]

Ah, I didn't even know that existed - thanks!

[pfmoore]

Looking at the basic wheel building code in pip, it seems to me that the "easy" place to start with this is pip._internal.wheel.WheelBuilder, method __build_one. I can rename __build_one as _build_one_legacy and add a _build_one_pep517 to do the same but for projects with PEP 517 information. That doesn't seem like it would be too hard to do, and would cover the basic case.

Moving on from there, I'd need to look at the path from requirement to install, to ensure that all requirements that have pyproject.toml go through the new route - source tree or sdist -> wheel -> install. It's at this point that we'd be eliminating all of the old "direct install" routes (or at least, marking them clearly for ultimate removal once we've switched fully to the sdist -> wheel route).

Once that's done, we have the basic process sorted, and it's just the corner cases (editable installs, ...) that need tidying up.

The most interesting bit will be writing tests. I'm planning on creating a dummy backend in our test directory that can be used to exercise the new code. That way we can check that all the steps are working in isolation from any actual backend processing.

By the way, there's a question regarding setuptools (well, a couple):

1. Does setuptools have a PEP 517 backend yet? Is anyone working on one, and if not, what's the plan for that?
2. If a project wants to use the setuptools PEP517 backend, do they have to explicitly request it in pyproject.toml? If we don't do anything, it seems to me that there's no incentive for setuptools-using projects to switch from using the legacy (no [build-system].build-backend key) route, to the PEP 517 backend route - however, there's also no provision in any of the PEPs for a default backend.

My suggestion would be:

1. If pyproject.toml has [build-system].build-backend, then it's a PEP 517 style source tree and we're done.
2. If pyproject.toml exists, but there's no build-backend key, but [build-system] exists, then it's not covered by PEP 517, so it's up to pip how we handle that case. I suggest that we initially raise a deprecation warning saying that we're currently invoking setup.py directly, but we will switch to PEP 517 processing assuming the setuptools backend in due course. Then, once the deprecation period has passed, we do precisely that.
3. If pyproject.toml does not exist (or has no [build-system] section) we issue a deprecation warning saying that we'll be moving to processing using PEP 517 (isolation, setuptools backend). Follow up by doing that, although we may want to have a longer deprecation period for that case.

We can't do (2) and (3) until setuptools provides a PEP 517 backend, of course.

[gaborbernat]

Everything has been implemented on setuptools side under https://github.com/pypa/setuptools/blob/master/setuptools/build_meta.py. In theory, we could demand requirement to specify the backend. Alternatively similarly how we default to setuptools, wheel for requires we could default to setuptools backend?

This will work (tested inside toxn when I was mocking around with it):

[build-system]
requires = ['setuptools >= 38.2.4']
build-backend = 'setuptools.build_meta'

A flit example for comparision:

[build-system]
requires = ["flit"]
build-backend = "flit.buildapi"

[pfmoore]

Fantastic, thanks for the confirmation.

Alternatively similarly how we default to setuptools, wheel for requires we could default to setuptools backend?

Yeah, that's what I was suggesting we do, it's just that unlike the ["setuptools", "wheel"] default, that's not something explicitly documented in the PEPs (in fact, we'd need to require a minimum setuptools version as well, so we're going to vary slightly from PEP 518 anyway).

By the way, you'd originally said that you wanted to look at this and now I've picked it up - are you OK with that, or did you still want to work on it? I don't want to tread on your toes here :smile:

[dstufft]

I think that as long as the setuptools build back end is calling setup.py and isn’t redirecting around it, we can just default to calling it. It should be the same as pip calling the setup.py then.

We maybe want a check that ensures the setuptools is new enough though. Possibly also a flag (at least for a little bit) that allows forcing the legacy behavior.

Sent from my iPhone

On Jul 24, 2018, at 6:03 AM, Bernát Gábor [e-mail] wrote:

Everything has been implemented on setuptools side under https://github.com/pypa/setuptools/blob/master/setuptools/build_meta.py. In theory, we could demand requirement to specify the backend. Alternatively similarly how we default to setuptools, wheel for requires we could default to setuptools backend?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[gaborbernat]

@pfmoore no worries :+1: I'm glad we're making progress on this. I'm caught in a few other projects so actually thanks for taking up this! :bowing_man:

38.2.5 is the first setuptools containing build_meta there, so that's probably sensible default.

And yes setuptools executes setup.py, see https://github.com/pypa/setuptools/blob/master/setuptools/build_meta.py#L64.

[pfmoore]

I just spotted a ping on #4383, which is about passing options to the setuptools egg_info command. In PEP 517 terms, calling egg_info will be delegated to the setuptools backend, and PEP 517 expects any custom settings to be sent explicitly via config_settings.

I've not even thought about the consequences of that yet, but I'm wondering whether we should add a label that we can use to tag issues like this, that will need to be rethought under PEP 517? Maybe "pep517 impact" (@pradyunsg I don't know what your single-letter prefixes mean, can you suggest the best prefix for this?). The implication would be "This issue will no longer be applicable under PEP 517, it will need to be reconsidered from the point of view of how it would work under a PEP 517 backend" and we'd expect input from the person who raised the issue on how they'd expect things to work under PEP 517. I'd expect to flag pretty much everything that refers to setuptools-specific behaviour, for a start. (In the specific example of #4383, for instance, I'd be inclined to say "config_settings handles this, nothing to do here" and it would need additional input from the submitter to clarify their requirements if that wasn't sufficient).

Comments?

[gaborbernat]

Saying that seems reasonable once PEP-517 support lands in pip; follow the logic of separation of concerns.

[pradyunsg]

@pradyunsg I don't know what your single-letter prefixes mean, can you suggest the best prefix for this?

"C" -- it's short for category; I'm basically using it for things like proxy, dependency resolution etc.

[pfmoore]

Under PEP 517, the default installation path for anything that's provided as source code only will be:

1. Create an isolated build environment.
2. Call the build_wheel hook for the backend.
3. Install the resulting wheel.

I'm ignoring the question of build-via-sdist for now, as that's not relevant here, and I plan on leaving that change as a follow-up discussion. Editable installs will still go via the existing code paths, as PEP 517 very sensibly deferred considering them.

Even setuptools builds can go via this route, as setuptools has a PEP 517 backend.

There's a question over to what extent we choose to switch existing projects to the PEP 517 code path. As with PEP 518 and isolation, we can choose our own path here - projects without pyproject.toml, or with pyproject.toml but no build-backend key are explicitly out of scope of PEP 517, so it's basically a decision for pip.

I propose a similar solution to what we've done for build isolation:

1. Projects without pyproject.toml get the existing behaviour.
2. Projects with pyproject.toml and build-backend get PEP 517 (clearly).
3. For projects with pyproject.toml but no build-backend, but which have a setup.py file, I propose we default to the setuptools backend, specifically to make sure that the new code paths get a decent level of exposure. If doing this causes problems for user builds, that's a bug either in the setuptools backend, or in our PEP 517 code, and better to expose those bugs sooner rather than later.
4. Projects with no build-backend and no setup.py, we have no way of knowing how to build them, and we reject them.

We should include a --use-legacy-setuptools flag to revert to the existing code paths in case (3). It might also absorb the --no-build-isolation flag - does anyone think there's a need for keeping the two separate?

I'd like to add a deprecation warning in case (1), saying that we'll move those projects over to behaviour (3) at some point. I'm not sure if that's something we should do as part of the PEP 517 implementation, or leave it for a release or so. I also don't have a feel yet for how long a deprecation is appropriate.

One point here - at the moment there are various conditions that will cause pip to skip building wheels, and fall back to a direct setup.py install. We can't do that under PEP 517, as there's no "direct install" hook, so if we are using PEP 517 we should simply fail if wheel building fails (even if we're using the setuptools backend via case 3 above). This will be a change in behaviour[1], but the same as with build isolation, we can tell people to use --use-legacy-setuptools as a workaround in the short term (and longer term we will need to look with them at why they can't build wheels and address the underlying problem).

[1] Cases where we might fail under PEP 517 where we currently fall back to setup.py install are mostly around when we can't use the wheel cache, for example if there's permission problems, or the user disabled it.

Does this all sound reasonable? (And does anyone have a feel for what sort of timescale might be reasonable for switching projects without pyproject.toml to go through the PEP 517 setuptools backend?)

[dstufft]

I think that sounds reasonable, to answer your questions:

1. I think that we should keep --use-legacy-setuptools and -no-build-isolation as separate flags. At some point I believe we're going to want to get rid of the older code path, and only support building via the PEP 517 backends. However, I can always see a use case for being able to build without isolation. I would maybe bikeshed the name a little bit, only because it sounds like we're not using setuptools anymore, instead of it just being a different way of invoking setuptools... but I don't really have a suggestion for a better name either.
2. I would wait to start throwing up deprecation warnings for (1), if only because I think it would make sense to let the new feature bake for a release or two to make sure there's nothing show stopping before we start throwing errors. One thing it might be nice is to add a --no-use-legacy-setuptools flag that forces (3) now, so we can have early adopters start opting in to it to see if they can shake out any issues.

[nicoddemus]

I think that we should keep --use-legacy-setuptools and -no-build-isolation as separate flags.

I second that, for example conda build probably will always use --no-build-isolation because it manages the build environment itself. --use-legacy-setuptools is a different
functionality, and can be dropped in the future as @dstufft commented.

[pfmoore]

@dstufft how about --[no-]use-pep517 as the flag? The default would depend on the project (basically if you have pyproject.toml then true, else false), but the user could override either way.

[nicoddemus]

conda manages build dependencies and build environment, so I think --no-use-pep517 would work.

cc @ocefpaf

[pfmoore]

@nicoddemus Be careful here, you might have misunderstood me. Based on your earlier comment, I'm expecting to retain --no-build-isolation. But --no-use-pep517 would not be a long-term thing - it's intended as a temporary transition flag to give an escape hatch for projects where (for some reason) using the setuptools PEP 517 backend doesn't work, but pip's current direct use of setup.py does. But that's a bug (in the project, or pip, or the setuptools PEP 517 backend) so we'd expect it to get fixed.

The long term goal is to only offer PEP 517 compliant building from pip. We'll retain --no-build-isolation on the basis that PEP 517 allows frontends to not isolate (in this section isolation is noted as a SHOULD). We're arguably deviating from the PEP in that if we don't isolate, we also don't install build requirements (in this section making requirements available is described as something frontends MUST do) but I'm OK with taking the view that we do this by assuming the user isn't lying when they supply the --no-build-isolation flag (which implies "I'll sort out the dependencies manually").

Once we can, we'll remove the direct setup.py code and the --[no-]use-pep517 flag altogether.

[nicoddemus]

I see, thanks for the explanation @pfmoore, I did misunderstand you.

--no-build-isolation is fine for conda's needs, AFAIU.

[msarahan]

conda-build is using the PIP_NO_BUILD_ISOLATION environment variable with the counterintuitive, but seemingly correct value of False to disable build isolation. We'll plan on sticking with that based on this thread, and ignore the --no-use-pep517.

[pfmoore]

To clarify further my assumption on how the two flags will behave, first note that a valid project must have at least one of setup.py and pyproject.toml.

• The following cases must use PEP 517: --no-use-pep517 is invalid, and --use-pep517 is the default.

  • Projects with pyproject.toml and no setup.py
  • Projects where pyproject.toml has a [build-system] section containing build-backend.

• The following cases can choose to use PEP 517 or not:

  • Projects with only setup.py (in this case --no-use-pep517 is the default).
  • Projects with both pyproject.toml and setup.py, but pyproject.toml does not contain a build-backend value (in this case --use-pep517 is the default).

When there's a choice over using PEP 517,

• With --no-use-pep517 we behave just the way we do right now (direct calls to setup.py). If we're isolating and there are no build requirements, we assume requirements of ["setuptools", "wheel"]. If we're not isolating, or there are explicit build requirements, it's an error if the build environment (once created) does not include the assumed requirements.
• With --use-pep517, we behave as if there were a pyproject.toml with build-backend = "setuptools.build_meta". For build requirements, we do the same as in the --no-use-pep517 case but our assumed requirements are ["setuptools>=38.2.5", "wheel"] (so that we have a version of setuptools with the backend).

Note that there's no case where --use-pep517 is invalid. So at the point where we want to make the switch, we can start by making --use-pep517 the default everywhere, and then later we just remove the option (and the code supporting --no-use-pep518).

Also note that if a project has a pyproject.toml file, it will get full PEP 517 behaviour by default. This follows the approach we took with PEP 518, where the new behaviour was triggered by the existence of pyproject.toml, so it's a tested route for introducing the new PEPs.

Now I just hope that I can actually implement all that! :smile:

[gaborbernat]

As tox maintainer I approve your plan 😁 will work just good for us what you described.

[dstufft]

The one sticking point is going to be when we have an older version of setuptools. Either manually in the build requires or implicitly without isolation.

This I think should just be an error when routing through the pep517 code path.

Sent from my iPhone

On Jul 25, 2018, at 5:11 PM, Bernát Gábor [e-mail] wrote:

As tox maintainer I approve your plan 😁 will work just good for us what you described.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[pfmoore]

My inclination was, if there is no backend explicitly specified in pyproject.toml, then we do a check for our "assumed requirements" (wheel plus any setuptools for legacy, wheel plus a sufficiently new setuptools for setuptools-backend) before we start calling hooks. We do that for both isolated environments that we built, and the main environment in the no-isolation case, and we error out if the assumed requirements aren't present. (So yes, a too-old version of setuptools is an error if we're using the setuptools backend).

That would replace the current "we haven't implemented PEP 517 yet" check for setuptools and wheel in the isolation case, as well as tightening up the non-isolation case.

[pfmoore]

One note: when we make --use-pep517 the default everywhere, everyone that uses --no-index will need to ensure that they make setuptools and wheel packages available to pip if they want to do any builds from source. This is basically #5402, but it's potentially going to hit a lot more people (it hits our test suite in a fair number of places, which is how I spotted it :smile:).

It's a direct consequence of how isolated environments work, so I'm not suggesting we don't do this, but when we do, we'll need to think carefully about how we communicate the change, and ensure that we warn people in advance, before we make that switch.

It's not a problem for the initial implementation, because at that point we default to --no-use-pep517 for setup.py based builds.

[pfmoore]

I should say, if anyone wants to follow along, my work in progress is at https://github.com/pfmoore/pip/tree/pep517_implementation. I've not made a PR out of it yet because I'm doing a bit of history rewriting to keep things clean. But I'll start splitting out PRs from it soon.

Current status: I've refactored req_install.py to add the "use PEP 517" logic discussed above (nothing actually uses PEP 517 yet, that flag currently just means basically "use isolation"). Next steps are:

• Expose the --[no-]use-pep517 option. The internal code is there, it's just the option itself to do.
• Vendor the pep517 library (a bit fiddly as it doesn't have a released version yet).
• Actually build wheels using the backend.
• Handle dynamic dependencies declared by the backend.
• Tests (hardest part here will be writing a set of dummy backends to exercise all the edge cases).
• Documentation.

Plus plenty of bug fixing and the like.

[dstufft]

The PEP 517 flag should be independent of the build isolation flag right? Some day we're going to remove the --pep517 flag, but we're going to keep --no-build-isolation. If we're not isolating the build, then you're just expected to have already installed the build dependencies and we won't call it inside the isolated environment?

[pfmoore]

Sorry, I wasn't clear. Yes, build isolation and PEP 517 are separate. What I've done is refactor the internals of InstallRequirement to read pyproject.toml and decide what to do. There's a new attribute, use_pep517, which encapsulates what we decided (and the --use-pep517 flag will affect this). The build isolation processing remains unchanged, it's just that in the absence of --no-build-isolation, rather than deciding whether to isolate based on "does pyproject.toml exist?", we're doing it based on this new flag (which returns the same answer at the moment, but which will ultimately be affected by the new --use-pep517 flag which will allow users to opt in to isolation in cases where they can't at the moment).

Also, all the tests currently pass, so I've not broken any existing functionality :wink:

tl; dr; Yes, I'm keeping the two independent :smile:

[dstufft]

Ok great!

[pfmoore]

Quick update. I've briefly stalled on the implementation, because of various family and other commitments. I should be getting back to it in a week or two (maybe sooner depending on how things go).

[pfmoore]

Can this now be closed? #5743 is now merged.

[gaborbernat]

Sure 😁when's next pip release?😁

[pfmoore]

January We don't have anyone who's volunteered to be RM for it yet, though :-)

[pradyunsg]

Whee!!! If no one beats me to it, I'll go ahead and open an issue for adding a pip build command tomorrow. :)

[pfmoore]

@pradyunsg What's a pip build?

[gaborbernat]

@pfmoore note phase two of this issue first post. Basically now pip should be able to build both wheel/sdist via PEP-517. So a generalization of pip wheel?

[pfmoore]

Oh yeah, thanks, I'd forgotten about that :-) (Or more accurately, I'd forgotten it was added into this issue as part of PEP 517 - I'd always considered it as a separate thing).

By the way, it's worth noting with regard to this comment from @pradyunsg, in particular

Should pip try to do local-dir -> sdist -> wheel unless build_sdist screeches in pain?
Yes.

that the current implementation of PEP 517 does not do this. I consider the change to always build via sdist to be an independent piece of work - we could have done it ages ago long before PEP 517/518, and it always got caught up in debates about incremental builds, and tools like flit that (at the time) didn't support sdists, etc. I still think this is worth doing, and I don't think the arguments against it are particularly compelling, but I didn't want to block implementing PEP 517 in order to debate that.

So if someone wants to finally switch to build-via-sdist, then I'm all for it, but it would also need a separate issue.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.