Tox: Add support for PEP-517

[ntninja]

Currently tox does not yet appear have support for the buildsystem.requires key of pyproject.toml as defined in PEP-518.

PIP has implemented this: https://github.com/pypa/pip/pull/4144 (will be part of PIP 10.0) and it would be nice if tox could also add support for installing packages defined using that key, instead of having to use the broken setup_requires key of ./setup.py.

Example of a pyproject.toml file that creates a setuptools 34+ environment:

[build-system]
requires = ["setuptools >= 34", "wheel"]

Example of a pyproject.toml file that creates an environment for building flit packages:

[build-system]
requires = ["flit"]

(Note that this kind of setup is pretty useless without support for PEP-517.)

Approximate step by step algorithm for supporting both PEP-517 and "legacy" ./setup.py deployments (based on PEP-518 & how PIP choose to implement it):

1. If skipsdist in the tox section of tox.ini is true, do nothing
2. Create a new virtualenv to host the sdist generation environment
3. If a file named pyproject.toml exists in the toxinidir:
   
   
   
   1. Parse the pyproject.toml as TOML file (using the python toml library for instance)
   
   
   2. Search for the build-system.requires key and treat each entry in the array as an PEP-508 dependency specification
   
   
   3. If the key was missing or not an array, report an informative error message and exit
   
   
   4. Install each found build dependency into the virtualenv
      
      
      
      
      
      1. Otherwise install setuptools and wheel into the virtualenv
      
      
      
      2. Execute ./setup.py sdist within the virtualenv to generate the sdist used for the subsequent testing steps
      
      
      
   
   

(The build environment may be reused between subsequent testing sessions, as long as the list of requirements doesn't change.)

[obestwalter]

Hi @alexander255, thanks for giving a detailed overview how this could work in tox. I have not yet looked into that topic at all yet and don't even know how many projects are using this. But if pip implements it, it might be a good idea to add it to tox. I don't see this happening for a long time though unless a champion steps up and takes over the implementation.

[ntninja]

don't even know how many projects are using this

Currently, I estimate the number to be around 0 because it's only available in the GIT version of PIP. 🙂

But if pip implements it, it might be a good idea to add it to tox.

Agreed.

I don't see this happening for a long time though unless a champion steps up and takes over the implementation.

I knew you'd say that. 😁 Any pointers where it would make sense to start refactoring to get this into tox?

[obestwalter]

I knew you'd say that. 😁 Any pointers where it would make sense to start refactoring to get this into tox?

I admit that I am a bit on a mission to prioritize on improving process and squashing bugs rather than adding (big) new features in core :)

About your question: I start leaning towards implementing new/experimental functionality like this as plugins. Check the hookspecs - if not everything needed is implemented as a hook I'd rather suggest adding the necessary hooks to core - you can also check the drafts done at the sprint last year about generalizing package builds, as this could have significant overlap: https://github.com/tox-dev/tox/tree/master/doc/drafts

About that toml library: tox is very conservative in pulling in new dependencies and they have to be compatible with all interpreters tox supports. So this would be another reason for implementing this as a plugin. If the toml library only supports Python3 or Python2.7 and above it would be a restriction of the plugin as atm tox still supports 2.6 (but we will drop that shortly after pip does it, which is also happening with the release of 10.0 (https://github.com/pypa/pip/issues/3955)).

[obestwalter]

@alexander255 I just looked into towncrier for changelog management and your estimate is at least off by one :)

https://github.com/hawkowl/towncrier/blob/master/pyproject.toml

[ntninja]

@obestwalter: Only insofar as that they respect the tool namespace collision avoidance rules of PEP-518:

All other top-level keys and tables are reserved for future use by other PEPs except for the [tool] table. Within that table, tools can have users specify configuration data as long as they use a sub-table within [tool], e.g. the flit tool would store its configuration in [tool.flit].

So one could probably make up an interesting legal case about whether that already entailes "using" PEP-518 or not :wink:.

[ntninja]

BTW, I have just tested the Python toml library with Python 2.6 and it (to my great surprise) managed to parse towncrier's pyproject.toml file just fine.

[RonnyPfannschmidt]

@alexander255 its the toml parser towncrier depends upon https://github.com/hawkowl/towncrier/blob/master/setup.py#L30

[ntninja]

I have now proposed #577 for review and while it doesn't implement PEP-518, it does give us the required building block for doing so: The ability of building sdists in a virtualenv, while being able to specify its dependencies.
The rest is basically just wiring up the information from pyproject.toml correctly.

[obestwalter]

Did you check first how this could be implemented as a plugin?

[takluyver]

TOML libraries: I had a brief look at the code of each a few months back, and at the time I felt more confident in pytoml than in toml. Consequently, pytoml is the one which is vendored into pip, and which will become a dependency of flit once PEP 517 is finalised.

The developer of pytoml did not express great enthusiasm for ongoing maintenance, IIRC. But if it's used in pip, there's an incentive for other people to take over the maintenance if it becomes necessary.

[obestwalter]

Hi @takluyver, thanks for pointing that out. It will be best to have a close eye on the developments in pip and follow their lead then whatever they will use for parsing toml.

[ntninja]

I honestly don't care which one (toml or pytoml) should be used, but toml seemed more active, so that's what I suggested (hence the for instance). If pytoml is preferred, then so be it 🙂.

Regarding a PEP-518 plugin: Currently tox definitely doesn't expose or support the required funcationality, but if the options of #577 land; it should be possible to use the tox_configure plugin hook to parse the pyproject.toml using [insert TOML library here] and apply the configuration changes accordingly.
The code for this should be trivial and unless I've missed something the decision of supporting PEP-518 natively or not will mostly depend on whether you're willing to depend a TOML library.

Thinking further ahead (PEP-517), there would also need to be a new configuration option to launch an abitrary command to generate the sdist instead of the currently hard-coded python ./setup.py sdist --formats=zip --dist-dir …. I have no idea how to wire up the subsitutions for this one through.

[obestwalter]

Regarding a PEP-518 plugin: Currently tox definitely doesn't expose or support the required funcationality

o.k. thanks for checking it out. As your PR is quite big it will take a while until somebody will have the time to review it properly, so I hereby thank you already in advance for your patience :)

[obestwalter]

Hi @alexander255, about that principle thing: in principle I think tox should support workflows that are considered "good practice" - which is hard to define already, but if it's an accepted PEP this is pretty much a no brainer here. So yes: we should support it and ideally not too long after pip starts supporting it. Whether your PR is the right direction to take is then to be judged by a thorough review.

If you provide documentation and tests I would say PR https://github.com/tox-dev/tox/pull/577 will definitely be the way to go and only a question of how to whip it into shape. So if you want to keep working on this and bring this PR up to speed, I assure you my assistance as good as I can.

[warsaw]

I have a project where we're trying to get rid of setup.py and just use flit and pyproject.toml. In reading over this issue it's not entirely clear to me whether tox already supports this operating mode (perhaps with a tox.ini setting?) or whether this issue is needed to enable that. I'm not seeing anything in the tox documentation, google-fu turns up nothing relevant, and I don't see any other related issues here. Of course I could be missing it.

So what's the current state of tox support for a setup.py-less pyproject.toml configured project?

[takluyver]

I don't think tox supports pyproject.toml yet. Implementing PEPs 518 & 517 is probably a moderately large job. It's proving tricky for pip as well (pypa/pip#4802).

I'm definitely happy to see more people trying flit, but there's still a way to go before it works smoothly with the rest of the ecosystem.

[gaborbernat]

@warsaw no support at the moment, however I plan to add it at some point 👍

[gaborbernat]

Drawing from the experience implementing this in toxn (and with pip 10 coming out with PEP-518 support too), I'll add support for this in the next month, drawing up and during the PyCon sprint.

[obestwalter]

Hey @gaborbernat nice - have fun at PyCon!

[gaborbernat]

Things I learned at todays packaging open space from PyCon. There actually is little we need to do ourselves. PEP-517 and PEP-518 define responsibilities for pip and setuptools/twine respectively. Now we do python setup.py sdist to build the package and then use pip install path_to_package. Instead what we should do is pip build (note this command does not exist yet). Both setuptools and flit already have support for PEP-517/518, pip is behind a bit on this, and is in desperate need of interested developers to work on it.

It is pips responsibility to build a package (and for this to create a virtual environment, install build dependencies, and then call the builder - setuptools/flit). This requires people interested to solve https://github.com/pypa/pip/pull/4589, https://github.com/pypa/pip/pull/4589 and then build on top of that to add such feature support.

Virtualenv needs renewed love (read rewrite), https://github.com/pypa/virtualenv/pull/697 is a rewrite of it, this also needs interested people to take it on.

[takluyver]

There's also a pep517 module providing an API for tools to go from a source tree to an sdist/wheel using the PEP 517 interface. I wrote that in the hope that it would make it easier for pip to add support for that interface.

[gaborbernat]

@takluyver what does that package actually contain out of:

• create virtual environment
• install build tool,
• install build requirements,
• invoke build operation by parsing the pyproject.toml and then invoking that,
• return the built package path?

Thanks!

[takluyver]

If you use the high-level API, it tries to deal with all of those steps. The lower level API is just the last two steps, if you want more control over the build environment. See the README for how to use the two different levels.

[gaborbernat]

Hmm, I wonder what's the hold up then of just exposing it as pip build, or what the challenges could be with it?

[RonnyPfannschmidt]

imho tox should collaborate with pip there to move the entire process of creating such build artifacts into pip and to simplify tox

[gaborbernat]

That's the goal here ^_^

[gaborbernat]

created https://github.com/pypa/pip/issues/5407

Once that's implemented the description of this issue changes to:

Use pip build . --wheel instead of pip setup.py sdist (or have some opt-in flag for that, or just make it a breaking change with major version change). Having the package as a wheel would significantly speed up environment creation as a side effect.