Dpl: [provider: pypi] support pyproject.toml

[webknjaz]

Hi, this new PEP is gaining more popularity and some tools already rely on it.

The current PyPI provider implementation is relying on flow where it builds dists using setup.py, while there's lots of flows like building manylinux1 wheels or using flit, which encapsulate this. In case of flit there's even no explicit setup.py in file system.

So I suggest improving the flow and adding more options/conditionals allowing to use custom tools.
For example, to support PEP 518 and PEP 517, it's enough to detect and read pyproject.toml and then install build time prerequisites from [build-system] section (option requires) and then run entry point from build-backend option.

Ref: https://www.python.org/dev/peps/pep-0518/

I'm considering to craft a PR at some point.

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[webknjaz]

Unstale

[pradyunsg]

It'd really be more preferable that Travis just switches to using pip for this.

That way we have one consistent implementation of how things are built locally and on Travis as well. It's easier to ensure that there aren't minor differences between builds on the two platforms when they use the same underlying tool to do it.

[webknjaz]

@pradyunsg think of dpl as a wrapper, of course it uses pip, but pip itself doesn't implement everything + pip doesn't do publishing.

[gaborbernat]

Yeah, this would basically use pip for build; and twine for publishing. Whatever pip did not implement yet let's contribute to the @pfmoore initiative to finish it instead of re-implementing it here.

[pfmoore]

I've no idea what functionality you need here, but with PEP 517/518 support, pip install FOO should transparently install FOO, using the build system specified in pyproject.toml without any additional action on your part.

If you want to build artifacts for upload, pip wheel FOO will do that. That will build a wheel - there's no support in pip at the moment for building sdists - that would be a separate piece of work (there has been discussion of a pip subcommand to do that, but as far as I know no-one's gone beyond talking about it yet).

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[pradyunsg]

@stale no.

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[pfmoore]

I believe this is still unsolved.

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[webknjaz]

unstale

[svenfuchs]

@webknjaz @pfmoore i would like to take a shot in this in dpl v2. however, i could use some guidance on how to integrate this. could you elaborate how the current flow would have to change in order to allow using pyproject.toml?

[webknjaz]

@svenfuchs it's only building dists part that has changed. You need to pip install pep517 and use it to generate a wheel and an sdist. Here's how it can be invoked: https://github.com/sanitizers/octomachinery/blob/df48c0c83ef9b9d17be882dd0f4516bec1b477f4/tox.ini#L81-L85
The result builds the dists (prepares a build env and invokes the entry points as configured in pyproject.toml)

There should be a way to redefine the args and even disable building dists. Since the current implementation calls setup.py unconditionally, I have to hack it like this: https://github.com/sanitizers/octomachinery/blob/df48c0c/.travis.yml#L47-L48. I build dists separately and then empty setup.py so that when dpl calls it, it doesn't wipe dists I've already built manually.
This is a very important use-case: some wheels have to be OS-specific and require a complicated setup so the provider shouldn't attempt to build stuff if a user opts in to build things on their own.

Finally, the publishing part stays as is. twine upload keeps being the only recommended way to publish dists.
Just one note: Warehouse (modern PyPI implementation) now supports token-based auth. You should still use Twine when using tokens. In this case, the username should be __token__ and the token starts with pypi- prefix (https://github.com/pypa/warehouse/issues/5661#issuecomment-518381138).
So maybe the provider settings could have a "token" key instead of requiring users to enter __token__ as a username.
Details: https://github.com/travis-ci/docs-travis-ci-com/issues/2452.
I think, as a part of this effort, the docs should be updated to recommend using scoped tokens.

[svenfuchs]

wait, why have i closed this? gonna reopen this.

thanks for the writeup, @webknjaz!

[pradyunsg]

pip install pep517 and use it to generate a wheel and an sdist

I'll note that there's also a broader discussion happening in the community, around how Python packaging should handle the "build this package" UX. It's a longer-term conversation that'll likely take half an year or so to affect things.

https://discuss.python.org/t/building-distributions-and-drawing-the-platypus/2062

[webknjaz]

Well, I think atm it's the best tool to use anyway. Even though it's intended to be a dependency of CLI front-ends, it already contains a good CLI which just works and seems to be a quite good fit for Travis' case. Anyway, it can be easily replaced should the new recommended UX implementation emerge.

[webknjaz]

@svenfuchs another thing to think about is that often devs test libs, not apps.
In this case, it would make sense to have this build process called even before test steps. This way, Travis CI users would be able to test exactly what's being distributed. The current implementation encourages users to only build a dist after testing meaning that there may be differences in the source vs. the corresponding distribution package. But this is out of the scope of this issue because it's a question of Travis CI design, not DPL.
Still, I'd like to point out that AppVeyor has before_build/build_script/after_build steps which nicely facilitate the proper flow: https://www.appveyor.com/docs/build-configuration/#build-pipeline.
So you may want to consider having something similar in .travis.yml

This is to extend the example of a case where we'd need a deployment provider to completely skip the build step which would enable users to build and install the dist in before_script step, for example.

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[webknjaz]

unstale

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[webknjaz]

unstale

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[pfmoore]

unstale

[pradyunsg]

sigh

[pfmoore]

What exactly is this waiting on? There's a lot of people repeatedly marking it as unstale, but no action. Who if anyone can take this forward?

If it's just going to be ignored indefinitely, it would probably save everyone's time if someone came forward and officially said "we have no intention of doing this" (and dealt with the subsequent responses 🙁)

[pradyunsg]

Given that it's been 2 years, #travisAlums and similar things; I think if there's no response from someone at @travis-ci until the next time stale closes this, we let this go stale and close. I think it'll be clear that they don't prioritize this high enough, so we should just stop putting in the effort to keep this open.

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues