Setuptools: Plea for release candidates / announcement to distutils-sig

[matthew-brett]

Sorry to be demanding, but this is a humble request that you add a release candidate phase to the release process.

Like others, most of my packaging builds started failing this morning with #1042 .

Of course, you're in the unenviable position of being at the bottom of many many stacks, and it's really hard to avoid these kind of problems. But - it would really help if you would consider making a release candidate and putting it up on pypi. The ideal would be if you could announce somewhere like distutils-sig, so we packagers know to check the release candidate somewhere with pip install --pre setuptools. For example, I am happy to do that by default on the many packaging repos that I have, so that I get the failures immediately, and before they go live by default across the ecosystem - I am sure others are happy to do that too.

That would really help make sure that people don't get unexpected breakage. Pretty please.

[dstufft]

The main downside with this I think is that setuptools doesn't have a --pre like pip does, so anyone using setuptools (easy_install, ez_setup.py, setup_requires, &c) will fetch the pre-release.

Beyond that, it seems reasonable to me.

[matthew-brett]

Thanks for considering.

I guess that will mean that the easy_install etc installs get what we get now, and other install methods will get a better tested version.

[graingert]

@dstufft This approach wouldn't have helped in the #1042 scenario: the 36 missing 'six' errors would only be present when bootstrapping setuptools ex nihilo. consider:

1. setuptools 35 is released

2. setuptools 36.pre is released

   python get-pip.py # run by travis/virtualenv to install setuptools 35 and six
   pip install --pre setuptools # installs setuptools 36.pre
   pip install tox -r requirements.txt # installs totally fine because six was already pulled in from 35.
   

3. all the RC people are totally happy, so setuptools 36.pre gets released as setuptools 36

   python get-pip.py # run by travis/virtualenv to install setuptools 36 and NO SIX
   pip install --pre setuptools # noop
   pip install tox -r requirements.txt # build fails with missing six.
   

[jaraco]

Thanks for the suggestion. At first blush, it sounds like an excellent idea.

In the past, I have made pre-releases available (example), especially for backward-incompatible releases that were expected to cause disruption (and commensurate with the amount of disruption expected). In practice, I find that even if I cut the release and tweet about it and lead up to it with discussion and invitations to test, if issues are going to emerge, they will emerge after the release, and usually only in a small percentage of total uses.

So I've chosen instead to create minimally-invasive, rapid releases such that if issues are reported, they can be readily identified and corrected or rolled back, depending on the severity of the failure and the difficulty of corrective action. Ideally, the unit-test suite will capture all use cases, such that a release is only cut when tests pass.

This approach failed us last night when the release went out late and I managed to fall asleep before noticing any issue.

I loathe to create more manual steps that are unlikely to add significant value but guaranteed to add delays to releases.

I'd prefer to focus on how a broken build was able to bypass the existing safeguards and bolster those safeguards (test suite).

If in light of this, you still think that release-candidates are worth the trouble, we probably need more volunteers on the projects to do that additional work.

[matthew-brett]

Jason - I doubt that the majority of packagers are following you on Twitter. I don't have a Twitter account myself. So they, like me, may well have missed your announcements. If nothing else, would you please consider announcing on the distutils-sig mailing list rather than Twitter? I think you'd get a lot more feedback.

It seems to me that doing release candidates would be a very small amount of extra work. If they don't cause any breakage, it's just the work of doing two uploads separated by a week or so. If they do cause breakage, then it will very like save work, not having to do damage control as here (even if, in this case, a pre-release would not have caught it). If you need some help as the release manager, in order to do this, I am happy to volunteer.

setuptools is really fundamental to our stack, and when it breaks, it causes a lot of pain. For example, I remember this one, earlier this year: https://github.com/pypa/setuptools/issues/945.

I am sure that, if you can think of some way of harnessing community effort for testing, you'll get a lot of people willing to help.

[jaraco]

Hi @matthew-brett. I do make announcements on distutils-sig when I believe manual intervention is necessary, such as here. In the past, people have asked for Twitter also, so I try to do that.

I think you'd get a lot more feedback.

Not really.

It seems to me that doing release candidates would be a very small amount of extra work. If they don't cause any breakage, it's just the work of doing two uploads separated by a week or so.

By introducing a delay event, you necessarily introduce the need for a schedule. Currently, issues are addressed, commits pushed, and releases cut in one session. There's no need to trigger an event for a subsequent action. Given that most releases don't lead to issues, this extra delay only reduces the proximity of the change with its release, requiring more effort to revisit emergent issues (to revive the context around the change in people's minds). Additionally, it's not always the case that one has contribution time in fixed intervals, so a delayed release may get further delayed due to availability of the release managers.

If you need some help as the release manager, in order to do this, I am happy to volunteer.

I'm always looking for more help maintaining the project, including shepherding releases. Why don't we start with a PR to the release docs describing precisely how you would like to manage the releases. If we can come to an agreement on a viable process, I'll gladly grant the necessary access. One feature I'd like to retain is that a maintainer should be able to trigger a release and then leave the system to do the rest.

I was also under the impression that PyPI doesn't accept pre-releases, so supporting a pre-release process might entail creation of another service for hosting those releases... which leads to the question - why would these environments that would be motivated to test pre-releases not simply test against pushes to master?

[dstufft]

Pypi supports pre releases. It doesn't support local versions.

Sent from my iPhone

On Jun 1, 2017, at 12:44 PM, Jason R. Coombs [e-mail] wrote:

I was also under the impression that PyPI doesn't accept pre-releases, so supporting a pre-release process might entail creation of another service for hosting those releases... which leads to the question - why would these environments that would be motivated to test pre-releases not simply test against pushes to master?

[matthew-brett]

Jason - ouch to the silence from your January email to distutils-sig. It may well be true that, in order to get a lot of testing, it will need to be possible to test on automated builds. That would be true for pypi pre-releases, in that it only needs an extra --pre flag in the test script.

I think most packagers would be happy to test pre-releases, because they would expect, in most cases, that they would not hit errors which will delay their builds and require them to investigate. If they are testing on master, they are more likely to hit a bug, which will use more of their time.

Can you think of any other ways to help recruit the good will out there to increase testing before release?

To deal with your worries about delays, I'm happy to commit to releasing the final version, if you release the pre-release version. So, the procedure would be exactly the same as now, except I'll monitor the project for a release candidate, send an email to distutils-sig (if you don't have time to do that), and release the full version after some short period (say one week) after no issues have been reported on github or on the mailing list - what about that?

I just started a WIP pull request, but I need some help from bump2version experts : https://github.com/pypa/setuptools/pull/1051

[dstufft]

FTR we used to do RC candidates for pip as well, and I don't think they ever resulted in someone finding a bug or a regression prior to the "real" release. That was prior to the --pre handling, so we weren't uploading them to PyPI so I don't know if that would end up having more people willing to participate or not.

[matthew-brett]

@dstufft - my guess is that more people would be willing to participate, because of the growth of automated testing - so testing the RC is just an extra line to your continuous integration configuration script. For example, I hit this immediately because I'm building numpy, scipy, matplotlib, h5py daily via travis-ci cron jobs.

Any way of improving continuous integration for setuptools is fine - if there's a better option than release candidates, I'm happy to help with that too.

[dstufft]

To be clear I'm not opposed to this, It very well could be worth a try! Just stating that when we tried with pip before the results were not super great.

[graingert]

repl.it have somehow managed to install 'every' package on PyPI. It would
be a great test if post publish we repeated that feat on various well known
CI providers.

On 1 Jun 2017 21:16, "Donald Stufft" [e-mail] wrote:

To be clear I'm not opposed to this, It very well could be worth a try!
Just stating that when we tried with pip before the results were not super
great.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pypa/setuptools/issues/1045#issuecomment-305607846,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAZQTI97XKFx7qH6tJELb4s1DM4aGtJRks5r_xwGgaJpZM4NswF1
.

[matthew-brett]

Would it be practical to set up a devpi index somewhere, to which pre-releases of pip, setuptools, wheel would be uploaded? Then I believe I am right in saying that willing packagers could add the relevant lines to their pip.conf and .distutils.cfg so that they are pulling in the pre-release versions of all these tools.

[dstufft]

If @jaraco decides he is OK with doing RCs we can get a DevPI instance set up.

[jaraco]

The authors of devpi do provide an instance, which may be available for this purpose, although depending on the usage, they may wish for PyPA to host its own. Of course, since it seems PyPI hosts pre-releases, that seems like a suitable enough host with less complication.

I do plan to review the proposed workflow, perhaps as soon as tomorrow.

[matthew-brett]

The reason devpi installation might be interesting is that it allows us to shield ordinary users from pre-releases installed via the easy_install etc routes - see https://github.com/pypa/setuptools/issues/1045#issuecomment-305466140

[graingert]

There's always the official PyPI test server

On 2 Jun 2017 5:05 pm, "Matthew Brett" [e-mail] wrote:

The reason devpi installation might be interesting is that it allows us to
shield ordinary users from pre-releases installed via the easy_install etc
routes - see #1045 (comment)
https://github.com/pypa/setuptools/issues/1045#issuecomment-305466140

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pypa/setuptools/issues/1045#issuecomment-305832830,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAZQTCCgScrmeqBbTk7mHOPtngq7HKuiks5sADKmgaJpZM4NswF1
.

[FRidh]

At times I thought setuptools could use pre-releases as well so that e.g. distro's could test the newer versions. However, I am wondering whether distro's would rebuild and test their packages (which a proper test would require) just for a setuptools pre-release.

I think @jaraco 's approach of rapid releases works well. Issues arise, but they typically get fixed quickly. Furthermore, one typically doesn't need the latest version of setuptools anyway. The main issue I think is not the rapid (and potentially breaking) releases, but that it is common for Python devs to always use the latest versions of everything (or pretty arbitrary combinations) instead of using sets of packages that are known to work together.

[matthew-brett]

@FRidh - I think there are two different ways of seeing setuptools etc. One is the perspective of the system administrator, who has full control of all Python package deployments. From that point of view, it makes sense to think about trying to specify setuptools versions, avoid upgrades when possible and so on. From the sysadmin perspective, it isn't a very great concern that a release causes widespread breakage, because the sysadmin can diagnose and fix it quickly enough - it's just a couple of hours of their working day every now and then. However, I'm thinking of the second perspective, which is that of the default or ordinary user . For that user, I'd like to do everything I can to make sure that they just never get a weird crash from using standard tools like setuptools. In my experience (as someone who teaches students and new programmers) - that can cause a lot of confusion, and tends to repel people from the community ("it's too fragile"). If we can do some low-maintenance continuous integration to make a crash less likely, I think that's a very good investment.

[dstufft]

Another possible option for us here is better testing infrastructure. Specifically what I'm thinking of is something that will essentially just run through a list of preselected packages (chosen to try and get some variation into the types of complexity of packages and what they do) and simply try to install them. That won't catch as many things as a well used pre-release, but it is faster and more on demand for the developers of setuptools/pip so it might be a reasonable middle ground.

[FRidh]

None should encounter any issues when doing a 'typical' installation. The problem is that the 'typical' installation installs latest versions of everything that have been put together without indeed any integration testing.

I agree that we should look at improving the testing infrastructure. Would it be possible to get sponsorship from say the PSF to pay for hardware that could be used for CI?

[pganssle]

On the question of release candidates: If the main problem is that people can't always filter them out, are people really worse off in a world with release candidates than a world where we just cut releases? Seems like the people who can't filter it out are no worse off, but the people who can filter it out are slightly better off. I think we release enough that it doesn't make a big difference, but I'd be fine with pre-releases for all major version releases.

Regarding announcements: Do we want announcements before the release, or is an announcement at the time of the release acceptable?

Since tagging the repo creates a release, it seems like we should be able to automatically trigger an e-mail to distutils-SIG as part of the release process. I think adding in a delay between announcement and release would make that a lot trickier, though.

[jaraco]

I think most packagers would be happy to test pre-releases, because they would expect, in most cases, that they would not hit errors which will delay their builds and require them to investigate. If they are testing on master, they are more likely to hit a bug, which will use more of their time.

I'd say for setuptools, there would be little difference between master and release candidates. Pushes to master are expected to be stable and release-ready, meaning that barring exceptional circumstances, master is always releasable and could be released at any moment. If integration tests catch a bug on master, that would be a bug worthy of reporting and blocking pending releases.

That's not to say there's not value in having a more deterministic delay between when a commit is pushed to master, candidate builds are available (possibly synonymous with the prior step), and later the final builds are released.

I did notice though, there is an issue - it's not possible to pip install setuptools from master due to the bootstrap process. I've reported that as #1527.