Tox: automate release

[gaborbernat]

We should automate the release process via a role account. When a maintainer tags a commit with a version, the CI should kick in and if all tests succeed automatically release the package.

[nicoddemus]

Definitely. I've been using this in a number of projects and this is really handy.

pytest can be used as an example:

https://github.com/pytest-dev/pytest/blob/b50911285a4d2f593afb92f568b21686473e25a5/.travis.yml#L45-L59

[gaborbernat]

@nicoddemus how do you handle the towncrier release?

[nicoddemus]

That's not automated yet: https://github.com/pytest-dev/pytest/blob/master/HOWTORELEASE.rst

Currently we run a pyinvoke task to generate release notes and CHANGELOG. After that's done, we open a PR and once that has been approved and passes all tests, we just push a tag. This triggers the actual publication to PyPI.

@RonnyPfannschmidt has suggested how this could be automated further: whenever a new merge is done to master (and features in the case of pytest), a bot could open a new PR with a "candidate release". If we decide to actually release it, it is just a matter of pushing a tag and merging it, otherwise we can close the automated PR.

Of course this bot could be reused by a bunch of projects in pytest-dev and tox-dev, if we can agree on a similar enough release process.

[gaborbernat]

I don't think that would be wise, we would have way to many false positives. Instead I could see maybe trigger on tag, generate the release commit and update the tag to that.

[nicoddemus]

I agree it would generate too many false positives. I'm particularly not a big fan of moving tags though, AFAIK this is not recommended in Git.

Some brainstorming: what we need seems to be a chatbot of sorts that we can say "hey start release X.Y.Z". Two possibilities come to mind:

• A Gitter or IRC bot which accepts commands from maintainers.
• A GitHub bot which watches issues/PRs, and accepts commands from maintainers.

The process would be then to:

1) command the bot to create the PR.
2) push the tag to publish the release.

[gaborbernat]

I like the idea of Github issue bot with the slight change of I would have the bot to due all: open the PR, once and if the CI passes merge it, push the tag, send celebration messsages.

[nicoddemus]

Good idea. 👍

One minor improvement: also wait for at least one maintainer approval before pushing the tag.

How to you imagine the "GitHub issue bot" could work exactly? Listen to all issues for a command like "@tox-bot start release 3.3.1"?

[gaborbernat]

Pretty much like that. Maybe more like @janus-bot as a reference to the roman god gatekeeper https://en.wikipedia.org/wiki/Janus.

[nicoddemus]

https://github.com/python-trio/trio/issues/220#issue-236077049 is about the same thing. This is one idea on how to start the release process which I like:

..I guess thinking a bit more, it might make more sense to do something like, you file an issue called "v0.2.0 release tracking" and say @bot: release 193428efabde39 as v0.2.0, and it checks out that branch, runs towncrier to generate the news (maybe setting up a branch in the process?), builds the release, merges the generated news back to mainline, tags it, uploads to pypi. While giving updates in the issue, and tells you if something went wrong.

I tentatively like the idea of the bot using a issue to report progress/issues.

[gaborbernat]

All it remains is to create a bot like that and host it somewhere :+1:

[obestwalter]

Sounds like fun. What would be needed to host such a bot?

[RonnyPfannschmidt]

some kind of vserver/host running scripts - wrt releases i propose something travis based combined with github so travis manages the commits/next release pull requests and the bot manages putting tags on them and triggering a merge after a release

that way the bot needs a really small cpu footprint, it might even fit a free tier at one of the hosters

[gaborbernat]

Cpython uses heroku for its own GitHub bot.

[nicoddemus]

http://plugincompat.herokuapp.com also is hosted on Heroku.

[njsmith]

My impression is that heroku is popular for these things because (a) it's generally a very convenient platform, (b) the way they limit their free tier is that they shut down your site if it hasn't had any visitors in a while, which makes it hard to do time-based stuff and the first visitor after the pause has to wait a bit for the site to be automatically rebooted. But if you have a bot that only needs to receive github's webhooks and do something with them, the free tier is totally sufficient.

[njsmith]

(but note that I don't actually run any github bots yet myself so I might be totally off base :-).)

[obestwalter]

Does anyone have experience with using https://www.pythonanywhere.com/ for this kind of stuff? I quite like them and they also have free tier.

[gaborbernat]

Not yet, but will put on my radar creating this (we can use the CPython one https://github.com/python/bedevere as example). Once we have the code we can decide where to host.

[nicoddemus]

Awesome! We would love to use this in pytest and a lot of plugins too.

[gaborbernat]

We'll tackle this but with a lower priority now that other parts are mostly automated. I'll mark it as part of release 3.3 as 3.2 has some important user requests I would prioritize higher.

[nicoddemus]

Just to see this in action at @conda-forge: https://github.com/conda-forge/pyinstaller-feedstock/issues/19 triggered https://github.com/conda-forge/pyinstaller-feedstock/pull/20.

Seems to work well in practice. 👍

[webknjaz]

I'm not sure about needing to have a bot, but I have successfully running flows based on pushing git tag + setuptools-scm + Travis. It doesn't create a new release commit, runs tests on tag as well and has an extra stage which builds dists and publishes them to PYPI.
Feel free to ask me about the experience + catch me at the EuroPython/sprints in person for help :)

[gaborbernat]

that only works if you don't need release notes generated for release via e.g. towncrier. the pypi release itself we have it too but a release involves much more than that (e.g. to tweet too the release).

[webknjaz]

Makes sense. I just add changelog in a commit before tagging.

[gaborbernat]

would be nice to not have to do that manually, hence the concept with the
bot

On Fri, Jul 27, 2018, 11:48 Sviatoslav Sydorenko [e-mail]
wrote:

Makes sense. I just add changelog in a commit before tagging.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/tox-dev/tox/issues/869#issuecomment-408383572, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAqIPt858-UOeKN_5XidUknUl5O-7SUSks5uKvAQgaJpZM4VA2g8
.

[webknjaz]

Yeah, I'm actually thinking of building bot using Deployments API, because I have even broader use-case, where binary wheels reach PYPI during 8-hours interval and I also need a synchronization primitive

[RonnyPfannschmidt]

@webknjaz im running such a setup for many projects as well, but its impossible to support towncrier/regendoc integration sanely with that as of now

[nicoddemus]

As of now, it seems that for both tox and pytest it is enough for the bot to:

1. Checkout the source
2. Create the branch from latest master (pytest has a features branch, not sure how to handle that yet).
3. Create a Python 3 venv and install tox
4. Run some tox command (tox -e release -- VERSION)
5. Push the branch to a fork
6. Open a PR.

Everything that needs to be customized can be done in the release tox env. This seems to be the way things are converging.

[RonnyPfannschmidt]

i believe we can do most of that job in a travis job (having a separate repo and a token that is allowed to push to it via pytestbot

whats missing is something that turns the pr into a tag into a merge

[nicoddemus]

i believe we can do most of that job in a travis job (having a separate repo and a token that is allowed to push to it via pytestbot
whats missing is something that turns the pr into a tag into a merge

In that scenario, how would a dev trigger a new build?

[RonnyPfannschmidt]

any new build would need a change in the base branch to begin with, as such any change should force-push a new rebuild version

[nicoddemus]

You mean the process would be to push a new commit to the separate repository, possibly directly from the GH interface?

That's the approach I used when testing pytest releases with devpi using https://github.com/nicoddemus/devpi-cloud-tester, indeed it works well. It doesn't even need credentials at all, it would use its own fork to open PRs and we could still do the actual publishing with a tag. 😁

[RonnyPfannschmidt]

@nicoddemus it needs the github credentials of a user - as such a token with fit limits is needed

[nicoddemus]

You are right, I was thinking of needing writing access to the main repository. We still need credential access so the bot can create forks and push PRs. 👍

[webknjaz]

Actually, I think you're going to need to use GitHub Apps API, which doesn't require the user credentials and authenticates as an actual bot, not a user. It's also a better portable approach.
This way you can use Deployments API, where something (define what) will trigger an API call for "hey, github! deploy this commit please" and then the "listener" part will get a deployment event through webhook and might trigger Travis with a custom config to do the actual deployment.

[RonnyPfannschmidt]

@webknjaz that is still authentication with credentials

additionally i yet have to find any documentation on how to handle a bot that owns git repos

[webknjaz]

Basically, you create an application @ https://github.com/settings/apps/new
Then, you generate and download a private key from https://github.com/settings/apps/{{ your_app }}.
Finally, you use that to authenticate against _Installation_.

So, installation is an entity, which represents binding between your app and user account or organization with a set of repositories where user installed it (listed @ https://github.com/settings/installations if installed into user account). It can be installed into user account and organizations, and those would be separate independent installations. You'll need those IDs to authenticate and get tokens for accessing them as bot. It's what new Travis CI integration (.com) uses.

I have a sketch of a CLI client for new Checks API @ https://github.com/webknjaz/check-in (click + attrs + pygithub).

P.S. Are you at EuroPython Sprints now? If yes, I can show/explain/demo it in person.
P.P.S. You can also create an app as an organization account, not a personal profile.

GitHub

Build software better, together

GitHub is where people build software. More than 28 million people use GitHub to discover, fork, and contribute to over 85 million projects.

GitHub

Build software better, together

GitHub is where people build software. More than 28 million people use GitHub to discover, fork, and contribute to over 85 million projects.

GitHub

webknjaz/check-in

check-in - Checks your test results metadata into github, commit-bound. Acts as a bot. You'll need a GitHub App to use it.

[RonnyPfannschmidt]

@webknjaz at first lance tat doesn't solve the issue i'm imagining, but it does help with some things

i`m not at EP, but i'D be happy to join a voice chat or sth like that

[webknjaz]

We could try if the wifi on the venue will work well. Otherwise, I will have to get back home first (Monday afternoon).

[gaborbernat]

I think this would require more along the lines of the following:

1. monitor issues, when a maintainer creates a release one trigger
2. create a pr with towncrier
3. wait until all checks passed (this is important we need both Linux and windows tests to pass)
4. when all checks passed perform pypi release
5. tag with version and push
6. merge pr via merge commit
7. do release notification (run tox env)
8. close release issue

[webknjaz]

@gaborbernat fair enough

@RonnyPfannschmidt do you have any preferred hours of interaction? I'm up to sharing some of my knowledge about GitHub APIs

[RonnyPfannschmidt]

@webknjaz today im relatively busy with work and childcare, in general I'm in CET and currently likely available on short notice

[gaborbernat]

@webknajz so you think we can do the above with just a GitHub App?

[webknjaz]

@gaborbernat Yes, it should be possible. We could discuss ideas, but basically, such integration would have access to GitHub's API and Travis CI API. And you can actually substitute .travis.yml in your API calls, making possible to instruct it doing different things from your workflow. Also, deployment API has a param where you can specify that you want certain checks to succeed before proceeding.

@RonnyPfannschmidt I'm in the same timezone and mostly available during daytime, so feel free to ping me on the day when you are ready.

[gaborbernat]

I'm not sure I follow why would need to involve any Travis in this.

[webknjaz]

That's in case you want to keep a bot codebase small and have only orchestration part in there.

[gaborbernat]

Deploying to pypi takes five lines of code tops, not sure the added complexity of Travis is worth the effort.

[webknjaz]

Yes, you're right. In general, this all depends on one's preferences. I would prefer using Travis CI, I think because it would give the log output, ability to restart/debug stuff. I would also think about reusability of such bot because this API essentially allows anyone to install your app in their repos (if you make it public), which might have use-cases extended to running something additional as part of this flow. You don't want to execute arbitrary things in your integration env, so providing a way of doing it in their repos would be a way to go.
But if it doesn't fit into your expectations it's also ok.

[obestwalter]

I might be missing something here, but I thought we basically settled to moving to vsts ... right?

[gaborbernat]

That only solves the problem of ensuring all platforms pass before doing a release (e.g. when all checks passed perform pypi release). It does not solve the full automation (create release pr, merge it, notify). And besides other tools/plugins might not have this luxury would be nice to create for them too a solution.

[gaborbernat]

Created https://github.com/gaborbernat/janus-release-bot, will be my playing ground for all this. Configurable sections are:

• release commit creation (e.g. run towncrier),
• release operation (this also includes release notifications).

For the second it's compelling to use a service (such as Travis as would mean we don't have to solve the problem of storing secrets). For the first, it's trickier as the bot should be the one making the commit. So probably we'll fallback to just invoking tox to take care of acquiring dependencies and running the operations to not hard code it. This does mean this will be bound to Python at build level.

GitHub

gaborbernat/janus-release-bot

Contribute to janus-release-bot development by creating an account on GitHub.

[gaborbernat]

This has been done up to a reasonable degree (aka release on tag) which is good enough for now.

[njsmith]

You might also find this discussion relevant: https://forum.bors.tech/t/running-custom-code-before-and-after-the-merge-for-continuous-deployment-or-other-uses/315

Bors Forum

Running custom code before and after the merge for continuous deployment (or other uses)

Hi all, I’ve been looking for a way to automate a continuous-deployment workflow: PR is approved. Automation bumps version number, does appropriate fiddling with release notes. And while we’re at it, let’s do other boring house-keeping like running a code auto-formatter. We test the new version (b/c of course you have to test after making these changes) If the test passes, merge to master If the test passes, upload the release artifacts to your favorite repository This isn’t exactly what bor...