Docker-stacks: Experimenting with quay.io?

[jakirkham]

I'd like to suggest we start experimenting with quay.io. We just started using this (today) at conda-forge and it has been pretty great thus far. Our experience with Docker Hub autobuilds was quickly going downhill as it did here. As it appears that they are basically dropping Docker Hub for Docker Cloud as mentioned in this comment, there is even more reason to contemplate having a second source. Currently, we (at conda-forge) our mirroring our image from quay back on Docker Hub so it can still be gotten over there.

Admittedly, we are going to need a longer experimentation period at docker-stacks and the system used here at docker-stacks is a bit more robust given the purpose it serves. So, if we were to switch at some point in the future, there are a number of things that need to be taken into account including how we will preserve history. Though it certainly doesn't hurt that quay uses the docker cache, which is something that Docker Hub does not.

Thoughts?

cc @parente @rgbkrk @minrk

[rgbkrk]

@smashwilson has been using quay.io with much success (from what I hear) - I'm in favor.

[parente]

I'm for experimentation.

Regarding history, github will always have it. If all the images disappeared, anyone could still rebuild the effective equivalents.

[minrk]

Folks here are using quay.io for fenics images, and seem to be having a good experience, so 👍

[parente]

@jakirkham did you create a jupyter org on quay.io? I see that it's taken. I might have some cycles to help with this if you add me. (I have a vested interest after the sudden influx of PRs and builds I had to do yesterday.)

[jakirkham]

Nope, but it does have tmpnb in it. So, conceivably someone Jupyter affiliated created it. Did either of you create it, @rgbkrk or @minrk?

[rgbkrk]

Sure enough, I own the jupyter org on quay.io. Added parente and jakirkham to the jupyter org on quay.

[parente]

It looks like quay has support for notifications on various build actions. I don't see a way to trigger one build when another finishes.

[jakirkham]

How are you wanting to trigger? They have webhooks. So, that might be something to explore. We used this at conda-forge to trigger "builds" on Docker Hub (pulling the quay.io image over).

Are you wanting triggering within quay.io?

[parente]

I'd like base-notebook to build and when that build completes automatically trigger minimal-notebook and so on down our inheritance chain. I checked with quay support and they noted this feature is "coming soon".

I agree that we can accomplish it today by writing a webhook receiver that notes one build A is done and calls the API to do B. But is it worth it?

[mghazizadeh]

@jakirkham et al We apologize for the inconvenience your team experienced with autobuilds. We would love to get your feedback and discuss how we can do better going forward. I sent you a note yesterday. If you like to discuss please drop me a note and we'll set up a discussion.

[parente]

@mghazizadeh thanks for checking in. When I asked support about builds triggering other builds on completion, I was added to a list of people who would get notified when the feature rolled out at quay. So we're not inconvenienced, just waiting patiently.

[parente]

@mghazizadeh And @jakirkham just corrected me that you're checking in from docker, not quay. 🙊 My bad!

[mghazizadeh]

@parente np. btw, Docker Cloud users can trigger builds from other builds by using post_build hooks to trigger other builds. Each build, once setup in Docker Cloud, has a trigger URL that can be used to trigger the build from anywhere including a post_build hook related to another build. The Docker Cloud build backend engine is the same one service Docker Hub autobuild. Same with the repository service. Happy to discuss further if you like.

[parente]

@mghazizadeh https://github.com/jupyter/docker-stacks/issues/15 has most of the history why we switched from using automated builds on Docker Hub to manually built and pushed images. Some of the details of how we tag builds here have changed and so not all of the problems we encountered with the automated builds remain relevant. I'll also admit that I haven't looked into Docker Cloud to see if it now fits the bill.

[mghazizadeh]

@parente no problem.. happy to help any way we can. actually the build trigger URLs on cloud are not yet available to public. we're working to make them available soon.. the same way they are on Docker Hub autobuils.

[parente]

Doing a bit of research today, https://docs.docker.com/docker-cloud/builds/advanced/#override-build-test-or-push-commands looks like it allows the level of customization we need to build the hierarchy of images defined in this repo.

I think an issue with this approach (and really any hosted CI build approach ala PR #321) is that all of the images wind up being built from scratch for every change, unless we can control the local caching story. Users who want to docker pull the latest version wind up having to download all of the layers from the newest image and don't benefit from any kind of caching.

[jakirkham]

I think the new Docker Cloud has an option to allows for caching. Don't know whether that helps or not.

[parente]

Yes. I'm not clear on how that works when overriding the build phase commands. Maybe @mghazizadeh can speak to that. Otherwise, some experimentation is in order.

[mghazizadeh]

@parente: @caervs @pchico83 can provide comments/feedback on what you can do with build caching in Docker Cloud.. it seems also from above you'd be interested in multi-stage build and smaller image sizes with only the necessary layers.. (something we just announced at Dockercon that is coming)..

[pchico83]

@parente yes, build cache is available in docker cloud. When it is activated for a build rule, we pre pull the last built image tag for the build rule being built, and we execute:

docker build --cache-from pre_pull_image:tag new_image:tag

Other options available in Docker Cloud (and not in Docker Hub) are:

• Ability to retry a build.
• Streaming logs for running builds.
• Hooks (scripts to be executed before/after the build command). This is a powerful feature that allows to customized whats get executed for every build. Some examples are here: https://docs.docker.com/docker-cloud/builds/advanced/#build-hook-examples
• Ability to run build/tests on pull requests.
• Integration with github commit statuses.
• Support for build context.
• Email/Slack notifications.

Let me know if you need help experimenting with Docker Cloud. I will be happy to help!

[parente]

@pchico83 Thank you for the info. I'll reach out if I have any questions when I get back to experimenting with the build setup.

[parente]

The docker-stacks.cloudet.xyz build VM is no more for ... reasons. I spent time yesterday and today continuing my research into building on Docker Cloud. Good news: I've succeeded in setting up a base-notebook build that triggers a minimal-notebook build, tags both images with latest and the 12-character git SHA we've been using, and pull layers from cache properly unless something has changed in the image or one upstream.

• https://cloud.docker.com/app/parente/repository/docker/parente/base-notebook/general
• https://cloud.docker.com/app/parente/repository/docker/parente/minimal-notebook/general

All it takes is a bit of configuration on Docker Cloud plus a set of post_push hook scripts in the repo here.

Next step is to see if I can re-configure the existing jupyter/* repositories on Docker Hub so that they autobuild without disturbing their legacy contents. I'll then send in the PR with the hook scripts and an updated README describing what has to be setup on Docker Cloud if and when we add new repositories.

[parente]

PR open for review with one remaining gotcha. The build order is dictated by a chain of post_push hooks invoking build triggers for descendent repositories. I could not find documentation on whether the build trigger API accepts the SHA of the git comment to build. So as it stands, every automated build is pulling master and assuming the latest commit is what should be built. This can result in missing build tags if a merge to master happens before all image builds complete for the prior merge.

We could solve this pretty easily if the build trigger API let a parent build POST a git SHA to its children. Maybe it does and I missed it, or maybe there's another way to solve the problem entirely. @pchico83, any help you can lend here would be greatly appreciated.