Repo2docker: Explore CNCF v3 buildpacks

[yuvipanda]

From @jchesterpivotal in https://github.com/jupyter/repo2docker/issues/707#issuecomment-505904267

By way of warning, what follows is hilariously biased: I've several times worked on two generations of buildpack technology over the past 5 years. Pride makes me defensive.

As it was related to me by a Red Hatter I asked, s2i was created largely because the previous generations of buildpack lifecycles from Heroku (v2a) and Cloud Foundry (v2b) were optimised to a rootfs+tarball target (Heroku's term is "slug", Cloud Foundry's is "droplet"). That was considered unsuitable for OpenShift v3, which was an image-centric architecture.

Whereas Heroku and Cloud Foundry would meet you at code and hid the underlying container infrastructure, OpenShift would meet you at the image, so the latter (this is a personal opinion) had a business need for something _like_ buildpacks to reduce the convenience gap.

But s2i never really found a home outside of OpenShift, while buildpacks have flourished in two massive, independent but genetically-related ecosystems.

Critically, the emergence of the v2 registry API enables features (particularly layer rebasing) that were previously impossible. In addition Google's Container Tools team developed and maintain the google-gocontainerregistry library which allows us to perform construction and rebasing operations with or without the docker daemon. The design of CNBs takes full advantage of both of these advances.

By way of speed improvements: We have observed some Java rebuilds drop from minutes to milliseconds. We expect large-cluster rollouts to drop from dozens of hours to potentially minutes.

Edit: I should add, your reasons for moving off s2i would apply to v2a and v2b buildpack lifecycles as well. One of the motivating problems faced by both Pivotal and Heroku has been exactly this sort of combinatorial explosion; CNBs are designed to make it possible to more easily compose buildpacks developed independently of one another.

I've bolded the bits that I think are most relevant to us. It would be great if someone could take a look at https://buildpacks.io to see if we can base repo2docker off v3 of buildpacks. http://words.yuvi.in/post/why-not-s2i/ contains reasons why we moved off s2i (which is similar to v2 of buildpacks).

A useful test case would be to try to make:

1. A buildpack for environment.yml
2. A buildpack for install.R
3. A buildpack for postBuild
4. A buildpack for apt.txt

And then see how easy / hard it is to have a repo with any combination of these 4 files produce one single image. My rudimentary math skills tell me that there's 4! possible combinations here (24), and we shouldn't have to write more than 4 buildpacks...

[betatim]

Is there a list somewhere of existing buildpacks and their implementation? I had a quick look at https://buildpacks.io/docs/ but couldn't find that list.

[yuvipanda]

From buildpacks.slack.com I found https://github.com/cloudfoundry/conda-cnb

[consideRatio]

Wow this was exciting stuff @yuvipanda ! :heart: :tada:

[zmackie]

https://github.com/buildpack/pack is probably the best way to get started playing around locally. (Disclosure I'm the "anchor" (tech lead) on the pivotal buildpacks team).
CNBs are, for now, generally consumed through a builder image (what that is is not especially important for the purpose of this comment). You can see the "blessed" builder images by running

# pack suggest-builders
Suggested builders:
    Cloud Foundry:     cloudfoundry/cnb:bionic         Ubuntu bionic base image with buildpacks for Java, NodeJS and Golang
    Cloud Foundry:     cloudfoundry/cnb:cflinuxfs3     cflinuxfs3 base image with buildpacks for Java, NodeJS, Python, Golang, PHP, HTTPD and NGINX
    Heroku:            heroku/buildpacks:18            heroku-18 base image with buildpacks for Ruby, Java, Node.js, Python, Golang, & PHP

If you inspect our cflinuxfs3 builder image, for example, you can get an idea of the CNBs that we've published that are "ready", although we have a bunch more in flight:

# pack inspect-builder cloudfoundry/cnb:cflinuxfs3
Inspecting builder: cloudfoundry/cnb:cflinuxfs3

Remote
------

Description: cflinuxfs3 base image with buildpacks for Java, NodeJS, Python, Golang, PHP, HTTPD and NGINX

Stack: org.cloudfoundry.stacks.cflinuxfs3

Lifecycle Version: 0.4.0

Run Images:
  cloudfoundry/run:full-cnb

Buildpacks:
  ID                                                VERSION           LATEST
  org.cloudfoundry.node-engine                      0.0.46            true
  org.cloudfoundry.npm                              0.0.29            true
  org.cloudfoundry.yarn                             0.0.24            true
  org.cloudfoundry.python                           0.0.20            true
  org.cloudfoundry.pip                              0.0.20            true
  org.cloudfoundry.pipenv                           0.0.14            true
  org.cloudfoundry.conda                            0.0.13            true
  org.cloudfoundry.go-compiler                      0.0.20            true
  org.cloudfoundry.go-mod                           0.0.18            true
  org.cloudfoundry.dep                              0.0.17            true
  org.cloudfoundry.php-dist                         0.0.28            true
  org.cloudfoundry.php-composer                     0.0.16            true
  org.cloudfoundry.httpd                            0.0.18            true
  org.cloudfoundry.nginx                            0.0.20            true
  org.cloudfoundry.php-web                          0.0.19            true
  org.cloudfoundry.openjdk                          1.0.0-RC03        true
  org.cloudfoundry.buildsystem                      1.0.0-RC03        true
  org.cloudfoundry.jvmapplication                   1.0.0-RC03        true
  org.cloudfoundry.azureapplicationinsights         1.0.0-RC03        true
  org.cloudfoundry.debug                            1.0.0-RC03        true
  org.cloudfoundry.googlestackdriver                1.0.0-RC03        true
  org.cloudfoundry.jmx                              1.0.0-RC03        true
  org.cloudfoundry.procfile                         1.0.0-RC03        true
  org.cloudfoundry.dotnet-core-conf                 0.0.20            true
  org.cloudfoundry.archiveexpanding                 1.0.0-RC03        true
  org.cloudfoundry.tomcat                           1.0.0-RC03        true
  org.cloudfoundry.jdbc                             1.0.0-RC03        true
  org.cloudfoundry.springautoreconfiguration        1.0.0-RC03        true
  org.cloudfoundry.springboot                       1.0.0-RC03        true
  org.cloudfoundry.distzip                          1.0.0-RC03        true

[zmackie]

I should further add that our slack https://slack.buildpacks.io/ is a great place to chat about all things buildpacks and https://hub.docker.com/r/cloudfoundry/cnb is a decent overview with links to a bunch of the CNBs.

[zmackie]

I'll dig in a bit more because I think this is a really cool project:

A buildpack for environment.yml

https://github.com/cloudfoundry/conda-cnb

A buildpack for install.R

We have not written an R cnb but its somewhere in our backlog. a julia cnb is slightly higher in priority.

A buildpack for postBuild

We don't have this written, but I don't think it would be massively hard to implement

A buildpack for apt.txt

This will possibly be handled by https://github.com/buildpack/rfcs/pull/23

[ltalirz]

In case it's useful, here is a link to a recent post on the topic by Jose Diaz-Gonzalez, the lead developer of dokku, including some notes on how CNB tech differs between cloudfoundry, heroku and herokuish
https://dokku.github.io/technology/comparing-buildpack-v3-to-herokuish

[manics]

Is this something that could be used for _new_ repo2docker buildpacks, e.g. Java? https://github.com/jupyter/repo2docker/issues/780