Cluster-api: RFE: Rally on a unified image stamping utility.

[timothysc]

Describe the solution you'd like

Right now several CAPI providers implement different means for installing software. The purpose of this issue to rally a conversation in the post v1alpha1 world on having a grand-unified image stamping utility that can be used by multiple parties.

/kind feature

/cc @vincepri @frapposelli @detiber @sflxn @chuckha

[chuckha]

I know @justinsb has thoughts here as well

[neolit123]

cc

[chuckha]

Here is how cluster-api-provider-aws does it: https://github.com/kubernetes-sigs/cluster-api-provider-aws/tree/master/build

Here is what that is based on: https://github.com/heptiolabs/wardroom

Both use packer. The biggest difference here is that cluster-api-provider-aws uses containerd as the container runtime.

[sflxn]

I just want to point out that for the cluster-api-provider-vsphere, there are two artifacts that requires stamping.

1. The initial install OVA (basically a fancy AMI with UI wizards) artifact
2. The node image.

We need both images because we support both self-service workflow (default cluster api with bootstrap VM/KIND) and a management cluster workflow. The install OVA deploys this management cluster.

cluster-api-provider-vsphere also has an image stamping process that we use to build the management bootstrap OVA. The location of our stamper is in our repo, here. This process has a lot of tech built from a previous Vmware OSS project. It doesn't use packer or any external tools and doesn't require a hypervisor. We actually build the VMDKs and package the OVA ourselves.

We are not yet stamping the node image. We would like to do so, but to do so, it brings up a lot of other questions.

• If we do node stamping, should we advertise this feature available for users to do so? If so, we need to improve the method for describing the image.
• If node images are created, where are they stored? Today, we require users to download a cloud-init image and create a vm template on vSphere. The cloud-init image is a known baked image hosted by a linux distro. If we start stamping node images, where should they be hosted?

These and other questions have prevented us from pursuing node stamping, but it's very much something we would like to do.

_I think before we head down the path of designing a solution, we should first ask what problems we want to solve first. I hate to bring up the word requirements among engineers, but we should try to solve this one time._

[sflxn]

I want to bring up an idea that shows that this machine stamping can be solved in many ways if we were to look at it from different angles and impress upon the community that we really need to decide which problems we want to solve.

I heard last week about the idea of a meta container for machine stamping. What if a "meta node" could be built, pulled from some registry, and started like a VM? Well, this is something that actually exist. At VMW, we built an OSS project called VIC. Kata container does something very similar. Both projects reuse the foundation of Docker to build a stamping process. A user can describe a "node" with a dockerfile, build a standard docker container and push it up to a registry.

During deployment, a user uses VIC/Kata to pull down the container image that then gets stamped into VM images and runs them.

In this approach, we answer quite a few questions:

1. How do we describe the VM image?
2. Where do we host the images?
3. Is there a way to ensure the images have not been tampered?
4. How can the user deploy the VMs from the image and have cached versions of the image around?

The last question is a really cool feature. Since both VIC and Kata implements an image store, both retain multiple versions of node images around. Every time a user does docker pull mycloudinit-node:Vxxx, that image gets stamped and stored locally, making future standing up of that VM very quick because the version of the image was previously cached.

I am not proposing using the same technology to do stamping. However, the workflow pattern solves a lot of problems. This same pattern can be use with linux kit in place of docker.

The point I am trying to make is that depending on which problems we want to solve, the solution maybe very different. Machine stamping is not a hard problem, but solving the concerns that involves all the aspects of the user workflow may take us down a different path. Of course, we may also decide the entire workflow is just too big to solve in one go. If that is the case, we can take a phased approach.

[Amit-PivotalLabs]

How coupled is the node/VM image, the software installed on it, and the specific Cluster API implementation? When a user chooses a Cluster API implementation to use, are they bound to a choice of VM image and installed software, or is there some ability to mix and match? When a user wants to update just the VM image or just the installed software or just the Cluster API implementation, can that be done or is it an all-in-one deal?

I can imagine wanting to install (non-container) software on the image in addition to the core K8s software (API, scheduler, kube-proxy, kubelet, etc.). I can also imagine operators managing a large fleet of (possibly heterogeneous) clusters wanting to apply a patched OS image to all VMs in all clusters in light of, say, a kernel CVE, without wanting to change any of the K8s (or other system-level) software. I could imagine multiple Cluster API implementations wanting to leverage a common VM image line without any one implementer team responsible for maintaining the image line.

I'm curious about what thoughts folks have with regards to decoupling for modularity and reusability between:

• image stamping, patching, distribution, and application to fleets
• core K8s software packaging, patching, distribution, and application to fleets
• cluster API implementations' packaging, patching, distribution, and application to an operator's Cluster API control plane being used to manage said fleets

[moshloop]

I have a suggestion similar to the VIC stamping model. Rather than trying to create a VM/Template from a docker container (With all the issues of VM customization/optimization that will be encountered), use the Dockerfile as an "interface" to execute commands on an existing VM.

This idea stems from jessfraz's implementation of docker build in bash.

There would be some limitations around mapping docker base images to cloud provider images, but this could be partially overcome by:

• Walking the FROM image tree and executing the commands in reverse
  AND/OR
• Limiting FROM to known base images

[timothysc]

/assign @timothysc

I'm going to start a KEP on this work pretty soon and will send out emails on requirements as well.

[vincepri]

/area release

[timothysc]

closing - xref: https://github.com/kubernetes/enhancements/pull/1102