Cluster-api: [cabpk] Define supported combinations

[moshloop]

We should define what combinations of OS / K8S / CRI / Cloud this provider will support, perhaps breaking it down into 3 categories:

1. Supported with E2E test
2. Known to work
3. Unsupported / Known not to work

Operating Systems:

• [x] Ubuntu 18.04
• [ ] Ubuntu 16.04
• [x] CentOS 7.2
• [ ] Amazon Linux 2
• [ ] RHEL 7.4
• [ ] RHEL 7.5
• [ ] RHEL 8
• [ ] RHEL Atomic
• [ ] CoreOS
• [ ] Debian
• [ ] Container Linux

Kubernetes:

• [x] Latest Kubernetes
• [ ] Latest patch for Kubernetes N - 3
• [ ] Open Shift v4

CRI:

• [x] Containerd N - ?
• [ ] Docker
• [ ] CRI-O

[chuckha]

This doesn't feel like the right repository for this. This feels like it should go in infrastructure providers.

This bootstrap provider will be providing a script for the machine controller to do something with. Our support matrix should be the same as kubeadm with any other details left up to the machine controller.

Can you tell me how you're thinking about this and why you created this issue in this repo?

[moshloop]

The cloud-init or bash that is returned by this provider is going to have some underlying assumptions/requirements - you can't just hand-off a script and expect it to work in every scenario. The scenario's under which it will work form the supported combinations.

For example:

• CoreOS prefers ignition, so cloud-init may not work or may require tweaking
• Is the cloud-init expected to work off a vanilla OS, or are the dependencies like kubeadm/docker expected to be pre-installed ?
• If the dependencies are expected to be pre-installed, how does the cloud-init script verify that they are the correct dependencies?
• if the dependencies are not installed, then installing them will be very OS specific
• Applying networking configuration will vary by OS

If this provider isn't going to include these items in scope, then it probably shouldn't be generating cloud-init either, rather an intermediary format that another provider can read to merge with it's own config and generate cloud-init

[sethp-nr]

Those are very good points. As someone interested in adapting the bootstrap flow to CoreOS, I can guarantee that the current cloud-init won't work (CoreOS has a ground-up reimplementation of cloud-init in golang, so it's not possible to add python modules like kubeadm to it), and pre-installing dependencies is sort of... awkward, to say the least. And don't get me started on networking 😄

That said,

rather an intermediary format that another provider can read to merge with it's own config and generate cloud-init

I had understood that the "intermediary format" was basically going to be kubeadm – if this provider picks a default format (cloud-config, bash, etc.) to wrap that in it's not meant to be exclusive of other options.

Honestly it feels a little early yet to define the support matrix. IMO it would be nice to see something working before we make those decisions so we can see how many of those assumptions we end up making (or not!).

[moshloop]

The support matrix doesn't need to be final, but I do think it is important to define the goals and non-goals, If the plan is to support CoreOS at some point it has implications on the design. Same applies to baked images vs vanilla

[neolit123]

@moshloop
you are asking for something that the k8s ecosystem is struggling to define and it's unlikely to happen in this project.

Operating Systems:

kubeadm does not have a support matrix for distros and it will never have one...
i don't think this provider should either.

if your distro/'image/machine does not work with kubeadm or wrapper scripts, it's up to you to fix it and if
if you can't, you should just move to Ubuntu 16.04 which is a known distro to work.
we have kubeadm docs for Fedora and ContainerLinux but we don't track versions there.

selinux is problematic too, disabling it is the advisable action for now.
else keep it enabled but add labels to key paths.

Kubernetes:

currently there is no easy way to get the dependencies of k8s component (such as kubeadm) in a machine readable way. @yastij is working on something in this area.

the common setup for kubeadm nodes, you'd need a supported docker version, control-plane and kube-proxy that is X or X-1, coredns and etcd can be obtained from kubeadm config images list and a sane version for CNI (which is more problematic than it seems).

CRI:

random comments:

• too many versions and too many options
• 70% of people are using docker, 15% containerd, 15% cri-o
• docker is tested by users a lot
• we really don't track supported containerd versions but it's known to work OK
• cri-o is untested in kubeadm e2e

[moshloop]

@neolit123 thanks - this is really good feedback.

too many versions and too many options

I am not by any means saying we should support all of these combinations, just that these are the potential dimensions in which combinations could come from.

I would suggest we just break these down into:
1) 0.1
2) 0.1+ or 1.0+
3) 2.0 or Never

70% of people are using docker, 15% containerd, 15% cri-o
docker is tested by users a lot
we really don't track supported containerd versions but it's known to work OK

I agree, those are the same arguments I would have for CAPA / CAPV to switch from containerd to docker.

The supported runtimes for this provider could then be:

• Docker - Supported with E2E
• Containerd - Unsupported, but known to work
• CRI-O - Unsupported and untested

kubeadm does not have a support matrix for distros and it will never have one...

I agree, kubeadm should only have prerequisites that must be met.

i don't think this provider should either.

If the output is OS specific (e.g. cloud-init/ignition), then it follows it should specify that OS.
If the output is not OS specific (e.g. bash), then it could defer to kubeadm prerequisites

[yastij]

ref kubernetes/kubernetes#79366

[chuckha]

/milestone next

[k8s-ci-robot]

@chuckha: The provided milestone is not valid for this repository. Milestones in this repository: [Next, v0.1]

Use /milestone clear to clear the milestone.

In response to this:

/milestone next

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[chuckha]

/milestone Next

[embik]

Just a heads up, Fedora CoreOS (FCOS) won't support cloud-init as far as I know. It would be neat if this provider supported different output, which is mentioned in kubernetes-sigs/cluster-api-bootstrap-provider-kubeadm#16 anway.

This issue should probably focus on output format, not operating systems; so, support cloud-init, the old Ingition scheme for CoreOS and the new Ignition scheme for FCOS. Something like that. Translating the initial bootstrap data from a generic scheme (cloud-init, ignition, etc) to operating system configuration is IMHO part of the tool (cloud-init, ignition) used and not something this provider has to define.

[detiber]

@embik hmm, I don't have any strenuous objections to adding some type of output filter that could be selected.

@chuckha @ncdc We could potentially add a new field KubeadmConfig.Spec.Format, that defaults to cloud-config (the more formal name for the declarative config used by cloud-init), but that would allow over time the addition of other output formats such as the ones mentioned above.

To make this happen we could basically use something similar to what @fabriziopandini built for running cloud-config for CAPD, we limit to a set of known actions and their attributes, and each output filter would just need to know how to translate each of those actions and attributes into the correct config format.

[ncdc]

@detiber +1 to supporting multiple output formats

[chuckha]

I think multiple output formats is pretty reasonable! @detiber do you see this as milestone next or first release? I'm leaning towards next, but want to get your input.

[detiber]

@chuckha What if we added a field for the output format for v1alpha2, and then additional output providers could be added for later v1alpha2-based releases as well without having to wait for v1alpha3.

[chuckha]

that sounds fine to me. I'll make the issue to add a new field that is a no-op/forced cloud-init for this release

[chuckha]

@moshloop I think this is a documentation issue. Do you think you could help write up some docs, even briefly, about supported combinations that you've outlined in the original issue?

/assign @moshloop
/kind documentation
/priority important-longterm

[chuckha]

/milestone v0.3.0

[dlipovetsky]

+1 to supporting multiple formats

[ncdc]

@moshloop did you see Chuck's comment above (https://github.com/kubernetes-sigs/cluster-api/issues/1576#issuecomment-540741462)?

[moshloop]

@chuckha @ncdc - While there is a documentation component to this issue, it isn't the defining factor.

What is more important is deciding on what is and is not supported, and what is necessary to take something from unsupported to supported.

Part of this involves documenting what supported entails e.g. We accept bug reports and test for supported combinations and ensure they work before release.

[ncdc]

From a bootstrapping perspective, cabpk generates cloud-init cloud config yaml with the expectation that it should be able to run on most or all debian/ubuntu; centos/rhel, and amazon linux 2.

As to supported Kubernetes versions, we can and should document that (based on the versions of the kubeadm types that we support).

As to container runtime, I don't think there's anything prescriptive in the code that assumes or forces one CRI over another.

If bootstrapping doesn't work with a particular OS, ideally someone would report that as an issue, and we can attempt to fix the problem.

[vincepri]

Bumping this to Next until we have someone working on it.

/milestone Next

[vincepri]

@moshloop are you still interested in working on this?

[vincepri]

/priority backlog

[rikatz]

@vincepri Do you know if there's someone working in extending this to support additional output providers as commented by Jason here?

I was making some tests here with ClusterAPI and really want to make this work with Flatcar Linux but in the right way :D

If no one is working on this, I'll take a look into the code and how hard is this to be extended it.

Thank you!

[vincepri]

@rikatz I don't think anyone has started working on that yet

[rikatz]

@vincepri it seems that the API already support the Format as in here. It was added in this commit

I'll take a look into the reconciler now to check whether this field is being used or not, and what's the size of effort to separate/create multiple output providers.

Thank you

[stormobile]

I think coreos/fcos/flatcar support is desired as those distributions as a base for cluster nodes are lighter and their lcm models correspond to k8s lcm models. There is at least one fork of cabpk https://github.com/minsheng-fintech-corp-ltd/cluster-api-bootstrap-provider-kubeadm-ignition that changes output format to ignition but this approach introduces a number of obvious challenges.

[dhawal55]

@rikatz I'm very interested in helping with this effort (support ignition output format). Let me know how you want to collaborate on this activity.

[rikatz]

@dhawal55 sure :) I've made a simple PoC to see if this was viable, but I'm getting really crowded with other stuff here.

If you want to take care of this, and got more cycles than me to deal with this please, really please go ahead :)

[dhawal55]

K, I will start getting familiar with kubeadm provider. From what I have seen so far, I don't see the Format field used anywhere

[detiber]

@dhawal55 that is correct, the Format field isn't currently used anywhere, https://github.com/kubernetes-sigs/cluster-api/issues/1582 is the tracking issue for it.

[ncdc]

Note, I just closed #1582 (see my comment at the bottom for why).

I think there are a couple possible ways to resolve this issue:

1. Document that you must use image-builder to create supported OS images
2. Document the requirements for an OS image to work with CABPK

2 is potentially a much larger effort than 1, especially if we were to document how to create images for different CRIs, different OSes, etc. My vote is to do 1, with the expectation that image-builder is sufficiently documented for how to pick a CRI and any other variations a user might desire.

[vincepri]

/close

Closing for lack of updates, we need to add some documentation bits about the images requirements

[k8s-ci-robot]

@vincepri: Closing this issue.

In response to this:

/close

Closing for lack of updates, we need to add some documentation bits about the images requirements

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.