Kops: Self Hosting of Resources

[chrislovecnm]

During the DNS attack today on github.com and other internet end-points, we were not able to deploy new K8s clusters. We need to address capability of self-hosting docker containers and other components that are downloaded. For instance with channels:

"https://raw.githubusercontent.com/kubernetes/kops/master/channels/stable": error fetching

Larger companies will want to self-host binaries, dockers, and metadata.

---

Keywords:

• Private S3 bucket
• No public IPs
• Isolated standalone cluster

[kris-nova]

This should be a global discussion about all dependency management in the project. We could/should offer a way to easily overload some of these parameters...

My gut makes me think yml with Viper....

[justinsb]

Agreed - this was really just an error/shortcut I made - I didn't think it through.

I'd say a "base directory" for all our resources would be helpful. And we should pull from there:

• protokube (moving it to a tar file)
• nodeup
• dns-controller docker image (moving it to a tar file)
• the "stable" channel metadata (though this should only be on kops upgrade, which is really just a shortcut over existing edit functionality)

And then you could repoint your base directory to your private builds / whatever.

We already can preload docker images over HTTP and then docker load them - we use that for e2e. So we're close.

I think it's mostly "just" a matter of improving our build process. I saw @mikedanese 's super cool work on getting bazel into the core, and I think it would be great to leverage that once it's in (though we will still want an easy "make" for building kops the CLI tool itself)

[chrislovecnm]

I would say that we can define a docker Reg for -

• all docker images like protocube
• all docker images like etcd

Http / s3 repo for all artifacts like nodeup.

[chrislovecnm]

https://github.com/kubernetes/kops/pull/730 another one

[kris-nova]

I1028 16:17:44.574437 78106 channel.go:68] Loading channel from "https://raw.githubusercontent.com/kubernetes/kops/master/channels/stable"

Will start logging these as I notice them

[chrislovecnm]

Thanks ... nodeup as well. All of the dockers :)

[chrislovecnm]

@robertojrojas this is what I was talking about. This is a tip of the ice burg problem. You interested in assisting?

[robertojrojas]

@chrislovecnm sure! So, there are deps needed at the time kops is executing and deps needed within the cloud provider (with or without internet access), right?

[chrislovecnm]

We have

• k8s binaries
• containers: etcd, dnscontroller, etc, etc
• and cni binaries
• nodeup and kops binaries <- already dynamic.

What is the best way to communicate this to you?

[chrislovecnm]

Oh and thanks. This is a huge need for the community btw. For example DNS attacks have stopped deployments. Aka not good.

[vendrov]

We should support K8S internal containers such as pause-amd64, for that we should pass the flag "--pod_infra_container_image" to the kubelet.
https://github.com/kubernetes/kubernetes/issues/4896

[sstarcher]

External dependencies

CNI

Can be specified using environment variable CNI_VERSION_URL
The current source is storage.googleapis.com https://github.com/kubernetes/kops/blob/789bfcf07b54bdfd852883fde79177950064a099/upup/pkg/fi/cloudup/networking.go#L72

Channel

Can be specified on the command line via --channel
The current source is github.com https://github.com/kubernetes/kops/blob/968366d444d056ffcfc978d5449f53df68b1ac4c/pkg/apis/kops/channel.go#L29

NodeUp / Protokube

The base url can be changed via KOPS_BASE_URL
The specific urls can be changed via NODEUP_URL and PROTOKUBE_IMAGE
The current source is s3

Protokube - https://github.com/kubernetes/kops/blob/de16272c894f1ee9e0da9307c3ce3faeb7d37497/upup/pkg/fi/cloudup/urls.go#L75

NodeUp - https://github.com/kubernetes/kops/blob/de16272c894f1ee9e0da9307c3ce3faeb7d37497/upup/pkg/fi/cloudup/urls.go#L61

Images

If c.Cluster.Spec.KubernetesVersion is a url the following images are loaded from that url.

• kube-proxy
• kube-apiserver
• kube-controller-manager
• kube-scheduler

imagePath := baseURL + "/bin/linux/amd64/" + component + ".tar"

Individual images can be specified for each of the above items via the config - https://github.com/kubernetes/kops/blob/97afdf9f97f56ab5a369b444d2c39621e8e6ba73/pkg/apis/kops/v1alpha2/componentconfig.go

Pause container

Can be specified on the kubelet via --pod-infra-container-image
The current source is gcr

Containers referenced on gcr

gcr.io/google_containers/hyperkube-amd64
gcr.io/google_containers/etcd:2.2.1
gcr.io/google_containers/exechealthz-amd64:1.2
gcr.io/google_containers/cluster-proportional-autoscaler-{{Arch}}:1.0.0
gcr.io/google_containers/kubedns-{{Arch}}:1.9
gcr.io/google_containers/kube-dnsmasq-{{Arch}}:1.4
gcr.io/google_containers/dnsmasq-metrics-{{Arch}}:1.0
gcr.io/google_containers/exechealthz-{{Arch}}:1.2

Currently these containers depend on gcr.io and can no be pre-loaded.

[chrislovecnm]

Networking provider such as weave or calico as well ...

[raghu67]

List above looks quite comprehensive. But I don't see kubelet in the list. Where does that come from?

[chrislovecnm]

Implementation

1. create kops toolbox bill-of-materials - which will generate a list of items that are installed with kops.
2. determine which components do not have configurations in the api
3. chunk up parts and start implementing

Extended List

@sstarcher has a great list, but here are a few more.

1. kubelet
2. custom kernel
3. packages installed kops that are from an external repo, such as docker

[chrislovecnm]

https://github.com/kubernetes/kops/pull/2419 provides a list of inventory items.

https://github.com/kubernetes/kops/issues/2571 will provide a tool to stage those items.

Final PR will be the implementation of API values to allow dynamic setting for the staging area. The staging area for assets will be a docker repo and VFS.

[DerekV]

Larger companies will want to self-host binaries, dockers, and metadata.

And also the security paranoid. In our case, it would potentially simplify some things.
I'm already using my own nodeup bucket, and freeze/promote nodeup testing environment with a small s3sync, but many things just pull from gcr.io without my direct control.

[chrislovecnm]

Still a work in progress

/assign

[chrislovecnm]

/close

as this is implemented

[s1rc0]

as this is implemented
@chrislovecnm , done? where can I read the documentation how to use ?

[smartlin5228]

Is there a document for us to refer?

[michaelajr]

@chrislovecnm Done? Is there a link on how to get started?

[ReillyTevera]

I would also like to see documentation on this as well. Our use case is that we would like to push a docker config to all of our nodes to require that all images must come from our private registry and must be signed which would obviously break cluster components without this.