Kops: Add support for CoreOS

[clanstyles]

(edited by justinsb: topic was changed from "why wasn't coreos the first supported OS" to track the effort to add support for CoreOS)

[justinsb]

kops is starting by providing an upgrade path for kubernetes users using kube-up on AWS, which runs Debian or Ubuntu.

We'll add support for CoreOS down the road, and contributions are welcome!

[justinsb]

Is it OK if I change the title to track the effort to support CoreOS @clanstyles ?

[clanstyles]

Yeah, that's fine with me @justinsb

[clanstyles]

I would like to point out one issue we ran into this week. I spoke with @justinsb about it on slack. The CoreOS team pushes out specific versions of Docker. In this case it's version 1.10.3 (as of right now). This version is now known to have issues under load where it freezes. We can't update our CoreOS clusters unless we use the CoreOS Alpha branch. Under the current system we could roll out specific updates for Docker.

[johnt337]

@justinsb i'm interested in contributing to this, where would be best to start?

[chrislovecnm]

@johnt337 please reach out!

[kris-nova]

@johnt337

Open up a pull request as a work in progress (WIP)... We can start to use the PR to track your changes and discuss ideas..

[johnt337]

@chrislovecnm @kris-nova thanks for responding. I haven't started anything, but will do.

[chrislovecnm]

@johnt337 can we add this to the Jan 1 release?? No is a valid answer :)

[johnt337]

@chrislovecnm my gut says no, but i'd like to look further before I commit to that statement.

So at a glance I appears that lives under nodeup and you already have a structure in place for different os families.

./upup/models/nodeup/_automatic_upgrades/_debian_family
./upup/models/nodeup/_kubernetes_master/_kube-addons/_debian_family
./upup/models/nodeup/docker/_systemd/_debian_family
./upup/models/nodeup/ntp/_aws/_debian_family
./upup/models/nodeup/top/_debian_family
...

My initials thoughts

1. docker/rkt are bundled by default with CoreOS so its a matter of what configs we need to apply to satisfy k8s. Patterns for handling this the "CoreOS way" are already documented. Typically handled as systemd drop-ins via cloud-init
2. Items that are installed via apt-get (or into the /usr partition) would require work arounds to deal with a read only partition and no installer (i.e. bundled via container)
3. Leveraging user-data will maximize configuration and immutability of the deployment
4. Size limits of cloud-init would influence approach
   
   
   
   • 16K limit - http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html
   
   
   • Embedding all the k8s manifests, certs, and add-ons directly in the user-data may not feasible if they exceed this limit
   
   
   • They might need to be externalized and retrieved during boot time (would that go in KOPS_STATE_STORE?)
   
   
5. Systemd units hopefully would be portable (need to confirm)
6. How much would you want to handle with nodeup versus straight cloud-init?
7. Networking - what type of networking would you want to tackle first?
8. Preference between cloud-init or ignition?

If you feel these are valid comments/observations and in the right direction, I can start the PR and reverse my way out of the upup package.

[callmeradical]

To number 4, particularly embedding the k8s manifests etc., couldn't it be bundled in a container as well? If the installed items are also to be containers?

[johnt337]

@chrislovecnm @kris-nova I opened PR #854 to move this conversation along.

[devops-senpai]

👍
please add support for CoreOS

[justinsb]

Experimental work here: https://github.com/kubernetes/kops/pull/1480

(It does work, or at least works for me, I'm just whittling down the mega-PR into reviewable chunks)

[aledbf]

@justinsb this can be closed. Covered by #1813