Kops: Add support for the platform of Openstack

[zengchen1024]

kops supports several platforms like AWS, GCE etc, but Openstack is not available. This issue tries to add support for the cloud platform of Openstack, which will deploy Kubernetes on it.

[chrislovecnm]

We have office hours on Friday, as listed in the README. If you can swing by let use know.

[zengchen1024]

@chrislovecnm Thank you for your responding. Sorry I am afraid I can't join the meeting, because I will have an examination on Saturday. But I have invited my colleague(@freesky-edward) to join instead.

[chrislovecnm]

If he is in China, we could setup a time that is not in the middle of the night ;)

[zengchen1024]

@chrislovecnm Many thanks for your kind intentions. Yeah, both of us are in China. It would be great if setting up a special time for the contributors like us.

[chrislovecnm]

Some time next week? Thursday afternoon Denver time?

[chrislovecnm]

Take a look at the vsphere implantation btw. It will be the closest to what you have. Cloud groups / instance groups is probably the biggest abstraction that we need to figure out.

[zengchen1024]

@chrislovecnm I am ok with the time. Please tell me the exact time on Thursday. Thanks! I have been learning the Kops and reading the codes of 'nodeup'.

[chrislovecnm]

Thursday 5:30 Denver?

[dims]

cc @bugosiphionah @ss22ever

[bugosiphionah]

Thanks for the ping @dims. @chrislovecnm I will join the chat as well and find out how to get involved as I am looking at working on this as part of my outreachy project if accepted.

[chrislovecnm]

@balajismaniam we can set up another time next week, but 5:30 is the latest that I can meet. We need to get this documented and determine who wants to code what.

[diafour]

Is it possible to join? Is this the correct time?
https://www.timeanddate.com/worldclock/fixedtime.html?msg=kops+openstack+support&iso=20171019T1730&p1=75&ah=1

[chrislovecnm]

That is it, and I will open the zoom room that is in the readme

[huzhengchuan]

cc @huzhengchuan

[zengchen1024]

@chrislovecnm The meeting time is ok for me. I will join on time. Thanks!

[ss22ever]

Thanks, @dims, @chrislovecnm works for me too.

[freesky-edward]

@zengchen1024 thanks for the ping. I will join this meeting.
@chrislovecnm will you send the meeting invite? please add me if so.

[chrislovecnm]

I do not have anyone's email address, so I will just open the bridge. If someone else wants to rummage around commits for email they can send out an invite

[zengchen1024]

@chrislovecnm Do we just access the URL listed in the README to join the meeting or some other URLs? thanks.

[chrislovecnm]

https://zoom.us/my/k8ssigaws to be clear

[chrislovecnm]

remind me and I will figure out where to post the video. I will get an email from zoom.

To recap:

1. vendor the required openstack dependencies in a PR by itself - https://github.com/kubernetes/kops/blob/master/docs/development/dependencies.md - missed talking about this
2. create vfs store for swift - get kops create cluster working with swift
3. start by coding some small tasks - apply_cluster.go is where most of wiring with tasks are done.
4. need to determine how to model your instance groups - @dims we have office hours next Friday, and you can get some feedback from @justinsb on how he is planning to do it with bare metal
5. you may need to make some modifications to nodeup in order to bootstrap and virtual instance in OpenStack, but the changes should be minimal.

In order to develop in China, look at the guide for using kops in AWS China's region for how to setup docker registry. Otherwise, HTTP proxy, which is complete, or assets API, which is still a work in progress is needed.

[dims]

Thanks @chrislovecnm - here are my notes

• pointers to how things work
  https://github.com/kubernetes/kops/issues/3612
  https://github.com/kubernetes/kops/blob/master/docs/development/how_it_works.md

• Need a place to store things
  https://github.com/kubernetes/kops/tree/master/util/pkg/vfs

• kops tasks unique for each cloud
  https://github.com/kubernetes/kops/tree/master/upup/pkg/fi/cloudup
  https://github.com/kubernetes/kops/tree/master/upup/pkg/fi/cloudup/vsphere
  https://github.com/kubernetes/kops/tree/master/upup/pkg/fi/cloudup/gcetasks

• main piece of code that calls others
  https://github.com/kubernetes/kops/blob/master/upup/pkg/fi/cloudup/apply_cluster.go

• Minimal kops example
  https://github.com/kubernetes/kops/tree/5199e7168ac93260aadf33d25e7b5c0b03d4c29c/tests/integration/create_cluster/minimal

• nodeup binary - pulls binaries and runs things

• weave for networking may work well with openstack (just need specific ports to be open)

• gossip for DNS may be easy to start with

• Start with VFS implementation that stores data, possibly OpenStack swift

• Start with vsphere tasks and see how to write openstack one
  
  
  
  • then look at GCE which is much better
  
  
  • then look at AWS which has the most features
  
  
• Start with single node, just kubelet via cloud init script
  
  
  
  • then add master node (will need gossip at this point)
  
  
  • then add a second kubelet node (will need gossip and weave)
  
  
• When you start, just make sure all ports are open then lock them down over time.

[zengchen1024]

@chrislovecnm @dims thanks for your guidance. I will start working on this "Start with VFS implementation that stores data, possibly OpenStack swift". I think it is a good start. see #3676

[chrislovecnm]

/area openstack

[zetaab]

could we make this object storage configurable? Like we are using openstack but we do not have swift installed, instead we have product which implements S3 API inside our datacenter. So we could use same S3 in openstack as well (if the s3 address is just configurable)

[dims]

@zetaab yes, makes sense.

[chrislovecnm]

@zetaab already works that way. VMware folks used an in house s3 clone. ftp and file:// are two others to access as well. I think file will work now with vfs, we would need to test it.

The thing to remember is that secrets reside in the object store.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[zengchen1024]

@fejta-bot Sorry for that. Because of my company's work, I had no time to add new PR to this issue for the past two months. But, from now on, I come back to work on it. Because there are still a lot of works to do to finish this issue, I need more time. Sorry!

[zengchen1024]

/remove-lifecycle stale

[chrislovecnm]

@zengchen1024 ... that is a silly bot, not a human.

/cc @fejta we may want to add the fact that this is a bot and not a human.

[gmwingard]

What's the status on this? I've got a dev deployment of OpenStack at my job, and would be more than happy to contribute.

[chrislovecnm]

@zengchen1024 which fit tasks are not completed yet? What about DNS? Can you break down what work needs to be completed?

[zetaab]

@chrislovecnm coredns exist already in kops (used in vmware?), so why we could not just use that in openstack

[chrislovecnm]

@zetaab we use dns-controller to set the DNS so we need support in dns-controller for that. More wiring and stuff.

[zengchen1024]

@gmwingard there are still many work to do. such as network, auto scaling etc. We can do it together.
@chrislovecnm I think there are two big steps to be completed. First, it needs complete all fit tasks. At present, only volume and security group tasks are done. network and auto scaling still wait to be done. Second, works belonging to nodeup have not started.

[chrislovecnm]

@zengchen1024 can you layout more detail on what the tasks need to do? I can guide on how to write and test the tasks, but some architecture guidance would help. When you have a chance more context and detail would be appreciated!

[zengchen1024]

@chrislovecnm Sorry to reply so late. From my understanding, at least the following fit tasks need to be done.

1. router
2. subnet
3. network
4. autoscaling. use openstack/Heat to implement this tasks

For 1~3, see https://developer.openstack.org/api-ref/network/v2/ to get details

[zetaab]

@zengchen1024 do you have plans how you are going to implement that autoscaling? Using heat yes, but how it is done exactly? At least we do not have senlin / ceilometer installed

[zengchen1024]

@zetaab Thank you for your kind reminding! At present, I almost have finished the tasks of network, volume, and security group/rule. The next step is to implement the task of autoscaling. But I have not started to consider these details. If you have interest in implementing it, I will be very happy!

[zetaab]

@hogepodge @dims @anguslees do you have ideas how that autoscaling part should be implemented to kops openstack installation? Making server group(s) using heat, but what kind of autoscaling should be used? Like I mentioned in earlier post at least we do not have anything installed which is needed for autoscaling server group, but I would like to hear your opinion what the best plugin for that. Senlin or ceilometer or something else?

[dims]

@zetaab why can't we use what's available in the environment? https://coreos.com/blog/autoscaling-with-prometheus-and-kubernetes-metrics-apis

[zetaab]

@dims i am not talking about pod autoscaler. I am talking about node auto scaling. For instance in aws kops implements auto scaling groups which means that it will automatically add/remove nodes depending on load

[dims]

@zetaab node level metrics are available also in the metrics API, not just pod-level

[zetaab]

so it means that we could make just normal stack with heat template server group. Then modify that server group using stack update

For start I think we could have just static number of servers? and then add autoscaling? @chrislovecnm what you think?

[dims]

@zetaab adding a requirement for OpenStack Heat would make it difficult for existing clouds that may not have it. any thoughts?

[zetaab]

also one thing that we should think of: at least for me security groups does not work in lbaas (using haproxy).

[zetaab]

Things that we need to do (before we can start installing machines, feel free to comment if something is missing):

• servergroups (instancegroups)
• servers (instances)
• some method to make LB for masters (lbaas? If we choose lbaas could someone confirm is security groups working in octavia? Because like I mentioned earlier in haproxy those does not work)

I think all of these tasks can be started now. So if someone has free time please go ahead.

We can later code heat support as well, but lets start without heat support (like AWS support both installation mechanisms if I read the code correctly). Also autoscaling can be added later, lets try to get "basic setup" up and running and after that start advanced stuff.

[rguichard]

@zetaab security groups do work with lbaas v2 (with haproxy as backend). You have to set them on the neutron port of the fip. If you don't want Heat to be a prerequisite, don't event think about Octavia ^^

[zetaab]

@rguichard using heat is not problem for me, but there are some guys who said that they do not have heat installed in openstack production :) That is why we should somehow support both ways to install this

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[mikesplain]

/remove-lifecycle stale

[ricolin]

IMO to support Heat, shouldn't be the problem, the autoscaling future can be optional and implement later anyway.
So, try to clarify, Heat does not depend on any specific monitor. Whatever can support to send a trust+http request, will be a good fit for Heat. (Aodh just happen to be the best integration now)

I'm planning to work on this soon, so let's see what we got from the discussion in PTG

[xgerman]

@zetaab can you give me details on security groups - Octavia supports security groups but we are not exposing them for the transitive trust. Is that your issue?

[zetaab]

@xgerman I have not used octavia, I think this problem is fixed in Octavia (hopefully). The problem exist when using haproxy lbaasv2, there you can setup sec groups to haproxy ports but those does not have any effect to anything.

[lblackstone]

FYI, @jrperritt and I were working on adding OpenStack support, but won't be able to continue that work at this time. Here's the WIP branch in case somebody wants to pick that up and run with it: https://github.com/jrperritt/kops/commits/openstack

[xgerman]

@zetaab you might be interested in how this shakes out: https://review.openstack.org/#/c/602564/

[drekle]

I've picked this up. I am currently going through protokube and nodeup tasks on each of my nodes.

[drekle]

What is the expectation here around DNSaaS?

I think the only thing really stopping me now is my lack of DNS. etcd, kubelet, kube-apiserver all running however I am resorting to implementing gossip for openstack to get around missing designate. Multi-master nodes will need to connect to multiple etcd instances. All nodes will need the address of the api server loadbalancer.

Is anyone else using an external provider, such as route53, for their openstack dns? It seems wrong to put in here as in every other case it appears kops is deploying to a cloud which encapsulates all functionality. I can likely clean this up by using my own devstack for the time being.

[zetaab]

I would say we cannot trust that people do have designate. We are using just hosts files in each node. We are updating those using ansible (not very good solution but works). But as far I remember there is already coredns and gossip support in kops? could we use those?

[drekle]

yes @zetaab . I have wrote a bit of gossip for openstack already as I am one of the people that does not have designate. I'll probably struggle a bit on creating the entry for the api-loadbalancer, im clueless here.

I am mostly preparing for a demo of this, so I have some hard coded resolutions to clean up prior to committing to my fork.

[zetaab]

@drekle could you make some PRs that you have made? Its easier to others to continue if you could show what you have done

[drekle]

@zetaab Sorry I am not actually focusing on providing individual features. Really just want this to completely work as I need concepts that it provides, such as instance groups, as we use kubespray at the moment.

Ill create this Working PR for now
https://github.com/drekle/kops/pull/1

My deployment works with one hardcoded bit, that maybe I just add support for.

$ kubectl get nodes
NAME                                         STATUS    ROLES     AGE       VERSION
delemon.k8s.local-master-cloud-zone-1-1   Ready     master    12m       v1.10.6
delemon.k8s.local-master-cloud-zone-2-1   Ready     master    13m       v1.10.6
delemon.k8s.local-master-cloud-zone-3-1   Ready     master    13m       v1.10.6
delemon.k8s.local-nodes-1                    Ready     node      11m       v1.10.6

Internally resolution to the masters will use duplicate entries to the api-server hostname in /etc/hosts. Externally I am spinning up an openstack loadbalancer, associating a floating ip to that which populates the generated kube/config

[drekle]

In Openstack there are disparate availability zones for compute, network, in storage. If these do not map 1-to-1 there is a bug here which is hard to solve.
https://github.com/kubernetes/kops/blob/master/pkg/model/master_volumes.go#L248

Any thoughts on this?

[zetaab]

@drekle we have situation that compute-zones are zone-1, zone-2, zone-3 and we have only one volume zone which is called nova. In kubernetes its pretty easy to solve (I have implemented ignore volume-az thing there). I think we can make some dirty fix for it in kops as well

[drekle]

@zetaab ours is the same. We will be making PR's for a complete deployment hopefully early January.

[zetaab]

@drekle any news? :)

[drekle]

Hey @zetaab. Been out for the holiday. We have about 3 PR's that we will raise hopefully in the next couple of days. These should create a cluster however not sure delete or upgrade yet.

[zetaab]

I started doing those delete tasks https://github.com/kubernetes/kops/pull/6216, that will delete all current resources. I think that its enough if it can just create new running cluster. Lets solve other problems after that code is merged

[hogepodge]

/sig openstack

[drekle]

I've rebased and tested a deployment today. Raised the larger set of PR's that I have. If this is merged I can raise the protokube PR's to support those without designate, as well as the changes to create cluster needed for openstack.

https://github.com/kubernetes/kops/pull/6345

[zetaab]

in my opinion this issue is solved. The alpha openstack support is added and it works. From now on we should start doing invidual tickets about issues and feature requests.

[dims]

Agreed. let's close this out and switch to new issues/prs

[dims]

/close

[k8s-ci-robot]

@dims: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.