Kops: Can it work without Route53 support on aws?

[yancl]

We have not Route53 function on aws, so can we use the kops? or how can we work around it
to build a production k8s cluster? thanks

[chrislovecnm]

DNS / Route53 is required. It is a base feature of aws. http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/Welcome.html

[yancl]

@chrislovecnm thanks for your reply, but the aws does not have Route53 in china region now:(.
So will you please give me some advise on how to deploy k8s for production?

I found the kube-up.sh is deprecated for production use :(

[chrislovecnm]

Can u run it out of a region it is in?

[chrislovecnm]

also tell aws to get there act together and get route53 up... Not joking

[yancl]

Thanks for your great advise, chrislovecnm, i will contact aws to make sure they can support that:)

[yancl]

@chrislovecnm according to the reply from aws, china region has its own IAM systems that is isolated from aws global. so there can't be an aws account that can simultaneously operate instances in china region and Route53 service in global. so we can't make it work in this situation.

So can i change the DNS record in my public/private DNS, and change the kops code to ignore the *_change DNS *_ operation?

[chrislovecnm]

@yancl part of the challenge is that DNS is used for internal system discovery. We would need another DNS server in place and configure the cluster to use it. Not impossible, but not trivial.

[yancl]

@chrislovecnm thank you for your detail explanation, it really helps me a lot. According to the feedbacks from aws, there will not be Route53 in our region. but there are a lot of k8s users and aws users, most of important, the kops is so great for aws k8s management.

So should i issue a feature for kops that support not only Rout53 for master-ip&etc discovery?
for example, specify the master or etc ip as args to bypass the problem or use a self-host DNS that support RESTful api like this for kops to change the name-ip mapping?

[chrislovecnm]

@yancl on another issue @justinsb mentioned that in non-ha-mode this should work. We have a PR in that will probably get around this problem. I do not recommend non-ha in production, and we probably need to look at other options. Can you recommend another DNS server that has an API that we could hook into?

[yancl]

@chrislovecnm thank you very much for this, can't agree with you more about that we should not use non-ha in production. Sorry, I don't know for now there is a DNS server that has programmable API and also HA like Route53. If i know later, i will ping you about that:)

[chrislovecnm]

@yanci do you code in go?

[yancl]

@chrislovecnm yes, i love it:)

[chrislovecnm]

@yancl so my question is how do we address this :) Ideas? Do we need internal dns pod? Do we need to not do DNS initially? @kris-nova and I have chatted and we are wondering how to do this best within the constraints that we have.

[yancl]

@chrislovecnm maybe we can use ENI with fixed private ip address that we specify? if so, we can start the cluster as following:

• create an ASG for some number(three/five) of etcds with each instance has an ENI and fixed ip
• create an ASG for one k8s master which also has an ENI and fixed ip, the master can find the etcds with step one's fixed ips
• create an ASG for minion nodes with master ip from step two's fixed ip

And when the instance terminated in ASG-etcd or ASG-master, the pre-allocated ENI&ip can be reassigned to the instance auto-launched later by ASG.

In fact, I have been confused several days by how to find the master&etcd's ip without introducing another component(DNS for example) that does not belong to the k8s to simplify the system，and make it robust with ASG meanwhile to reduce manual operation when instance fails.

This is by far what i think about it, expecting for your feedbacks:)

[chrislovecnm]

@kris-nova can u jump in here?

[yancl]

@chrislovecnm @kris-nova excuse me, does the ENI way described above make sense?

[kris-nova]

Hey all!

So @yancl this seems like a great start to a solution to the DNS problem in China...

How would this work with k8s and kops?

What are the concerns with using a static IP instead of a DNS record? Would this be as simple as a drag/drop - or do we need some other work put in place here to get this going?

What about DHCP and IP Conflict?

createNetworkInterface.PrivateIpAddress

The primary private IP address of the network interface. If you don't specify an IP address, Amazon EC2 selects one for you from the subnet range. link

So this makes me wonder how this will work with DHCP.. I am worried without a concept of a reservation for these static IPs are subject to IP pollution.. so my outstanding question is Do we have a concern if we are using AWS DHCP and ENIs with static addresses? Basically - do we need to call out any extra work here? I am not an expert in AWS DHCP - and don't know how it would handle this case.

Do we have concerns with running etcd in their own IG?

We already have called out some concerns here.. See https://github.com/kubernetes/kops/issues/772 for more information.. but I think this would be another concern...

In theory - if we can hammer out the details with these 3 concerns this should work... The scope of this in fairly large, but not impossible.. This would be a huge win for kops - and all of our friends in China!

[yancl]

Thanks @kris-nova , sorry for replying so late.

Yes, all the 3 concerns you showed are related to the ENI way. I don't think ENI is that good as DNS, its private ip just keep unchanged when the node in master ASG changed(down/up). But as understanding kops code more, I found it need a lot of works to integrated into the existing code or architecture. so maybe it is not a good idea in this situation:(

Currently I start a cluster using kops in our region with protokube&dns-controller not enabled on master, and make apiserver pointing to a pre-build etcd servers(whose ip is known in advance), and change the nodeup.sh(aws cloud init userdata) to auto attach an ENI so that minion nodes know that in advance.

The process above make it robust in one AZ, so i also add another cluster in another AZ to make it HA, then the upper balancer(our DNS provider) route traffic to these two cluster :)

BTW, event we have Route53, we only have two zones in one region, that does not satisfy the kops's requirment for HA(at least 3 AZ in one region).

Thanks again for your great help, hoping we can use kops totally without that much hack :)

[MilanDasek]

Hello,

would it be possible to use https://api.cloudflare.com/ ?

I would like to use kops in AWS China (no Route53) along with Cloudflare.

Thanks

Milan

[Globegitter]

We are just looking to move over to AWS and realising that Route53 does not offer DNSSEC and making this a potential blocker for us. For us only a HA cluster is an option. I have also seen there is https://github.com/kubernetes/kops/issues/172 - do the steps described there (setting up an LB) only work for clusters in one AZ or also for HA clusters? Otherwise coudflare is looking like a great options.

[Globegitter]

Just seeing https://github.com/kubernetes/kops/pull/2649/ - this would be a way to resolve this issue, right?

[chrislovecnm]

No #2649 is actually not related. We have an earily alpha feature where we use gossip for ha lookups, but it is in alpha. It needs more testing to be production grade.

[wangycc]

Hello,
Is this problem solved now? I found that still rely on route53, but my region does not provide this service(no Route53)

[chrislovecnm]

We have a very earily feature for gossip. @justinsb can you provide directions for gossip setup?

[macropin]

I believe that Gossip is now supported so this can be closed. Just name your cluster ending with .k8s.local https://github.com/kubernetes/kops/blob/master/docs/aws.md#configure-dns

[qqshfox]

Agree with @macropin. Successfully deployed multiple clusters by using kops 1.7 out-of-the-box in AWS China which doesn't have Route53 at all.

[yancl]

@chrislovecnm @kris-nova @justinsb thank you all for this, it works well :)

[yancl]

Currently we found that gossip based DNS with calico network may have a chance to cause the whole k8s cluster failed, for anyone who may cares, so post it here:)

https://github.com/kubernetes/kops/issues/3808