Kops: Documentation doesn't clearly explain what "Utility Subnets" are for

[mattparkes]

Edit: This ticket was originally called "kops create cluster --subnets won't use public and private subnets from the same zone" however I've worked out the cause of my misunderstanding.

If you are using existing/shared subnets and put private subnet IDs in --subnets and public subnet IDs in --utility-subnets then everything works as expected (nodes get created in private subnets and ELBs get created in public subnets).

I think potentially "utility-subnets" could be renamed to public-subnets to clarify how it is meant to be used.

[mattparkes]

/close

[finkbit]

This terminology also confused me.

[cjbottaro]

This terminology also confused me.

Same here. I definitely think it should be called public-subnets and explain it's not only for the masters load balancer, but also for service load balancers.

But further, with the advent of fully configurable Ingress Controllers, what's the point of inline service load balancers?

I think this option should just be: --what-subnets-do-you-want-your-api-load-balancer-to-be-in.

[ObviousDWest]

I'm not sure I agree with calling it --public-subnets if it, indeed is for service LBs and master LBs. Because what if I've created my cluster using --topology=private? In that case, I would want my master LBs on a private subnet (that I reach via VPN or bastion). And then I would like to have my service LBs on either a private or a public subnet based on my k8s configs. Some are internal services, some are external that I would like routed into my cluster. But now I'm probably asking for a new feature: allow (some) service LBs to be on a different subnet than master LBs.

[dee-kryvenko]

Found this from here https://github.com/kubernetes/kops/issues/5835. --utility-subnets definitely needs better docs.

@ObviousDWest, I'm not sure you are correct. My understanding --topology is not what used to pick up a subnet for the api LB, but --api-loadbalancer-type. Wherever LB is internal or external is controlled by service spec using service.beta.kubernetes.io/aws-load-balancer-internal annotation. That said, unless there are some other stuff utility subnets are used for that I'm not aware, it sounds reasonable to call them --public-subnets.

[KristianWindsor]

So what are utility subnets used for?

If I want my entire cluster to be private, using existing private subnets, the utility subnets should also be private, correct?

[cjbottaro]

The utility subnets are public and they are for things like the a load balancer in front of the kube-apiserver pods or a bastion to ssh onto nodes. I think a typical kops setup is in AWS where you are accessing the apiserver from your laptop, as well as ssh'ing from the your laptop (which is presumably on a different network from your Kubernetes nodes).

[drewwells]

I have no idea what public utility subnets are used for private or public. I would not recommend public at all, use an ELB for your bastion and apiserver instead with strict security groups. I also seemed to run into an error where there must be one utility subnet per AZ in the cluser (unconfirmed).

From inspecting my setup with private utility subnets and topology private. The utility subnet is used for the creation of the bastion. This is the only instance I see assigned to this subnet. Everything else in this subnet is ELBs, made by kops or by service: LoadBalancer. This means there are security groups in place to allow traffic from the ENIs in these subnets to the nodes in my instance group(s). The ELBs are not private nor are they limited to CIDR of utility subnets, this represents the upstream side of the ELB. You choose the public/private via your k8s object or the kops configuration. Hope this helps somebody, it would be great to get an authoritative answer on the subject.

[tuapuikia]

Is it possible to create instance group in utility subnet? I have service that utilizing UDP and the only option we have now is nodeport + EIP + udp.

[dee-kryvenko]

You should be able to use nginx ingress for UDP, since nginx now have UDP support.
From the top of the google:
https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/
https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/tcp-udp

[tuapuikia]

I solved it by creating new ig with utility subnet and associate public ip to it. I can run nginx ingress on new external IG and using udp with it.

The only thing left missing is, when i define service using nodeport and udp port. The node security group do not get updated. I have to add the rule manually.

Also external-dns do not support nodeport with node external ip.