Aws-load-balancer-controller: Subnet Selection (auto-detection) - Suggestion for Change.

[whithajess]

I am definitely a fan of using auto-detection

Wondering if it could be more closely aligned to Kops given this is the suggested way to install Kuberenetes on AWS.

The tags given to subnets by default with kops are:

KubernetesCluster | $CLUSTER_NAME
kubernetes.io/cluster/$CLUSTER_NAME | owned

[blytheaw]

Kops seems to recommend naming clusters to match the domain name you set up for DNS records in order to ensure uniqueness between clusters. This could commonly be a pretty long cluster name; would this not break the auto-detection feature since the alb-ingress has a character limit in CLUSTER_NAME, which has to match the same tag on the subnets used by Kops?

I'm not sure if this is a separate issue to be discussed.

[whithajess]

@blytheaw That was fixed via https://github.com/coreos/alb-ingress-controller/issues/223.

Specifically https://github.com/coreos/alb-ingress-controller/commit/918d034df4f3994ca34f5d70d3de81f74d55e934

If you are still having problems I would suggest raising another issue.

[joshrosso]

^ 👍 Let us know!

[joshrosso]

Sorry didn't mean to close -- @whithajess, you're suggesting re-using those existing kops annotations, correct?

[whithajess]

@joshrosso Yeah basically it would just be nice if it found default tags created by kops, and by default would auto-select them without me having to either manually enter them in either my ingress setup or by tagging AWS resources.

[ese]

These tags are a kubernetes convention in aws beyond kops
https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/tags.go#L30

[whithajess]

Oh cool so then follow Kubernetes convention not just Kops :) thanks for the info @ese

[cyburgee]

I too would like to see auto detection of subnets work with a cluster tag value of "owned". I can open a PR for the change if it wouldn't cause folks problems

[jhohertz]

Being able to use "owned" lifecycle subnets in addition to "shared" would be nice for being able to use w/ kops out of the box. However this would also require being able to change the other tag.

This component expects: kubernetes.io/role/alb-ingress
Where kops (and kubernetes) work with these tags: https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws.go#L87-L93

[rifelpet]

Agreed on autodetecting a cluster tag value of owned. Kops doesn't currently support adding custom AWS tags to specific subnets. Maybe the alb ingress controller could take a set of subnet tag filters as a parameter somehow?

It could use the following order of preference:

1. subnets annotation on the ingress resource
2. filter parameter on the controller if set
3. Existing default of kubernetes.io/cluster/clustername=shared and kubernetes.io/role/alb-ingress

In the case of my Kops clusters I can use kubernetes.io/cluster/clustername=owned along with SubnetType=Private for internal ALBs and SubnetType=Utility for internet-facing ALBs

[BrianChristie]

Another voice agreeing; the correct thing to do here is use kubernetes.io/cluster/$CLUSTER_NAME=owned. This is consistent with the Kubernetes conventions, as well as Kops. https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/tags.go#L30

Does anyone have input on making this change prior to starting a PR?

[BrianChristie]

To follow up on the comment from @jhohertz, is there a compelling reason to use kubernetes.io/role/alb-ingress instead of the preexisting label kubernetes.io/role/internal-elb?

According to AWS documentation, an ALB is a subtype of ELB. Therefore, I believe we should continue using the internal-elb label rather than creating a new one.

Elastic Load Balancing supports three types of load balancers: Application Load Balancers, Network Load Balancers, and Classic Load Balancers

[argeualcantara]

Hi I've managed to create a method that discovers subnets based on the subnet tags
kubernetes.io/cluster/<cluster-name>=shared and kubernetes.io/role/alb-ingress='' also to sort them based on the scheme annotation: internal gets private subnets and internet-facing gets public subnets.

I'm gonna fix somethings up, but hopefully I'll be able to make a PR soon

[whithajess]

@argeualcantara kubernetes.io/cluster/<cluster-name>=owned is going to be the most common use-case

both options owned/shared should be necessary.

[argeualcantara]

@whithajess Ok, is there any difference (configuration, security groups, etc..) for the ALB ingress if the tag kubernetes.io/cluster/<cluster-name> has the shared or owned values?

[bigkraig]

I'm leaning towards using the same tags as the aws cloud provider, see https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws.go#L87-L93

[whithajess]

@bigkraig I think the point of ___auto-detection___ should be that it picks up defaults, if I have to set specific tags on my instances to do it then it defeats the purpose (as it exists in its current state I just set them manually rather then tagging) If it was implemented as suggested with elb tags I would still manually set it (very little difference to me tagging the instances manually and not really ___auto___) I think nearly every use case would be to set it across the cluster, and for the edge cases they can do it manually.

[bigkraig]

@whithajess I am not sure I understand, if I go with the tags defined in that link, you would need to have 2 tags on your subnets which appear to be pretty standard.

All subnets:
kubernetes.io/cluster/<cluster name> = shared

Subnets for internal ALBs:
kubernetes.io/role/internal-elb = ``

Subnets for external ALBs:
kubernetes.io/role/elb = ``

[whithajess]

@bigkraig By default if i spin up a cluster with I will get:

kubernetes.io/cluster/<cluster name> = owned

The other elb tags are something I would have to manually tag.

[bigkraig]

@whithajess got it, so you are suggesting only require that tag? how should we know if its an internal or external subnet?

[whithajess]

@bigkraig Yeah sorry you are right, I can't think of a way you could tell without setting that and I actually just checked my cluster and it does have kubernetes.io/role/elb = 1 not sure how I missed that the first time 👍 on your tag suggestions.

[bigkraig]

Have a look at #455

[whithajess]

Should be able to close this now @bigkraig? or want me to verify with a new release?

[bigkraig]

@whithajess I just cut 1.0-beta.4, it'll be building for a while but we can close this one 👍