Aws-load-balancer-controller: [v2] Using existing ALB with v2 RC

[JanJakes]

Browsing through comments under some of the issues I figured out it should be possible to use an existing ALB (i.e. defined with Terraform). I made it work somehow, but there are some caveats and I'm not sure what's the best way to do it.

Using v2.0.0-rc1

I get the following error:

{
  "error": "open /tmp/k8s-webhook-server/serving-certs/tls.crt: no such file or directory",
  "level": "error",
  "logger": "setup",
  "msg": "problem running manager",
  "stacktrace": "github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\nmain.main\n\t/workspace/main.go:147\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204",
  "ts": 1601291927.422289
}

It seems v2.0.0-rc1 requires all of the cerificate-handling logic that I don't need when I define certificates using Terraform.

Using v2.0.0-rc0

This attempt was a bit more successful.

1. I added the following tags to the existing ALB:

ingress.k8s.aws/resource: LoadBalancer
ingress.k8s.aws/cluster:  mycluster
ingress.k8s.aws/stack:    mygroup

1. I added alb.ingress.kubernetes.io/group.name: mygroup label to the ingress definition.

2. Tried it, got an error:

{
  "controller": "ingress",
  "error": "cannot found at least two subnet from different Availability Zones, discovered subnetIDs: [subnet-aaa subnet-bbb]",
  "errorVerbose": "cannot found at least two subnet from different Availability Zones, discovered subnetIDs: [subnet-aaa subnet-bbb]\nsigs.k8s.io/aws-alb-ingress-controller/pkg/ingress.(*defaultModelBuildTask).buildLoadBalancerSubnetMappings\n\t/workspace/pkg/ingress/model_build_load_balancer.go:154\nsigs.k8s.io/aws-alb-ingress-controller/pkg/ingress.(*defaultModelBuildTask).buildLoadBalancerSpec\n\t/workspace/pkg/ingress/model_build_load_balancer.go:44\nsigs.k8s.io/aws-alb-ingress-controller/pkg/ingress.(*defaultModelBuildTask).buildLoadBalancer\n\t/workspace/pkg/ingress/model_build_load_balancer.go:26\nsigs.k8s.io/aws-alb-ingress-controller/pkg/ingress.(*defaultModelBuildTask).run\n\t/workspace/pkg/ingress/model_builder.go:162\nsigs.k8s.io/aws-alb-ingress-controller/pkg/ingress.(*defaultModelBuilder).Build\n\t/workspace/pkg/ingress/model_builder.go:95\nsigs.k8s.io/aws-alb-ingress-controller/controllers/ingress.(*GroupReconciler).Reconcile\n\t/workspace/controllers/ingress/group_controller.go:80\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:233\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357",
  "level": "error",
  "logger": "controller-runtime.controller",
  "msg": "Reconciler error",
  "name": "alb-main-group",
  "namespace": "",
  "stacktrace": "github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90",
  "ts": 1601291221.5320227
}

The error message is a bit confusing to me (cannot found at least two subnet from different Availability Zones, discovered subnetIDs: [subnet-aaa subnet-bbb]) – the subnet-aaa and subnet-bbb are the correct public subnets tagged with kubernetes.io/cluster/mycluster: shared and kubernetes.io/role/elb : 1 and attached to the ALB.

1. Adding label alb.ingress.kubernetes.io/subnets: subnet-aaa,subnet-bbb to the ingress definition fixes the issue.

2. I noticed the controller removes any other tags that I had on the ALB.

3. It seems that having deletion protection on the ALB solves the problem of accidental deletion when removing ingresses. But there's a catch as well – the ingress gets never deleted, in fact.

---

I guess this is a summary of what I've been able to achieve with the RCs so far.

With RC0 it seems to work quite well, except for the subnet auto-discovery, full tag override, and non-deletable ingress.

With RC1 I was not able to make it work without having dummy unused certificates (didn't try this option).

So my question is – what would be the optimal way to use an existing ALB? I noticed fully-managed ALBs get a kubernetes.io/cluster/mycluster: owned, maybe using the same tag on my Terraform-created ALB with another value (i.e. shared, adopted, etc.) could tell the controller not to remove my tags and never try to delete it?

[kishorj]

Thanks for trying it out and the detailed feedback.

RC0 error was due to a bug, it has been addressed in RC1.

RC1 requires cert-manager to inject the webhook secrets/certificates during initialization. Based on the error message, it looks like the cert-manager is either not configured, or not working as expected. You can verify if the secret gets created by running

`kubectl get secret -n kube-system webhook-server-secret

Also verify the webhooks have the caConfig injected

kubectl get mutatingwebhookconfigurations aws-load-balancer-webhook  -oyaml
kubectl get validatingwebhookconfigurations aws-load-balancer-webhook  -oyaml

If cert manager is configured and working, you should see the caConfig like the following -

webhooks:
- admissionReviewVersions:
  - v1beta1
  clientConfig:
    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1J...

The helm-chart will support installing without the cert-manager.

[JanJakes]

@kishorj Thanks for your reply. What I was trying with RC1 was to use it _without_ the cert-manager but it seems that the Docker image itself simply requires a certificate at the moment.

One more thing I noticed when using Terraform is that ALB's security groups are managed by the ingress controller as well and the controller creates new security group for each alb.ingress.kubernetes.io/group.name. My EKS node group is defined in Terraform and I don't have access to these security groups from there, which results in the nodes having a different security group than ALB. I resolved that by allowing the nodes being accessed from all of the EKS subnet CIDR blocks (not just within the same security group):

resource "aws_security_group_rule" "eks-nodes-vpc-ingress" {
  description       = "Allow access from all EKS subnets (for Terraform-managed ALB)"
  security_group_id = aws_eks_cluster.eks.vpc_config[0].cluster_security_group_id
  from_port         = 0
  to_port           = 0
  protocol          = "-1"
  type              = "ingress"
  cidr_blocks       = values(data.aws_subnet.eks-subnets)[*].cidr_block
}

data "aws_subnet" "eks-subnets" {
  for_each = aws_eks_cluster.eks.vpc_config[0].subnet_ids
  id       = each.value
}

With that it seems to be a quite complete and working usage of an existing ALB. Still on RC0 though (so I don't need to add the cert-manager).

[MartinEmrich]

I have the same issue. I want to try out the new NLB-ip-mode feature, but have (for now) no need for the webhook-readinessGate-features. So I removed all resource regading this (Webhook, Certififcate manager, etc.)

I also tried to set the option --enable-pod-readiness-gate-inject=false, but no success.

The helm-chart will support installing without the cert-manager.

Where can I find the helm chart? I do not use helm, but maybe I find some inspiration there to get it to work.

[kishorj]

@MartinEmrich
Once released, helm chart will be available from the repo is https://aws.github.io/eks-charts
The source code is in https://github.com/aws/eks-charts/tree/master/stable/aws-load-balancer-controller

For NLB-IP mode, you'd need k8s with this fix - https://github.com/kubernetes/kubernetes/pull/92839
User guide is at https://github.com/kubernetes-sigs/aws-alb-ingress-controller/blob/v2_ga/docs/guide/nlb-ip.md
If your k8s service controller does not have the fix, you need to create a service of type NodePort, not LoadBalancer

[MartinEmrich]

@kishorj Thanks. I took a look at the helm chart files, but could not find any "if..then.else"-style code to disable the webhook part.
So I generated a bogus SSL cert with openssl and made the secret myself, this made the ingress controller start fine.

It would be great if the final 2.0 would honor the command line option to disable this functionality.

Nevertheless, the NLB-IP-mode service works great! (Using stock Amazon EKS 1.17, no fixes, service is NodePort). My pods are now registered at the target group in IP mode. I hope to avoid this issue https://github.com/kubernetes/autoscaler/issues/1907 with this method as soon as the stable 2.0 is released.

[M00nF1sh]

@MartinEmrich
The server-cert will always be need. e.g. in v2.0.0-rc2, we added mutating/validating webhooks for the TargetGroupBindingCRD.
In future version post final v2.0.0, we might add webhooks for Ingress/Service to validate the annotations before it's been created.

When install via helm chart, by default we don't need cert-manager and generates certificates and store as secret when the chart is installed. We also support cert-manager if user prefer to use it

[M00nF1sh]

@JanJakes
Hi Jan,
we don't support to use an existing LoadBalancer in our v2.0.0(the tags is just a hack to make it work). we'll add support for it post v2.0.0 release after we have clear definition on the behavior of "use existing LoadBalancer".
To help us deliver such feature, would you help provide your desired use case/behavior to use an existing LoadBalancer, e.g.

1. How would you expect existing settings/listeners on that LoadBalancer to be managed, should the controller remove unnecessary listeners on the LoadBalancer?
   
   
   
   • Currently Listeners don't support tagging(it's coming soon though), so the controller cannot differentiate listener created by controller vs listener created by external identity by check the AWS state. (we can do save some metadata via annotation to Ingresses, but it would be complicated and can be easily overwritten by user).
   
   
2. How would you expected the UX be to use existing LoadBalancer?
   
   
   
   • Alternative 1(personally I prefer this), explicitly add an annotation to Ingress like alb.ingress.kubernetes.io/load-balancer-arn: xxxx. It's more explicit than the Tagged approach, as I prefer to keep the tags as implementation details.
   
   
   • Alternative 2, tag the resources as described, with an additional tag like ingress.k8s.aws/ownership: owned|shared.
   
   
3. Did the new TargetGroupBinding CRD solves your use case?
   
   
   
   • You create the LoadBalancer/Listener/Rule/TargetGroups via terraform/cdk/cloudformation.
   
   
   • You create a TargetGroupBinding resource for each of your TargetGroup.
   
   

[JanJakes]

Hi @M00nF1sh,

thanks for the details. As for your questions:

1. Good point. I have no other listeners at the moment but it seems to make sense to allow for user-defined listeners as well, without removing them (i.e. some fixed-response actions, redirects, etc.). If tagging is coming soon, that could solve it. Otherwise it may also be possible to store that as tags on the LB? Not sure how to order them though (controller-managed listeners vs. user-managed ones).

2. Yes, that sounds great. It's the most explicit (and "correct", I'd say) since you're clearly stating "the ALB exists already and it has this ARN".

3. I didn't give this a try yet, didn't find much info. Does it work with alb.ingress.kubernetes.io/target-type both ip and instance? This would essentially mean keeping all of the ingress rules in Terraform and only saying "this target group references this service" on the Kubernetes side? If so, it sounds like a pretty solid option as well, at least for my use case where listeners and rules shouldn't change very often.

---

One more question on this:

The server-cert will always be need. e.g. in v2.0.0-rc2, we added mutating/validating webhooks for the TargetGroupBindingCRD.

What are these webhooks for, exactly? This new (now mandatory) certificate is somehow needed for the webhooks?

[M00nF1sh]

Hi @JanJakes
Yes, the TargetGroupBinding works for both IP targetType and instance TargetType.
we have three webhooks currently:

1. I think we'll only support the explicit own all settings of LoadBalancer model before ALB supports tagging on listener.
2. we'll add support for this after our initial release :D
3. pod mutationWebhook: It's used to auto-inject podReadinessGate into pods if a service is used with IP type targetGroup.
4. TargetGroupBinding mutationWebhook: it's used to auto-inject targetType if unspecified.
5. TargetGroupBinding validationWebhook: it's used to enforce some fields like targetGroup are immutable.

We'll offer a helm chart with generates a self-signed certificate upon install, so no cert-manager is needed :D

[MartinEmrich]

Hello @M00nF1sh! I'm also interested in using an existing ALB just like @JanJakes, so I'll cheekily answer your questions, too ;)

1. I would expect listeners to be identified by port: i.e. if there's not yet a :443 listener, the controller shall create it. If it exists, it shall be reused as-is.
2. I would prefer binding the ALB to the Ingress in one of these ways, keeping "code" (ingress rules) and "infrastructure" (ALB ARNs, SSL certificate ARNs) separate:
   a) Store all the AWS-specific info (ALB ARN, Certificate ARN, etc.) in another resource, and refer to it from the Ingress resource. This way, the Ingress can be shared between multiple installations (e.g. different regions, or a production and a development installation).
   b) If that's too tedious, I'd prefer going the AWS tag route: annotating the Ingress with a user-defined tag value, and writing the same value in a tag on the ALB
3. I did not yet check out the TargetGroupBinding concept at all.. I tried googling (and searching Github) for it, but I only see it mentioned in some issues here... Where could I find documentation?

Thanks
Martin

[M00nF1sh]

actually alb listeners/rules supports tags as well.

[dev-e]

@JanJakes , @MartinEmrich https://kubernetes-sigs.github.io/aws-load-balancer-controller/guide/targetgroupbinding/targetgroupbinding/

[MartinEmrich]

Yay, finally tried the 2.0 release today, thank you very much! 👏🏼
All of ingress groups, targetGroupBindings and NLB-IP-mode work great!

Now I am looking for something between Ingress(Groups) and TargetGroupBindings:

• An Ingress (or Group) "owns" all of ALB, Listener, Rules, TargetGroups and registered targets.
• A TargetGroupBinding "owns" only the TargetGroup registered targets; everything else is outside.

I am looking for something that "owns" at least the rules, and the target groups and registered targets,, but not the ALB (and possibly NOT the listener). Something like an Ingress "light" where I have to bring my own ALB.
Is this already somewhere in 2.0.0? Or is this what what you mean by "we'll add support for this after our initial release :D"?

Thanks

Martin

[M00nF1sh]

@MartinEmrich thanks, that's something we'll add :D

[alexandrud]

I'm also interested in a way to manage the ALBs with some other provisioning tool and manage just the targetGroups and targetGroupBindings with the controller.

We provision the LBs (group.name / ingress.k8s.aws/stack included ), webAcls and rulegroups with terraform because we don't control the CNAMEs pointing to the LBs and we don't want to lose them.