Aws-load-balancer-controller: AWS ALB Ingress Controller doesn't update Security Groups

Created on 22 Apr 2020 · 8Comments · Source: kubernetes-sigs/aws-load-balancer-controller

Hi,
we have an EKS cluster (1.14). There are spot-instances and ondemand-instances, which were created via eksctl. We use the AWS ALB Ingress Controller also.

We had following issue yesterday.
It was caused due to missing security policies, so that the ALB could not access the pods, which are running on the worker nodes.
Weeks ago I had updated the Spot-Instances-NodeGroups via eksctl, so new NG were created and the old deleted. The On-Demand-NG wasn't touched. Since the pods have been running on the On-Demand-workers and all was fine.
Yesterday we had an deployment and the pods were moved to the Spot-Instances. The associated Security Groups weren't updated, so the ALB couldn't connect to the targets.

I think the AWS ALB Igress Controller should update the SG: https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/ingress/annotation/

We do not use the security group annotation.

Could you please check this issue?

Source

albertschwarzkopf

Most helpful comment

Hi @M00nF1sh ! We seem to be running into the same issue here, and updating to alpha.2 did not help.

We are running the aws-alb-ingress-controller on EKS 1.14. Creating this ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  namespace: alb-test-ch-35640
  name: nginx
  annotations:
    alb.ingress.kubernetes.io/listen-ports: |
      [{"HTTP": 80}]
    alb.ingress.kubernetes.io/scheme: internet-facing
    kubernetes.io/ingress.class: alb
spec:
  backend:
    serviceName: nginx
    servicePort: 80

Gives us an ALB with a correct Target Group, and assigns a new Security Group to that ALB, but it never updates the Nodes' security group (or create a new one on the ENIs that host these pods).

Here are the logs for the creation (AWS account id redacted):

{"level":"info","ts":1592314118.6369686,"logger":"alb-test-ch-35640/nginx","msg":"start reconcile","groupID":"alb-test-ch-35640/nginx","activeMembers":["alb-test-ch-35640/nginx"],"leavingMembers":[]}
{"level":"info","ts":1592314119.0915253,"logger":"alb-test-ch-35640/nginx","msg":"successfully built model","groupID":"alb-test-ch-35640/nginx","model":"{\"ID\":\"alb-test-ch-35640/nginx\",\"ManagedLBSecurityGroup\":{\"metadata\":{\"name\":\"ManagedLBSecurityGroup\",\"creationTimestamp\":null},\"spec\":{\"securityGroupName\":\"k8s-alb-test-ch-35640-392f61f0a2\",\"description\":\"[k8s] Managed SecurityGroup for LoadBalancer\",\"permissions\":[{\"fromPort\":80,\"toPort\":80,\"ipProtocol\":\"tcp\",\"cidrIP\":\"0.0.0.0/0\"}]},\"status\":{}},\"InstanceSecurityGroups\":[],\"LoadBalancer\":{\"metadata\":{\"name\":\"LoadBalancer\",\"creationTimestamp\":null},\"spec\":{\"loadBalancerName\":\"k8s-alb-test-ch-35640-9aaca6b191\",\"ipAddressType\":\"ipv4\",\"schema\":\"internet-facing\",\"subnetMappings\":[{\"subnetID\":\"subnet-0d15eaa832c22d54e\"},{\"subnetID\":\"subnet-046b18b5c6f8fbf17\"},{\"subnetID\":\"subnet-0b1843a1d2ad5abf3\"}],\"securityGroups\":[{\"securityGroupRef\":{\"name\":\"ManagedLBSecurityGroup\"}}],\"attributes\":{\"deletionProtection\":{\"enabled\":false},\"accessLogs\":{\"s3\":{}},\"idleTimeout\":{\"timeoutSeconds\":60},\"routing\":{\"http2\":{\"enabled\":true}}},\"listeners\":[{\"port\":80,\"protocol\":\"HTTP\",\"defaultActions\":[{\"type\":\"forward\",\"forwardConfig\":{\"targetGroup\":{\"targetGroupRef\":{\"name\":\"alb-test-ch-35640/nginx-nginx:80\"}}}}]}]},\"status\":{}},\"TargetGroups\":{\"alb-test-ch-35640/nginx-nginx:80\":{\"metadata\":{\"name\":\"alb-test-ch-35640/nginx-nginx:80\",\"creationTimestamp\":null},\"spec\":{\"targetGroupName\":\"k8s-alb-test-nginx-1a468fc291\",\"targetType\":\"instance\",\"port\":1,\"protocol\":\"HTTP\",\"healthCheckConfig\":{\"intervalSeconds\":15,\"path\":\"/\",\"port\":\"traffic-port\",\"protocol\":\"HTTP\",\"timeoutSeconds\":5,\"healthyThresholdCount\":2,\"unhealthyThresholdCount\":2,\"matcher\":{\"intervalSeconds\":\"200\"}},\"attributes\":{\"deregistrationDelay\":{\"timeoutSeconds\":300},\"slowStart\":{\"durationSeconds\":0},\"stickiness\":{\"enabled\":false,\"type\":\"lb_cookie\",\"lbCookie\":{\"durationSeconds\":86400}}}},\"status\":{}}},\"EndpointBindings\":{\"alb-test-ch-35640/nginx:alb-test-ch-35640/nginx-nginx:80\":{\"metadata\":{\"name\":\"alb-test-ch-35640/nginx:alb-test-ch-35640/nginx-nginx:80\",\"creationTimestamp\":null},\"spec\":{\"targetGroup\":{\"targetGroupRef\":{\"name\":\"alb-test-ch-35640/nginx-nginx:80\"}},\"targetType\":\"instance\",\"serviceRef\":{\"namespace\":\"alb-test-ch-35640\",\"name\":\"nginx\"},\"servicePort\":80},\"status\":{}}}}"}
{"level":"info","ts":1592314119.1173694,"logger":"alb-test-ch-35640/nginx","msg":"creating targetGroup","resource":"alb-test-ch-35640/nginx-nginx:80"}
{"level":"info","ts":1592314119.273125,"logger":"alb-test-ch-35640/nginx","msg":"created targetGroup","resource":"alb-test-ch-35640/nginx-nginx:80","arn":"arn:aws:elasticloadbalancing:eu-west-1:111111111111:targetgroup/k8s-alb-test-nginx-1a468fc291/ff8abc23cda7f30c"}
{"level":"info","ts":1592314119.2731774,"logger":"alb-test-ch-35640/nginx","msg":"modifying tags","arn":"arn:aws:elasticloadbalancing:eu-west-1:111111111111:targetgroup/k8s-alb-test-nginx-1a468fc291/ff8abc23cda7f30c","changes":"{\n  ingress.k8s.aws/resource: \"alb-test-ch-35640/nginx-nginx:80\",\n  ingress.k8s.aws/cluster: \"growth-staging-eks\",\n  ingress.k8s.aws/stack: \"alb-test-ch-35640/nginx\"\n}"}
{"level":"info","ts":1592314119.322347,"logger":"alb-test-ch-35640/nginx","msg":"modified tags","arn":"arn:aws:elasticloadbalancing:eu-west-1:111111111111:targetgroup/k8s-alb-test-nginx-1a468fc291/ff8abc23cda7f30c"}
{"level":"info","ts":1592314119.3378253,"msg":"adding targets","tgArn":"arn:aws:elasticloadbalancing:eu-west-1:111111111111:targetgroup/k8s-alb-test-nginx-1a468fc291/ff8abc23cda7f30c","changes":"i-06ca89092d0541a40:32753, i-05d4cc198241f557f:32753, i-01763c4c890e112cd:32753, i-03eae8cbc41a70f19:32753, i-0f9e2c8b086f6bd77:32753, i-0de95b833fea0a86d:32753, i-0843fc38d8da09ba7:32753, i-0656d9f5da5e581a5:32753, i-05e2fe6ab77c7f8e1:32753, i-031b186eb505e01f8:32753"}
{"level":"info","ts":1592314119.3424978,"logger":"alb-test-ch-35640/nginx","msg":"creating SecurityGroup"}
{"level":"info","ts":1592314119.4134605,"logger":"alb-test-ch-35640/nginx","msg":"created SecurityGroup","sgID":"sg-0563aec90e4bf64c0"}
{"level":"info","ts":1592314119.4135573,"logger":"alb-test-ch-35640/nginx","msg":"modifying tags","ID":"sg-0563aec90e4bf64c0","changes":"{\n  ingress.k8s.aws/resource: \"ManagedLBSecurityGroup\",\n  ingress.k8s.aws/cluster: \"growth-staging-eks\",\n  ingress.k8s.aws/stack: \"alb-test-ch-35640/nginx\"\n}"}
{"level":"info","ts":1592314119.5342352,"logger":"alb-test-ch-35640/nginx","msg":"modified tags","ID":"sg-0563aec90e4bf64c0"}
{"level":"info","ts":1592314119.534299,"logger":"alb-test-ch-35640/nginx","msg":"granting inbound permission","sgID":"sg-0563aec90e4bf64c0","permissions":"[{\n    FromPort: 80,\n    IpProtocol: \"tcp\",\n    IpRanges: [{\n        CidrIp: \"0.0.0.0/0\",\n        Description: \"\"\n      }],\n    ToPort: 80\n  }]"}
{"level":"info","ts":1592314119.5820162,"msg":"added targets","tgArn":"arn:aws:elasticloadbalancing:eu-west-1:111111111111:targetgroup/k8s-alb-test-nginx-1a468fc291/ff8abc23cda7f30c"}
{"level":"info","ts":1592314119.7199116,"logger":"alb-test-ch-35640/nginx","msg":"granted inbound permission","sgID":"sg-0563aec90e4bf64c0"}
{"level":"info","ts":1592314119.94296,"logger":"alb-test-ch-35640/nginx","msg":"creating LoadBalancer","resource":"LoadBalancer"}
{"level":"info","ts":1592314120.5016015,"logger":"alb-test-ch-35640/nginx","msg":"created LoadBalancer","resource":"LoadBalancer","ARN":"arn:aws:elasticloadbalancing:eu-west-1:111111111111:loadbalancer/app/k8s-alb-test-ch-35640-9aaca6b191/1c4476907339db76"}
{"level":"info","ts":1592314120.5198634,"logger":"alb-test-ch-35640/nginx","msg":"creating listener","lbArn":"arn:aws:elasticloadbalancing:eu-west-1:111111111111:loadbalancer/app/k8s-alb-test-ch-35640-9aaca6b191/1c4476907339db76","port":80}
{"level":"info","ts":1592314120.5471013,"logger":"alb-test-ch-35640/nginx","msg":"created listener","lbArn":"arn:aws:elasticloadbalancing:eu-west-1:111111111111:loadbalancer/app/k8s-alb-test-ch-35640-9aaca6b191/1c4476907339db76","port":80,"lsArn":"arn:aws:elasticloadbalancing:eu-west-1:111111111111:listener/app/k8s-alb-test-ch-35640-9aaca6b191/1c4476907339db76/18267c004a3d7f42"}
{"level":"info","ts":1592314120.5624223,"logger":"alb-test-ch-35640/nginx","msg":"successfully deployed model","groupID":"alb-test-ch-35640/nginx"}

I can't see any errors, just the fact that the controller never attempts to update the Nodes' security group.

I tried with both target-type: ip and target-type: machine with the same results (the target groups are different obviously, but same result re security group).

Am I correct in the assumption that this is a bug? Or is there something I'm missing for the controller to allow traffic through properly?

Thank you!

florent-tails on 16 Jun 2020

👍3

All 8 comments

@albertschwarzkopf can you provide the following details

Controller Version
Which traffic mode you are using with ALB Ingress Controller
Relevant logs from ALB Ingress Controller
What steps did you take to update the spot instance nodegroup?
Was the alb routing traffic to the pods before the upgrade?
Cluster ARN (you can send over email [email protected])

abhipth on 22 Apr 2020

@albertschwarzkopf can you provide the following details

Controller Version

Which traffic mode you are using with ALB Ingress Controller

Relevant logs from ALB Ingress Controller

What steps did you take to update the spot instance nodegroup?

Was the alb routing traffic to the pods before the upgrade?

Cluster ARN (you can send over email [email protected])

@abhipth
We use the Chart in version 0.1.14 and image m00nf1sh/aws-alb-ingress-controller:v1.2.0-alpha.1.
The annotation "alb.ingress.kubernetes.io/target-type: ip" for ingress is set, so we use the IP mode.
There aren't any unusual logs.
Weeks ago, I had changed the instance types for the NG with spot instances, so I edited the cluster-config file, changed the instance types and set a new NG-name like "spot-group-a-v2" and run the command "eksctl create nodegroup --config-file=cluster-config/prod.yaml"
After the NG were ready, I deleted the old NG: "eksctl delete nodegroup --cluster=prod-* --name=spot-group-a-v1.
The NG with OnDemand-instances wasn't touched.
I think before the application deployment, the pods were running on the OnDemand-instances and all was fine, because the SG-settings are correct here.

albertschwarzkopf on 23 Apr 2020

@albertschwarzkopf can you provide the following details

Controller Version

Which traffic mode you are using with ALB Ingress Controller

Relevant logs from ALB Ingress Controller

What steps did you take to update the spot instance nodegroup?

Was the alb routing traffic to the pods before the upgrade?

Cluster ARN (you can send over email [email protected])

@abhipth
We use the Chart in version 0.1.14 and image m00nf1sh/aws-alb-ingress-controller:v1.2.0-alpha.1.
The annotation "alb.ingress.kubernetes.io/target-type: ip" for ingress is set, so we use the IP mode.
There aren't any unusual logs.
Weeks ago, I had changed the instance types for the NG with spot instances, so I edited the cluster-config file, changed the instance types and set a new NG-name like "spot-group-a-v2" and run the command "eksctl create nodegroup --config-file=cluster-config/prod.yaml"
After the NG were ready, I deleted the old NG: "eksctl delete nodegroup --cluster=prod-* --name=spot-group-a-v1.
The NG with OnDemand-instances wasn't touched.
I think before the application deployment, the pods were running on the OnDemand-instances and all was fine, because the SG-settings are correct here.

Sorry it is another chart version: 0.1.8. The same image.

albertschwarzkopf on 24 Apr 2020

@albertschwarzkopf I tried to reproduce the issue as close as possible. I created an ingress resource with IP mode.

Create two node groups - on-demand instance and spot instance.
Created a sample app with multiple replicas running on the the on demand instance.
Verified no inbound rules initially exists on the spot instance as the pods are running on the on demand instance.
Verified the pods were reachable running on on demand instance.
Changed the deployment to run the replicas on spot instance node group. As you can see the security group was updated.

I could also verify from the controller log that the ingress rule was added.

I0424 20:49:47.883357 1 instance_attachment_v2.go:192] 2048-game/2048-ingress: granting inbound permissions to securityGroup sg-01e220006804b2711: [{ FromPort: 0, IpProtocol: "tcp", ToPort: 65535, UserIdGroupPairs: [{ GroupId: "sg-00f701bf40d7a9ceb" }] }]

Is there a reason why you are using image "m00nf1sh/aws-alb-ingress-controller:v1.2.0-alpha.1."? Maybe try updating the controller to the latest version v1.1.4 which I used for my testing?

abhipth on 24 Apr 2020

We use the 1.2 alpha because it supports the group feature which is very important for us. Furthermore even @M00nF1sh stated that the 1.2 has several other advantages which are not backported yet. It would be great to get an official 1.2 release soon.

runningman84 on 25 Apr 2020

@abhipth My colleague runningman84 has answered your question.
We have several systems. On some of them we use an older chart version (0.1.8) and another systems use chart version 0.1.14. But the image is the same.
The problem has occurred on a system with chart version 0.1.8.
I don't think there should be any impact, but I want to mention it.

albertschwarzkopf on 27 Apr 2020

hi.
I have pushed the fix to 59708ee, and built a image contains the fix: m00nf1sh/aws-alb-ingress-controller:v1.2.0-alpha.2
you can build your own by check v2 branch at your $GOPATH/src/sigs.k8s.io/ and run "dep ensure && make docker-build"

M00nF1sh on 3 Jun 2020