Aws-load-balancer-controller: Kops / kubenet pod ips are not registered with AZ "all" to targetGroup anymore

[pesimon]

Hi,

I'm running a Kops-based cluster with Kubenet configuration where pods get IPs from nonMasqueradeCIDR, which is not listed in the VPC config. In this configuration Kops adds direct routes to the VPC routing table so that the k8s nodes can access pods and services on other nodes.

With alb-ingress-controller:1.0-beta.6 this worked well; with the ALB being deployed to the same subnet and target-type: ip, it could register pod endpoint to the ALB.

With the upgrade to v1.0.1 (same with v.1.0.0) this raises an error:

kubebuilder/controller "msg"="Reconciler error" "error"="failed to reconcile targetGroups due to failed to reconcile targetGroup targets due to ValidationError: The Availability Zone for IP address '100.96.7.56' must be 'all' for Application Load Balancer target groups, when not within the VPC\n\tstatus code: 400, request id: 83067527-fec9-11e8-88c7-15463d1956f9"  "Controller"="alb-ingress-controller" "Request"={"Namespace":"<snipped>","Name":"<snipped>"} 

Ingress config:

  annotations:
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:<arn snipped>
    alb.ingress.kubernetes.io/healthcheck-path: /healthz
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/subnets: subnet-XXXX,subnet-YYYYY
    alb.ingress.kubernetes.io/success-codes: "200"
    alb.ingress.kubernetes.io/target-type: ip
    kubernetes.io/ingress.class: alb

relevant KOPS settings:

  kubernetesVersion: 1.10.11
  networkCIDR: 10.10.0.0/16
  networkID: vpc-xxxxx
  networking:
    kubenet: {}
  nonMasqueradeCIDR: 100.64.0.0/10

[Benjamin-Dobell]

Also running into this error with a Kops based cluster:

I1213 19:27:54.538610       1 elbv2.go:85] Request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "<ARN>",  Targets: [{      Id: "100.96.2.5",      Port: 80    }]}
E1213 19:27:54.763210       1 api.go:2819] Failed request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "<ARN>",  Targets: [{      Id: "100.96.2.5",      Port: 80    }]}, Error: ValidationError: The Availability Zone for IP address '100.96.2.5' must be 'all' for Application Load Balancer target groups, when not within the VPC
    status code: 400, request id: 3056747e-ff0d-11e8-85b0-db627847483e
E1213 19:27:54.763244       1 targets.go:80] default/shared-ingress: Error adding targets to <ARN>: ValidationError: The Availability Zone for IP address '100.96.2.5' must be 'all' for Application Load Balancer target groups, when not within the VPC
E1213 19:27:54.763251       1 targets.go:80] default/shared-ingress:    status code: 400, request id: 3056747e-ff0d-11e8-85b0-db627847483e
E1213 19:27:54.763381       1 :0] kubebuilder/controller "msg"="Reconciler error" "error"="failed to reconcile targetGroups due to failed to reconcile targetGroup targets due to ValidationError: The Availability Zone for IP address '100.96.2.5' must be 'all' for Application Load Balancer target groups, when not within the VPC\n\tstatus code: 400, request id: 3056747e-ff0d-11e8-85b0-db627847483e"  "Controller"="alb-ingress-controller" "Request"={"Namespace":"default","Name":"shared-ingress"}

[M00nF1sh]

/kind bug
Sorry the trouble, that use case is not captured when removing the code.(missing documentation about this case in original code).

I'll fix this asap and release a new version.

[M00nF1sh]

The situation is harder for managed securityGroups.

needs to figure out the ENI for the fake IP based on configuration of VPC RouteTable. And reorganize the code dealing with it for easier maintain.

Not sure whether there is an clean way to detect whether we are running on Amazon CNI or Kubenet

[EricDube]

@M00nF1sh We're you able to find a fix for this issue? I am currently hitting the same issue and my current work-around is to manually add the ip to the target groups but it only works for testing as it will change if we need to reload the ingress.

[M00nF1sh]

@EricDube
The currently workaround is use mode:instance instead for mode:ip if you are using kubenet.
Simple fixes do exists for customer managed securityGroup, but full fix to make it work under managed securityGroup needs more work than i originally expected, i havn't worked on it given the priorities is low(since it's not hard blocker).

[rahulwa]

This is also happening when running in EKS.
This is happening when trying to create ingress having external service pointing to ec2 running on different peered VPC. Using:
ALB Ingress: v1.1.1
CNI: amazon-k8s-cni:1.3.0

This was working in beta versions.
Error:

E0222 12:22:25.390598       1 targets.go:80] xxx: Error adding targets to arn:aws:elasticloadbalancing:us-west-2:xxxx:targetgroup/xxxx-0f7d85428a614c67908/9917d1b975437722: ValidationError: The Availability Zone for IP address '172.30.xx.xx' must be 'all' for Application Load Balancer target groups, when not within the VPC
E0222 12:22:25.390612       1 targets.go:80] xxx:   status code: 400, request id: 82efeb04-369c-11e9-a6e8-097b3c7f56de