Aws-load-balancer-controller: 1.0-beta.6::Validation error when creating target group with target-type ip

[akshayks]

I seem to be running into a strange error when attempting to create an ingress resource with target-type ip. When I change the target-type to instance or remove the annotation altogether, ALB and Target group creation proceed without errors. My cluster is running version 1.0-beta.6 of the ALB Ingress controller on Amazon EKS.

Here is the relevant snippet from kubectl logs:
```I0821 16:35:40.370087 1 targetgroup.go:221] kube-system/stage-us-west-2-cluster1-a01-kube-system: Start TargetGroup creation.
E0821 16:35:40.370166 1 api.go:685] Failed request: elasticloadbalancing/CreateTargetGroup, Payload: { HealthCheckIntervalSeconds: 15, HealthCheckPath: "/health", HealthCheckPort: "traffic-port", HealthCheckProtocol: "HTTP", HealthCheckTimeoutSeconds: 5, HealthyThresholdCount: 2, Matcher: { HttpCode: "200" }, Name: "e2f84795-6825f822af2a82f5b0a", Port: 0, Protocol: "HTTP", TargetType: "ip", UnhealthyThresholdCount: 2, VpcId: "vpc-99d6e3e0"}, Error: InvalidParameter: 1 validation error(s) found.

• minimum field value of 1, CreateTargetGroupInput.Port.
  I0821 16:35:40.370307 1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kube-system", Name:"stage-us-west-2-cluster1-a01-kube-system", UID:"55e0ca2e-a4df-11e8-882c-0a5cf1309096", APIVersion:"extensions/v1beta1", ResourceVersion:"1434110", FieldPath:""}): type: 'Warning' reason: 'ERROR' Error creating target group e2f84795-6825f822af2a82f5b0a: InvalidParameter: 1 validation error(s) found.
• minimum field value of 1, CreateTargetGroupInput.Port.
  I0821 16:35:40.370360 1 listener.go:260] Request: elasticloadbalancing/ModifyListener, Payload: { Certificates: [{ CertificateArn: "arn:aws:acm:us-west-2:962575642969:certificate/145dae67-efb9-472d-9cea-e80cb3137daf" }], DefaultActions: [{ Type: "forward" }], ListenerArn: "arn:aws:elasticloadbalancing:us-west-2:962575642969:listener/app/e2f84795-kubesystem-stageu-e659/a2c9f256cac91735/eef0b19f4cefc292", Port: 443, Protocol: "HTTPS", SslPolicy: "ELBSecurityPolicy-2016-08"}
  E0821 16:35:40.399632 1 api.go:2333] Failed request: elasticloadbalancing/ModifyListener, Payload: { Certificates: [{ CertificateArn: "arn:aws:acm:us-west-2:962575642969:certificate/145dae67-efb9-472d-9cea-e80cb3137daf" }], DefaultActions: [{ Type: "forward" }], ListenerArn: "arn:aws:elasticloadbalancing:us-west-2:962575642969:listener/app/e2f84795-kubesystem-stageu-e659/a2c9f256cac91735/eef0b19f4cefc292", Port: 443, Protocol: "HTTPS", SslPolicy: "ELBSecurityPolicy-2016-08"}, Error: ValidationError: A target group ARN must be specified
  status code: 400, request id: 3d9b6e88-a560-11e8-b06c-b770cd2890ea
  I0821 16:35:40.399670 1 targetgroup.go:221] kube-system/stage-us-west-2-cluster1-a01-kube-system: Start TargetGroup creation.
  E0821 16:35:40.399721 1 api.go:685] Failed request: elasticloadbalancing/CreateTargetGroup, Payload: { HealthCheckIntervalSeconds: 15, HealthCheckPath: "/health", HealthCheckPort: "traffic-port", HealthCheckProtocol: "HTTP", HealthCheckTimeoutSeconds: 5, HealthyThresholdCount: 2, Matcher: { HttpCode: "200" }, Name: "e2f84795-6825f822af2a82f5b0a", Port: 0, Protocol: "HTTP", TargetType: "ip", UnhealthyThresholdCount: 2, VpcId: "vpc-99d6e3e0"}, Error: InvalidParameter: 1 validation error(s) found.
• minimum field value of 1, CreateTargetGroupInput.Port.
  E0821 16:35:40.399747 1 albingress.go:290] kube-system/stage-us-west-2-cluster1-a01-kube-system: Failed to reconcile state on this ingress
  E0821 16:35:40.399756 1 albingress.go:292] kube-system/stage-us-west-2-cluster1-a01-kube-system: - Failed TargetGroup creation: InvalidParameter: 1 validation error(s) found.
  E0821 16:35:40.399759 1 albingress.go:292] kube-system/stage-us-west-2-cluster1-a01-kube-system: - minimum field value of 1, CreateTargetGroupInput.Port.
  E0821 16:35:40.399763 1 albingress.go:292] kube-system/stage-us-west-2-cluster1-a01-kube-system: .
  E0821 16:35:40.399770 1 albingress.go:292] kube-system/stage-us-west-2-cluster1-a01-kube-system: - Failed Listener modification: ValidationError: A target group ARN must be specified
  E0821 16:35:40.399773 1 albingress.go:292] kube-system/stage-us-west-2-cluster1-a01-kube-system: status code: 400, request id: 3d9b6e88-a560-11e8-b06c-b770cd2890ea
  I0821 16:35:40.399759 1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kube-system", Name:"stage-us-west-2-cluster1-a01-kube-system", UID:"55e0ca2e-a4df-11e8-882c-0a5cf1309096", APIVersion:"extensions/v1beta1", ResourceVersion:"1434110", FieldPath:""}): type: 'Warning' reason: 'ERROR' Error modifying 443 listener: ValidationError: A target group ARN must be specified
  status code: 400, request id: 3d9b6e88-a560-11e8-b06c-b770cd2890ea
  E0821 16:35:40.399779 1 albingress.go:292] kube-system/stage-us-west-2-cluster1-a01-kube-system: - Failed TargetGroup creation: InvalidParameter: 1 validation error(s) found.
  E0821 16:35:40.399783 1 albingress.go:292] kube-system/stage-us-west-2-cluster1-a01-kube-system: - minimum field value of 1, CreateTargetGroupInput.Port.
  E0821 16:35:40.399787 1 albingress.go:292] kube-system/stage-us-west-2-cluster1-a01-kube-system: .
  I0821 16:35:40.399789 1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kube-system", Name:"stage-us-west-2-cluster1-a01-kube-system", UID:"55e0ca2e-a4df-11e8-882c-0a5cf1309096", APIVersion:"extensions/v1beta1", ResourceVersion:"1434110", FieldPath:""}): type: 'Warning' reason: 'ERROR' Error creating target group e2f84795-6825f822af2a82f5b0a: InvalidParameter: 1 validation error(s) found.
• minimum field value of 1, CreateTargetGroupInput.Port.
  E0821 16:35:40.399796 1 albingress.go:295] kube-system/stage-us-west-2-cluster1-a01-kube-system: Will retry to reconcile in 1m4.032278813s

My ingress definition is as follows:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: stage-us-west-2-cluster1-a01-kube-system
annotations:
kubernetes.io/ingress.class: "alb"
external-dns.alpha.kubernetes.io/dns-type: "public"
alb.ingress.kubernetes.io/scheme: "internal"
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
alb.ingress.kubernetes.io/success-codes: '200'
alb.ingress.kubernetes.io/target-type: 'ip'
alb.ingress.kubernetes.io/security-groups: ''
alb.ingress.kubernetes.io/subnets: ''
alb.ingress.kubernetes.io/certificate-arn: ''
labels:
app: kube-system-lb
tier: kube-system
partition: a01
spec:
tls:
- hosts:
-
rules:
- host:
http:
paths:
- path: /
backend:
serviceName: cargo-repo-chartmuseum
servicePort: http
```

Apologies if this has been reported already. Also, happy to provide more information if required.

[d-nishi]

/assign @M00nF1sh
cc @bigkraig

[M00nF1sh]

I guess the problem is caused by using named string format targetPort in service spec(which is supported by kubernetes)

This this the code that causing problem, we need to find a way to convert named port to numeric port

[d-nishi]

@akshayks - can you send us your service specification in your manifest to confirm Yang's assertion?

[neerfri]

@d-nishi My service has used a named port. Supporting @M00nF1sh's thoughts.

[M00nF1sh]

cool, I'll create an PR to fix this

[akshayks]

@d-nishi @M00nF1sh Here is my service definition:

apiVersion: v1
kind: Service
metadata:
  annotations:
    alb.ingress.kubernetes.io/healthcheck-path: /health
  labels:
    app: chartmuseum
    chart: chartmuseum-1.6.0
    heritage: Tiller
    release: cargo-repo
  name: cargo-repo-chartmuseum
  namespace: kube-system
spec:
  externalTrafficPolicy: Cluster
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: http
  selector:
    app: chartmuseum
    release: cargo-repo
  type: NodePort

I can try modifying the service definition to eliminate the named port and see if that makes a difference.

[akshayks]

I tried editing the service definition to use a numeric port and that seems to work with target type ip. This should be a reasonable work around until the controller can be modified to expect named ports.