Aws-load-balancer-controller: Error with ingress about subnets and target groups

[jitesh-prajapati123]

I am getting following errors highlighted in bold. Please check attached yaml.

W1206 11:47:08.348514 1 client_config.go:552] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.

AWS ALB Ingress controller
Release: v1.0.1
Build: git-ebac62dd

Repository: https://github.com/kubernetes-sigs/aws-alb-ingress-controller.git

### E1206 11:47:54.458578 1 :0] kubebuilder/controller "msg"="Reconciler error" "error"="failed to reconcile targetGroups due to failed to load serviceAnnotation due to no object matching key \"default/service1\" in local store" "Controller"="alb-ingress-controller" "Request"={"Namespace":"default","Name":"alb-eks-dev-ingress"}

### E1206 11:47:55.459306 1 :0] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to retrieval of subnets failed to resolve 2 qualified subnets. Subnets must contain the kubernetes.io/cluster/\u003ccluster name\u003e tag with a value of shared or owned and the kubernetes.io/role/internal-elb tag signifying it should be used for ALBs Additionally, there must be at least 2 subnets with unique availability zones as required by ALBs. Either tag subnets to meet this requirement or use the subnets annotation on the ingress resource to explicitly call out what subnets to use for ALB creation. The subnets that did resolve were []" "Controller"="alb-ingress-controller" "Request"={"Namespace":"2048-game","Name":"2048-ingress"}

ingress.yaml
`
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: alb-eks-dev-ingress
labels:
app: alb-eks-dev
annotations:
# trigger the alb-ingress-controller
kubernetes.io/ingress.class: alb

# set ALB parameters
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: instance
alb.ingress.kubernetes.io/security-groups: sg-03fddf828ea7cd3eb
alb.ingress.kubernetes.io/subnets: subnet-0ffe7c5009c4bc894,subnet-0f6576de092e78ba5
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:374323471964:certificate/65bfa94c-69c9-48cd-8c8c-86b9ac49cb3d
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80,"HTTPS": 443}]'

# allow 404s on the health check
alb.ingress.kubernetes.io/healthcheck-path: "/"
alb.ingress.kubernetes.io/success-codes: '200,404'

spec:
rules:

• http:
  paths:
  
  
  
  • path: "/service1"
    
    backend:
    
    serviceName: service1
    
    servicePort: 3001
    
    `
  
  

service1.yaml
{ "apiVersion":"v1", "kind":"Service", "metadata":{ "name":"service1", "labels":{ "app":"service1" } }, "spec":{ "ports": [ { "port":80, "targetPort":3001 } ], "selector":{ "app":"service1" }, "type": "NodePort" } }
service_nodeport.txt
ingress.txt

[M00nF1sh]

Hi,

1. I suppose the above error only happens before you added the alb.ingress.kubernetes.io/subnets: annotation, am i correct?
   The error message is misleading, you need to add kubernetes.io/role/elb tag onto public subnets instead of kubernetes.io/role/internal-elb to let controller auto-discovery it for internet-facing ingresses.

1. What's the new logs after you applied the alb.ingress.kubernetes.io/subnets annotation?
   (By default only error & modify actions are shown), you can add an -v=1 arg into the ingress controller's yaml, which will output Successfully Reconciled messages.

[jitesh-prajapati123]

I have redeployed it and now I am getting following error. Please check attachments.

E1207 09:49:47.896715 1 :0] kubebuilder/controller "msg"="Reconciler error" "error"="failed to find existing LoadBalancer due to RequestError: send request failed\ncaused by: Post https://elasticloadbalancing.us-east-1.amazonaws.com/: dial tcp: i/o timeout" "Controller"="alb-ingress-controller" "Request"={"Namespace":"myservices","Name":"alb-eks-dev-ingress"}

[jitesh-prajapati123]

I have experienced weird behavior.

It works if i enable debugging in "alb ingress controller". (- ---aws-api-debug).

other wise i am getting following error.

E1207 09:49:47.896715 1 :0] kubebuilder/controller "msg"="Reconciler error" "error"="failed to find existing LoadBalancer due to RequestError: send request failed\ncaused by: Post https://elasticloadbalancing.us-east-1.amazonaws.com/: dial tcp: i/o timeout" "Controller"="alb-ingress-controller" "Request"={"Namespace":"myservices","Name":"alb-eks-dev-ingress"}

[M00nF1sh]

This is indeed super weird.... the --aws-api-debug shouldn't have changed anything in network level.
Is it possible to me to get access to your cluster for debugging? There is so less information to figure this out...

[jitesh-prajapati123]

Unfortunately I can not share access of cluster, as its my company's AWS account.
But, I have shared asset files with your personal email address.

[M00nF1sh]

Hi, you can send your assets to [email protected]

[jitesh-prajapati123]

Had sent on same.
Let me know if you haven't received.

[jitesh-prajapati123]

I have re-tested it.
It only works for me if I enable debugging (--aws-api-debug) for alb-ingress-controller.

PFA logs file
logs.log

[Ohadbasan]

Hi,

1. I suppose the above error only happens before you added the alb.ingress.kubernetes.io/subnets: annotation, am i correct?
   The error message is misleading, you need to add kubernetes.io/role/elb tag onto public subnets instead of kubernetes.io/role/internal-elb to let controller auto-discovery it for internet-facing ingresses.
2. What's the new logs after you applied the alb.ingress.kubernetes.io/subnets annotation?
   (By default only error & modify actions are shown), you can add an -v=1 arg into the ingress controller's yaml, which will output Successfully Reconciled messages.

This worked for me. it is not in the docs.

[jitesh-prajapati123]

This issue is producing to me only. while its not producing to other with same asset.
Considering some environment / firewall issue and closing same.

[bconner22]

While going through the https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/walkthrough/echoserver/ walkthrough,I had to go back to using the alb.ingress.kubernetes.io/subnets annotations. The tags on my eks worker node subnets(spanning 3 AZ's) and pulling out the alb.ingress.kubernetes.io/subnets annotations doesn't work. I'm using an internal load balancer rather than internet-facing for what it's worth, but I tried the internet facing one that's in the walkthrough originally, no dice with that one. I did enable the - ---aws-api-debug(you guys have an extra - in there by the way). I get the reconcile error, as well as one not being able to find any subnets:

kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to retrieval of subnets failed to resolve 2 qualified subnets. Subnets must contain the kubernetes.io/cluster/\u003ccluster name\u003e tag with a value of shared or owned and the kubernetes.io/role/internal-elb tag signifying it should be used for ALBs Additionally, there must be at least 2 subnets with unique availability zones as required by ALBs. Either tag subnets to meet this requirement or use the subnets annotation on the ingress resource to explicitly call out what subnets to use for ALB creation. The subnets that did resolve were []" "controller"="alb-ingress-controller" "request"={"Namespace":"echoserver","Name":"echoserver"}

I have the following tags on my eks worker node subnets:
kubernetes.io/role/internal-elb with a value of 1
kubernetes.io/cluster/myclustername with a value of shared
And, not sure if relevant anymore, I have a
kubernetes.io/role/alb-ingress witha a value of

I left those tags on and brought back the alb.ingress.kubernetes.io/subnets annotations, and now the controller and ingress seem to be behaving, although Route53 is acting up, I may be using it incorrectly(more on that below). I do have the cool automagically generated Security Groups on both my Load balancer and eks instances now that opens comms between the 2.

In the ingress, I'm using my host as blah.nameofmydomain.com rather than nameofmydomain.com because I only want to affect one record set right now, not the entire domain. This seemed to have worked in the past, but now I'm only getting one of the route53 nameservers when I dig blah.nameofmydomain.com. To fix this, I have to keep manually deleting the recordset in route53 and remaking it. Eventually I'll get everything to propagate, I'll get all the nameservers in that zone when I dig, and finally nslookup blah.nameofmydomain.com will work. I'm not sure if this is an alb-ingress-controller problem, an ingress problem, or a route53 problem.

[Erokos]

Hi,
we're having a similar issue as described by jitesh-prajapati123. Similar in a way that we have an EKS cluster and wanted to set up an ALB using the aws alb ingress controller. We were following the steps carefully as described by the documentation and our public and private subnets have the necessary tags specified for deploying an external (internet facing) ALB.

Our worker nodes are in private subnets and security groups have been setup correctly in order for the cluster to successfully deploy our services. This has been verified. We've also created the ingressController-iam-policy and attched it to the worker nodes.

When executing this command:

kubectl logs -n kube-system $(kubectl get po -n kube-system | egrep -o "alb-ingress[a-zA-Z0-9-]+")

the first line from the logs:

client_config.go:549] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.

The last lines where any errors are:

kubebuilder/controller "msg"="Reconciler error" "error"="failed to reconcile targetGroups due to failed to reconcile targetGroup targets due to Unable to find the default/echo service: no object matching key "default/echo" in local store" "controller"="alb-ingress-controller" "request"={"Namespace":"default","Name":"api"}

we've deployed the following ingress:

`
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: api
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: instance
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80}]'
alb.ingress.kubernetes.io/success-codes: '200,404'
spec:
tls:

• hosts:
  
  
  
  • 
    
    rules:
  
  
• host:
  http:
  paths:
  
  
  
  • path: /echo
    
    backend:
    
    serviceName: echo
    
    servicePort: 80`
  
  

the following deployment and NodePort type service:

`
apiVersion: apps/v1
kind: Deployment
metadata:
name: echo
labels:
app: echo
spec:
replicas: 2
selector:
matchLabels:
app: echo
template:
metadata:
labels:
app: echo
spec:
containers:
- name: echo
image: 064211542743.dkr.ecr.eu-west-1.amazonaws.com/echo-api:1.0
ports:
- containerPort: 8080

apiVersion: v1
kind: Service
metadata:
name: echo
spec:
type: NodePort
selector:
app: echo
ports:

• port: 80
  targetPort: 8080
  protocol: TCP`

We're testing this using this command:
curl -XPOST 'http://<my_host_domain>/echo/echo' -d '{"data": {"Hello": "World"}}' but it keeps hanging.
It also hangs when using
curl -XPOST 'http://<cluster-ip>:<nodeport>/echo/echo' -d '{"data": {"Hello": "World"}}'

Does anyone have a suggestion? Any help would be greatly appreciated

[esker]

@Erokos I would add a namespace: default to your echo service right under the name