ingress-nginx controller not creating the ELB

[roytanmoy]

NGINX Ingress controller version:
Release: 0.21.0
Build: git-b65b85cd9
Repository: https://github.com/aledbf/ingress-nginx

Kubernetes version (use kubectl version):
v1.13.1

Environment:
AWS

What happened:
I am trying to set up ingress-nginx-controller with type as LoadBalancer
using this guide for the configuration.
Couple of issues I'm observing -
issue#1) It does not create and deploy a ELB onto the AWS.
issue#2) The External-IP status is showing as pending for ever.

$ kubectl get services -n ingress-nginx  
NAME            TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE   
ingress-nginx   LoadBalancer   10.108.245.210   <pending>     80:30742/TCP,443:31028/TCP   41m

I've followed each step mentioned there

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml  
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/aws/service-l4.yaml  
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/aws/patch-configmap-l4.yaml

here is the logs from the pod -

~$ kubectl get pods -n ingress-nginx
NAME                                        READY   STATUS    RESTARTS   AGE
default-http-backend-7b8bdbc579-5tgd5       1/1     Running   0          5h57m
nginx-ingress-controller-766c77b7d4-w6wkk   1/1     Running   1          6h16m

$ kubectl logs -n ingress-nginx nginx-ingress-controller-766c77b7d4-w6wkk | more  
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.21.0
  Build:      git-b65b85cd9
  Repository: https://github.com/aledbf/ingress-nginx
-------------------------------------------------------------------------------

nginx version: nginx/1.15.6
W1228 06:50:15.592738       7 client_config.go:548] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1228 06:50:15.593115       7 main.go:196] Creating API client for https://10.96.0.1:443
I1228 06:50:15.699540       7 main.go:240] Running in Kubernetes cluster version v1.13 (v1.13.1) - git (clean) commit eec55b9ba98609a46fee712359c7b5b365bdd920 - platfo$
m linux/amd64
I1228 06:50:16.060958       7 nginx.go:258] Starting NGINX Ingress controller
I1228 06:50:16.098387       7 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"nginx-configuration", UID:"60a8d471-0a6c-11e9-9$
55-024a9b465fb2", APIVersion:"v1", ResourceVersion:"1425", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/nginx-configuration
I1228 06:50:16.098548       7 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"tcp-services", UID:"60acc5c1-0a6c-11e9-9a55-024$
9b465fb2", APIVersion:"v1", ResourceVersion:"1426", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/tcp-services
I1228 06:50:16.103229       7 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"60ae5849-0a6c-11e9-9a55-024$
9b465fb2", APIVersion:"v1", ResourceVersion:"1428", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services
I1228 06:50:17.262403       7 nginx.go:279] Starting NGINX process
I1228 06:50:17.263106       7 leaderelection.go:187] attempting to acquire leader lease  ingress-nginx/ingress-controller-leader-nginx...
I1228 06:50:17.267252       7 controller.go:172] Configuration changes detected, backend reload required.
I1228 06:50:17.280698       7 leaderelection.go:196] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I1228 06:50:17.281401       7 status.go:148] new leader elected: nginx-ingress-controller-766c77b7d4-w6wkk
I1228 06:50:18.113969       7 controller.go:190] Backend successfully reloaded.
I1228 06:50:18.114768       7 controller.go:202] Initial sync, sleeping for 1 second.
[28/Dec/2018:06:50:19 +0000]TCP200000.000
I1228 06:51:07.572896       7 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"nginx-configuration", UID:"60a8d471-
0a6c-11e9-9a55-024a9b465fb2", APIVersion:"v1", ResourceVersion:"1783", FieldPath:""}): type: 'Normal' reason: 'UPDATE' ConfigMap ingress-nginx/nginx-configu
ration
I1228 06:51:07.995982       7 controller.go:172] Configuration changes detected, backend reload required.
I1228 06:51:12.846822       7 controller.go:190] Backend successfully reloaded.
[28/Dec/2018:06:51:13 +0000]TCP200000.000
W1228 10:14:17.301340       7 queue.go:130] requeuing &ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,Cre
ationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string
{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,}, err services "ingress-nginx" not found
W1228 10:14:17.313074       7 queue.go:130] requeuing &ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,Cre
ationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string
{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,}, err services "ingress-nginx" not found
W1228 10:14:17.320110       7 queue.go:130] requeuing &ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,Cre
ationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string
{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,}, err services "ingress-nginx" not found
<truncated>

service status

~$ kubectl get service -n ingress-nginx
NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
default-http-backend   ClusterIP      10.100.2.163    <none>        80/TCP                       5h49m
ingress-nginx          LoadBalancer   10.108.221.18   <pending>     80:32010/TCP,443:31271/TCP   170m

$kubectl describe service ingress-nginx -n ingress-nginx
Name:                     ingress-nginx
Namespace:                ingress-nginx
Labels:                   app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"service.beta.kubernetes.io/aws-load-balancer-type":"nlb"},"label
s":{"app.k...
                          service.beta.kubernetes.io/aws-load-balancer-type: nlb
Selector:                 app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx
Type:                     LoadBalancer
IP:                       10.108.221.18
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  32010/TCP
Endpoints:                10.244.0.4:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  31271/TCP
Endpoints:                10.244.0.4:443
Session Affinity:         None
External Traffic Policy:  Local
HealthCheck NodePort:     30401
Events:                   <none>

I know there is a service specification tag externalIPs but expectation here is that once ELB instance is created, it will be auto populated -

spec:
  type: LoadBalancer
  externalIPs:
  - {{ ingress_lb_address or vip or masterIP }}

[roytanmoy]

/kind bug

[aledbf]

Closing. The ingress controller does not create any cloud resource. This is done by Kubernetes.
To see the reason for the error, please run kubectl describe service -n ingress-nginx ingress-nginx
Usually, you will see the reason in the events

[roytanmoy]

thanks Aledbf. There is no event showing up in describe service(log given above ). Could you please refer me any code which I can follow?

[aledbf]

@roytanmoy then check the events in the namespace kubectl get events -n ingress-nginx

[aledbf]

@roytanmoy also, you can run kubectl cluster-info dump | grep LoadBalancer

[roytanmoy]

here is ourput . It suggests about default-http-backend service -

`~$ kubectl get events -n ingress-nginx
LAST SEEN   TYPE      REASON      KIND   MESSAGE
23m         Warning   Unhealthy   Pod    Liveness probe failed: Get http://10.244.0.4:10254/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)   
23m         Warning   Unhealthy   Pod    Readiness probe failed: Get http://10.244.0.4:10254/healthz: net/http: request canceled (Client.Timeout exceeded w$ile awaiting headers)`

[aledbf]

http://10.244.0.4:10254/healthz

That means the kubelet in the node where the pod is running cannot reach that URL. I suggest you SSH to the node and try to use curl to check this.
Usually, this means you have some network issue with the CNI provider.

[roytanmoy]

`$ telnet 10.244.0.4 10254
Trying 10.244.0.4...
Connected to 10.244.0.4.
Escape character is '^]'.`  


~$ curl http://10.244.0.4:10254/healthz
ok

[roytanmoy]

@aledbf thanks...the issue is found now-
~$ kubectl cluster-info dump | grep LoadBalancer E1228 14:35:47.072444 1 core.go:76] Failed to start service controller: WARNING: no cloud provider provided, services of type LoadBalancer will fail

[aledbf]

Being new into this, no idea of where to configure the "cloud provider" tag.

This should be configured during the installation of the cluster. How did you installed the cluster in AWS?

[sgandewar]

@roytanmoy also, you can run kubectl cluster-info dump | grep LoadBalancer

I am having the same issue and here is what I see for the o/p.

"LoadBalancerIP": "",
"LoadBalancerSourceRanges": null,
"LoadBalancer": {
"LoadBalancerIP": "",
"LoadBalancerSourceRanges": null,
"LoadBalancer": {
"LoadBalancerIP": "",
"LoadBalancerSourceRanges": null,
"LoadBalancer": {
"LoadBalancerIP": "",
"LoadBalancerSourceRanges": null,
"LoadBalancer": {

Can you please advise?

[passionatedevops]

run : helm install --namespace nginx --name nginx
you can check the helm install --namespace nginx --name nginx log using : kubectl describe service -n ingress-nginx ingress-nginx

if you are getting error like Error creating load balancer:

Add ElasticLoadBalancingFullAccess policy in the the eksservicerole:

aws iam attach-role-policy `
 --role-name **eksServiceRoleName** `
 --policy-arn arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess