Ingress-nginx: service default/... does not have any active endpoints

[foxylion]

Hi,

I've upgraded gcr.io/google_containers/nginx-ingress-controller from 0.8.3 to 0.9.0-beta.1. But now I'm facing the issue that no service is available through the ingress controller. It will always result in 503 status code responses.

Log looks like this:

W0208 15:59:58.498949       6 controller.go:806] service default/webapp-master does not have any active endpoints
W0208 15:59:58.499025       6 controller.go:560] service default/ingress-nginx-default-backend does not have any active endpoints
(repeating every 10 seconds)

The generated nginx.conf contains ony upstream entries with 127.0.0.1:8181 as backend endpoint:

   upstream default-webapp-master-80 {
        least_conn;
        server 127.0.0.1:8181 max_fails=0 fail_timeout=0;
    }
    upstream upstream-default-backend {
        least_conn;
        server 127.0.0.1:8181 max_fails=0 fail_timeout=0;
    }

Any idea what is going wrong?

I only updated the ingress controller. ingress resources where not changed. One example ingress resource looks like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: webapp-master
  annotations:
    proxy-body-size: 100m
spec:
  rules:
  - host: hostname.domain.example
    http:
      paths:
      - path: /
        backend:
          serviceName: webapp-master
          servicePort: 80

The matching service like this:

kind: Service
apiVersion: v1
metadata:
  name: webapp-master
spec:
  ports:
  - port: 80
    targetPort: http
  selector:
    app: webapp-master

[aledbf]

@foxylion there is a bug in 0.9.0-beta.1 already fixed in master (https://github.com/kubernetes/ingress/pull/121).
In the meantime if you change targetPort to 80 instead a string it will work.
Apologies for the inconvenience.

[foxylion]

@aledbf Thanks for your quick response. I'll try that.

Is there any schedule when new releases are done?

[aledbf]

Fix included in next release #303
Please reopen if you still have issues.

[JonatanE]

This is not fixed in 0.9.0-beta.4

2017/04/27 09:19:30 [warn] 12477#12477: *39354 using uninitialized "proxy_upstream_name" variable while logging request

[madaojisan]

still in in 0.9.0-beta.5

2017/05/24 08:05:38 [warn] 379#379: *7 using uninitialized "proxy_upstream_name" variable while logging request, client: myclient, server: _, request: "GET / HTTP/1.1", host: "myhost"

[paolomainardi]

I have the same problems described here and i confirm with the latest beta6 and i confirm that changing targetPort from a string to the port fixed the ingress, but i have a lot of panic like this:

 0.054 401
panic: runtime error: index out of range

goroutine 660 [running]:
k8s.io/ingress/vendor/github.com/paultag/sniff/parser.GetSNBlock(0xc4204d8070, 0xf90, 0xf90, 0xc4204d8070, 0xf90, 0xf90, 0x0, 0x0)
    /home/nicksardo/go/src/k8s.io/ingress/vendor/github.com/paultag/sniff/parser/parser.go:91 +0x1d1
k8s.io/ingress/vendor/github.com/paultag/sniff/parser.GetHostname(0xc4204d8000, 0x1000, 0x1000, 0x1000, 0x205, 0x0, 0x0)
    /home/nicksardo/go/src/k8s.io/ingress/vendor/github.com/paultag/sniff/parser/parser.go:42 +0xee
main.(*proxy).Handle(0xc4204d4f60, 0x1d6ada0, 0xc4204470e8)
    /home/nicksardo/go/src/k8s.io/ingress/controllers/nginx/pkg/cmd/controller/tcp.go:45 +0x1dc
created by main.newNGINXController.func1
    /home/nicksardo/go/src/k8s.io/ingress/controllers/nginx/pkg/cmd/controller/nginx.go:121 +0x260
nginx git:dev ❯ kubectl get pod -nnginx-ingress

[paolomainardi]

nginx git:dev ❯ kubectl get pod -nnginx-ingress                                                                                                                                                   ✹
NAME                                    READY     STATUS             RESTARTS   AGE
default-http-backend-3723061492-lbbbv   1/1       Running            0          18d
nginx-3602923031-h2582                  0/1       CrashLoopBackOff   6          15m
nginx-3602923031-hm4l8                  1/1       Running            5          15m

[paolomainardi]

I had to revert back the ingress controller to the 0.8.3.

What could i share to debug better the issue ?

[foxylion]

@paolomainardi I would suggest you to create a separate issue (this shouldn't be caused by this one).
And then ideally add a minimal example to reproduce the issue (including ingress and service configurations).

[paolomainardi]

@foxylion to me it seems related as i had "service default/service does not have any active endpoints" that was causing the failure that has been fixed changing the targetPort from a string to an integer.

[foxylion]

@paolomainardi Okay, so replacing the labeled targetPorts fixes the "index out of range" panic? Sorry, then this should maybe be reopened.

ping @aledbf

[aledbf]

@paolomainardi can you share the service and ingress definition?

[f0]

@aledbf i have the same problem

In gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.5 it does work, with

gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.7 i got the error

W0527 15:27:27.667881       1 controller.go:842] service external-logging/kibana-svc does not have any active endpoints

Here are the deployment, service and Ingress

metadata:
  name: kibana-deployment
  namespace: external-logging
spec:
  replicas: 1
  template:
    metadata:
      labels:
        component: kibana
    spec:
      containers:
      - name: kibana
        image: example.net/kibana:5.2.2
        imagePullPolicy: IfNotPresent
        resources:
          # keep request = limit to keep this container in guaranteed class
          limits:
            cpu: 100m
          requests:
            cpu: 100m
        env:
        - name: "ELASTICSEARCH_URL"
          value: "http://elasticsearch-external-logging:9200"
        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kibana-ingress
  namespace: external-logging
spec:
  rules:
  - host: kibana.example.net
    http:
      paths:
      - path: /
        backend:
          serviceName: kibana-svc
          servicePort: 5601



md5-a6fb2ddac916ed064cdbb942b9db7b05



apiVersion: v1
kind: Service
metadata:
  name: kibana-svc
  namespace: external-logging
  labels:
    component: kibana-svc
spec:
  type: NodePort
  ports:
  - port: 5601
    protocol: TCP
    targetPort: 5601
  selector:
    component: kibana

[chainlink]

I had the same error and changed my path from /* to / and it worked (I see you have / above though

[chainlink]

The offending line (I think) https://github.com/kubernetes/ingress/blob/3f0307a96ee58dce89b66cf99a4464e18969e171/core/pkg/ingress/controller/controller.go#L507

[jhdehart]

I've encountered the same issue with the fluentd-elasticsearch service and it turned out to be RBAC related.

I viewed the following in my logs

W0601 18:40:36.085507 1 controller.go:1106] error mapping service ports: error syncing service kube-system/kibana-logging:
User "system:serviceaccount:nginx-ingress:nginx-ingress-serviceaccount" cannot update services in the namespace "kube-system". (put services kibana-logging)

An easy fix was to change the targetPort parameter from ui to 5601 in the kibana-logging service definition. Proper fix was updating the permissions.

[ianmartorell]

I encountered this issue after upgrading from 0.8.3 to 0.9.0-beta.17. Changing targetPorts in services from labels to integers fixed it, so I guess this should be reopened.

[aledbf]

@ianmartorell please open a new issue

[nthypes]

I was having this error in the newest version and follow the @chainlink advise.

Change the path to / and works like a charm!

It was /* before.

[0312birdzhang]

still in quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.13.0

I0509 02:52:13.119086      17 controller.go:179] ingress backend successfully reloaded...
W0509 02:52:27.241087      17 controller.go:775] service kube-system/mongoclient does not have any active endpoints
W0509 02:52:27.241269      17 controller.go:315] service go/service-ll-onegameinfo does not have any active endpoints for port 60104 and protocol TCP
W0509 02:52:27.241303      17 controller.go:315] service default/service-xxzhushou-script-detail04 does not have any active endpoints for port 12004 and protocol TCP
W0509 02:52:27.241323      17 controller.go:315] service go/service-ipa-pkg-relation does not have any active endpoints for port 20401 and protocol TCP
W0509 02:52:27.241329      17 controller.go:315] service go/service-ll-open-game-list does not have any active endpoints for port 60102 and protocol TCP
W0509 02:52:27.241353      17 controller.go:315] service go/service-gift-center does not have any active endpoints for port 30001 and protocol TCP
W0509 02:52:27.241369      17 controller.go:315] service default/service-xxzhushou-script-detail02 does not have any active endpoints for port 12002 and protocol TCP
W0509 02:52:27.241378      17 controller.go:315] service default/service-banner-center does not have any active endpoints for port 13010 and protocol TCP
W0509 02:52:27.241395      17 controller.go:315] service go/service-ipa-base-info-init does not have any active endpoints for port 20402 and protocol TCP
W0509 02:52:27.241414      17 controller.go:315] service go/service-ll-onegameinfo does not have any active endpoints for port 14000 and protocol TCP
W0509 02:52:27.241435      17 controller.go:315] service go/service-game-center-brief does not have any active endpoints for port 13000 and protocol TCP
W0509 02:52:27.241443      17 controller.go:315] service go/service-ipa-push-collect does not have any active endpoints for port 15020 and protocol TCP
.......
ame-consult-center-etcd does not have any active endpoints for port 25006 and protocol TCP
I0509 02:52:27.241807      17 controller.go:170] backend reload required
I0509 02:52:27.241816      17 util.go:67] rlimit.max=65536
I0509 02:52:27.241821      17 nginx.go:560] maximum number of open file descriptors : 7168
I0509 02:52:27.305661      17 nginx.go:661] NGINX configuration diff

[rikatz]

@0312birdzhang what's your kubernetes version?

[0312birdzhang]

@rikatz sorry it's my fault, the endpoints has been deleted, so the log is right.

[ChenyuanZ]

Hi here. I observed the issue in quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0. My Kubernetes version is v1.10.9-gke.5.

Ingress controller log:

I0121 22:10:29.344764       7 main.go:240] Running in Kubernetes cluster version v1.10+ (v1.10.9-gke.5) - git (clean) commit d776b4deeb3655fa4b8f4e8e7e4651d00c5f4a98 - platform linux/amd64
I0121 22:10:29.386501       7 main.go:101] Validated polyaxon/polyaxon-polyaxon-http-backend as the default backend.
I0121 22:10:30.392505       7 nginx.go:258] Starting NGINX Ingress controller
I0121 22:10:30.457125       7 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"polyaxon", Name:"polyaxon-polyaxon-ingress-config", UID:"5a4a5e0d-1dc9-11e9-83c0-42010a800260", APIVersion:"v1", ResourceVersion:"13351", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap polyaxon/polyaxon-polyaxon-ingress-config
I0121 22:10:31.514043       7 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"polyaxon", Name:"polyaxon-polyaxon-ingress", UID:"5ad20259-1dc9-11e9-83c0-42010a800260", APIVersion:"extensions/v1beta1", ResourceVersion:"13486", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress polyaxon/polyaxon-polyaxon-ingress
I0121 22:10:31.631037       7 leaderelection.go:187] attempting to acquire leader lease  polyaxon/ingress-controller-leader-polyaxon-ingress...
I0121 22:10:31.631418       7 nginx.go:279] Starting NGINX process
W0121 22:10:31.633066       7 controller.go:373] Service "polyaxon/polyaxon-polyaxon-http-backend" does not have any active Endpoint
W0121 22:10:31.633276       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:31.633521       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
I0121 22:10:31.633775       7 controller.go:172] Configuration changes detected, backend reload required.
I0121 22:10:31.643935       7 status.go:148] new leader elected: polyaxon-polyaxon-ingress-controller-555f798499-t4ckk
I0121 22:10:31.948223       7 controller.go:190] Backend successfully reloaded.
I0121 22:10:31.948498       7 controller.go:202] Initial sync, sleeping for 1 second.
[21/Jan/2019:22:10:32 +0000]TCP200000.000
W0121 22:10:35.350817       7 controller.go:373] Service "polyaxon/polyaxon-polyaxon-http-backend" does not have any active Endpoint
W0121 22:10:35.351764       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:35.351810       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:38.704066       7 controller.go:373] Service "polyaxon/polyaxon-polyaxon-http-backend" does not have any active Endpoint
W0121 22:10:38.731051       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:38.731161       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:42.017665       7 controller.go:373] Service "polyaxon/polyaxon-polyaxon-http-backend" does not have any active Endpoint
W0121 22:10:42.017717       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:42.017748       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:45.351979       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.
W0121 22:10:45.352048       7 controller.go:826] Service "polyaxon/polyaxon-polyaxon-api" does not have any active Endpoint.

Service spec:

apiVersion: v1
kind: Service
metadata:
  name: polyaxon-polyaxon-api
spec:
  externalTrafficPolicy: Cluster
  ports:
  - name: api
    nodePort: 30760
    port: 80
    protocol: TCP
    targetPort: 80
  - name: streams
    nodePort: 31901
    port: 1337
    protocol: TCP
    targetPort: 1337
  selector:
    app: polyaxon-polyaxon-api
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: {}

Ingress spec

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: polyaxon-polyaxon-ingress
spec:
  rules:
  - host: chart-example.local
    http:
      paths:
      - backend:
          serviceName: polyaxon-polyaxon-api
          servicePort: 80
        path: /
      - backend:
          serviceName: polyaxon-polyaxon-api
          servicePort: 1337
        path: /ws
status:
  loadBalancer: {}

BTW, I verified that Endpoints exists

$ kubectl get endpoints -l app=polyaxon-polyaxon-api
NAME                    ENDPOINTS                       AGE
polyaxon-polyaxon-api   x.x.x.x:1337,x.x.x.x:80         26m

Please advise how to resolve this issue. Thanks in advance!

[aledbf]

@ChenyuanZ if you use type: LoadBalancer in the service, you are creating a new cloud load balancer ($$$), please change that. Also, the service and the ingress must be present in the same namespace (polyaxon/polyaxon-polyaxon-api -> namespace: polyaxon, service: polyaxon-polyaxon-api)

[ChenyuanZ]

Hi @aledbf

I have tried both NodePort and ClusterIP. But the does not have any active endpoints issue exists in all cases.

The service and ingress are in the same namespace polyaxon as shown in the example below.

In the example below, I changed service from LoadBalancer to NodePort, and restarted ingress-controller. But the same error persisted.

$ kubectl get ingress --all-namespaces
NAMESPACE   NAME                        HOSTS                 ADDRESS         PORTS     AGE
polyaxon    polyaxon-polyaxon-ingress   chart-example.local   x.x.x.x   80        38m

kubectl get svc --all-namespaces -l app=polyaxon-polyaxon-api
NAMESPACE   NAME                    TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                       AGE
polyaxon    polyaxon-polyaxon-api   NodePort   x.x.x.x   <none>        80:30760/TCP,1337:31901/TCP   39m

[aledbf]

@ChenyuanZ please post the output kubectl get ep --all-namespaces -l app=polyaxon-polyaxon-api

[ChenyuanZ]

Hi @aledbf

$ kubectl get ep --all-namespaces -l app=polyaxon-polyaxon-api
NAMESPACE   NAME                    ENDPOINTS                       AGE
polyaxon    polyaxon-polyaxon-api   x.x.x.x:1337,x.x.x.x:80   47m

[aledbf]

@ChenyuanZ please open a new issue. Before that, please change the ingress controller deployment and add the flag --v=5 and post the log in the issue (create a gist to post the log).
Also, post the deployment of the ingress controller itself.

[ChenyuanZ]

Hi @aledbf. I've created an issue at #3687

[kariae]

Hi here, I have some Service "X" does not have any active Endpoint. also in the controller logs, I'm running nginx-ingress 0.24.1 my kubernetes version is v1.14.1, i'm running an empty nginx image to test everything, here is my deployment

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    run: nginx
  name: nginx-deploy-main
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx-main
  template:
    metadata:
      labels:
        run: nginx-main
    spec:
      containers:
      - image: nginx
        name: nginx

The service

apiVersion: v1
kind: Service
metadata:
  name: demo-example-com-nginx
spec:
  selector:
    app: nginx-deploy-main
  ports:
  - targetPort: http
    port: 80

and the ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: demo-example-com
  annotations:
    kubernetes.io/ingress.class: "nginx"
    certmanager.k8s.io/cluster-issuer: "letsencrypt-prod"
    certmanager.k8s.io/acme-challenge-type: http01
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  tls:
  - hosts:
    - demo.example.com
    secretName: letsencrypt-prod
  rules:
  - host: demo.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: demo-example-com-nginx
          servicePort: 80

The certificate is well generated, but when I try to access the url I have a 503 error with those logs in the controller

W0521 09:29:12.240234       6 controller.go:797] Service "default/demo-example-com-nginx" does not have any active Endpoint.

Any help, please? Thank you

[aledbf]

@kariae please use the kubectl plugin to get information about the ingresses https://kubernetes.github.io/ingress-nginx/kubectl-plugin/#ingresses
Also, you can use kubectl get ep demo-example-com-nginx to get the endpoints. If you don't see any, it means you have issues with your service selectors

[kariae]

@aledbf I have an endpoint demo-example-com-nginx <none> 40m, I'm installing the plugin to get more details

Edit:
I used the plugin to get the list of ingresses, I have it there but without endpoint:

INGRESS NAME      HOST+PATH          ADDRESSES   TLS   SERVICE                 SERVICE PORT   ENDPOINTS
demo-example-com   demo.example.com/               YES   demo-example-com-nginx   80             0

[aledbf]

That means you don't have a pod running for that service. You should see an IP address in that field

[kariae]

@aledbf thanks so much, I did mis-configure the selector in my service 😅