Ingress-nginx: Nginx ingress controller - CrashLoopBackOff - no errors

Created on 19 Jul 2018 · 4Comments · Source: kubernetes/ingress-nginx

NGINX Ingress controller version:
0.17.1 (tried from 0.14.0 and later with no difference)

Kubernetes version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0-rancher1", GitCommit:"e27e2f5a930cfc136b28aa7d509b1d1c8ecb13d7", GitTreeState:"clean", BuildDate:"2018-03-29T00:13:11Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Environment

Cloud provider = Rancher
OS = CentOS Linux release 7.5.1804 (Core)
Kernel = 3.10.0-862.3.2.el7.x86_64

What happened

Unable to run nginx ingress controller, after running the mandatory.yml, it results to "CrashLoopBackOff"
I cannot get any specific error of why it fails.

kubectl --namespace=ingress-nginx logs nginx-ingress-controller-d774fd595-swvfq
NGINX Ingress controller
Release:    0.17.1
Build:      git-12f7966
Repository: https://github.com/kubernetes/ingress-nginx.git

 nginx version: nginx/1.13.12
 W0719 18:02:29.969473       7 client_config.go:552] Neither --kubeconfig nor --master was specified.                    Using the inClusterConfig.  This might not work.
  I0719 18:02:29.969898       7 main.go:191] Creating API client for https://10.43.0.1:443`

kubectl  --namespace=ingress-nginx get pods
NAME                                       READY     STATUS             RESTARTS   AGE
default-http-backend-5c6d95c48-hxktx       1/1       Running            0          3m
nginx-ingress-controller-d774fd595-swvfq   0/1       CrashLoopBackOff   4          3m

Anything else we need to know:
Increasing the log verbosity will not produce any more output. Would I appreciate if anyone has any ideas or hints how to find out what is wrong :)

Source

alogoc

Most helpful comment

@aledbf yes were right! Indeed there was a connectivity issue with the internal kubernetes network in the end so it seems that nginx pod couldn't reach the kubernetes API, that explains everything.

In case this helps somebody in the future, I was testing this deploymet on Rancher, but apparently rancher implements iptables rules on the fly, and after restarting the network on some nodes they lost connectivity to the kube-api service. Simple reboot fixed the issue, nginx works fine :)

alogoc on 21 Jul 2018

👍7

All 4 comments

@alogoc please use --v=10 in the deployment. If you still see CrashLoopBackOff please remove the probes from the deployment. If you don't see more output from the logs it means you have some connectivity/firewall issue

aledbf on 19 Jul 2018

@aledbf I did use the --v=10 but I don't see any useful information on the output

  I0720 09:31:55.745448   49031 round_trippers.go:411] Response Headers:
  I0720 09:31:55.745459   49031 round_trippers.go:414]     Content-Type: application/json
  I0720 09:31:55.745468   49031 round_trippers.go:414]     Date: Fri, 20 Jul 2018 09:31:55 GMT
  I0720 09:31:55.745477   49031 round_trippers.go:414]     Content-Length: 501
  I0720 09:31:55.745537   49031 request.go:874] Response Body: {"kind":"Namespace","apiVersion":"v1","metadata":{"name":"ingress-nginx","selfLink":"/api/v1/namespaces/ingress-nginx","uid":"bd02944c-8bff-11e8-b62c-02e079376de8","resourceVersion":"10093727","creationTimestamp":"2018-07-20T09:31:53Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"name\":\"ingress-nginx\",\"namespace\":\"\"}}\n"}},"spec":{"finalizers":["kubernetes"]},"status":{"phase":"Active"}}
  I0720 09:31:55.745806   49031 request.go:874] Request Body: {"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"extensions/v1beta1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{},\"name\":\"nginx-ingress-controller\",\"namespace\":\"ingress-nginx\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"ingress-nginx\"}},\"template\":{\"metadata\":{\"annotations\":{\"prometheus.io/port\":\"10254\",\"prometheus.io/scrape\":\"true\"},\"labels\":{\"app\":\"ingress-nginx\"}},\"spec\":{\"containers\":[{\"args\":[\"/nginx-ingress-controller\",\"--default-backend-service=$(POD_NAMESPACE)/default-http-backend\",\"--configmap=$(POD_NAMESPACE)/nginx-configuration\",\"--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services\",\"--udp-services-configmap=$(POD_NAMESPACE)/udp-services\",\"--publish-service=$(POD_NAMESPACE)/ingress-nginx\",\"--annotations-prefix=nginx.ingress.kubernetes.io\"],\"env\":[{\"name\":\"POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"POD_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.namespace\"}}}],\"image\":\"quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.17.1\",\"livenessProbe\":{\"failureThreshold\":3,\"httpGet\":{\"path\":\"/healthz\",\"port\":10254,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":10,\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"name\":\"nginx-ingress-controller\",\"ports\":[{\"containerPort\":80,\"name\":\"http\"},{\"containerPort\":443,\"name\":\"https\"}],\"readinessProbe\":{\"failureThreshold\":3,\"httpGet\":{\"path\":\"/healthz\",\"port\":10254,\"scheme\":\"HTTP\"},\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"securityContext\":{\"capabilities\":{\"add\":[\"NET_BIND_SERVICE\"],\"drop\":[\"ALL\"]},\"runAsUser\":33}}],\"serviceAccountName\":\"nginx-ingress-serviceaccount\"}}}}\n"},"name":"nginx-ingress-controller","namespace":"ingress-nginx"},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"ingress-nginx"}},"template":{"metadata":{"annotations":{"prometheus.io/port":"10254","prometheus.io/scrape":"true"},"labels":{"app":"ingress-nginx"}},"spec":{"containers":[{"args":["/nginx-ingress-controller","--default-backend-service=$(POD_NAMESPACE)/default-http-backend","--configmap=$(POD_NAMESPACE)/nginx-configuration","--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services","--udp-services-configmap=$(POD_NAMESPACE)/udp-services","--publish-service=$(POD_NAMESPACE)/ingress-nginx","--annotations-prefix=nginx.ingress.kubernetes.io"],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}],"image":"quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.17.1","livenessProbe":{"failureThreshold":3,"httpGet":{"path":"/healthz","port":10254,"scheme":"HTTP"},"initialDelaySeconds":10,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"name":"nginx-ingress-controller","ports":[{"containerPort":80,"name":"http"},{"containerPort":443,"name":"https"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/healthz","port":10254,"scheme":"HTTP"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"securityContext":{"capabilities":{"add":["NET_BIND_SERVICE"],"drop":["ALL"]},"runAsUser":33}}],"serviceAccountName":"nginx-ingress-serviceaccount"}}}}
  I0720 09:31:55.745877   49031 round_trippers.go:386] curl -k -v -XPOST  -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: kubectl/v1.10.3 (linux/amd64) kubernetes/2bba012" -H "Authorization: Bearer QmFzaWMgUlRkRE5qUTRNemszUWpRMlJEVXlPVE13UVRrNlNtMWpkMFJPY25veU5qUTFPVFZ6TVhaTVF6bE5RVWh6YlZSNWFtTXlZbXR6Wm0xUldtcFRVdz09" https://194.42.88.157:8080/r/projects/1a7/kubernetes:6443/apis/extensions/v1beta1/namespaces/ingress-nginx/deployments
  I0720 09:31:55.843022   49031 round_trippers.go:405] POST https://194.42.88.157:8080/r/projects/1a7/kubernetes:6443/apis/extensions/v1beta1/namespaces/ingress-nginx/deployments 201 Created in 97 milliseconds
  I0720 09:31:55.843051   49031 round_trippers.go:411] Response Headers:
  I0720 09:31:55.843058   49031 round_trippers.go:414]     Content-Type: application/json
  I0720 09:31:55.843063   49031 round_trippers.go:414]     Date: Fri, 20 Jul 2018 09:31:55 GMT
  I0720 09:31:55.843164   49031 request.go:874] Response Body: {"kind":"Deployment","apiVersion":"extensions/v1beta1","metadata":{"name":"nginx-ingress-controller","namespace":"ingress-nginx","selfLink":"/apis/extensions/v1beta1/namespaces/ingress-nginx/deployments/nginx-ingress-controller","uid":"be249a41-8bff-11e8-b62c-02e079376de8","resourceVersion":"10093763","generation":1,"creationTimestamp":"2018-07-20T09:31:55Z","labels":{"app":"ingress-nginx"},"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"extensions/v1beta1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{},\"name\":\"nginx-ingress-controller\",\"namespace\":\"ingress-nginx\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"ingress-nginx\"}},\"template\":{\"metadata\":{\"annotations\":{\"prometheus.io/port\":\"10254\",\"prometheus.io/scrape\":\"true\"},\"labels\":{\"app\":\"ingress-nginx\"}},\"spec\":{\"containers\":[{\"args\":[\"/nginx-ingress-controller\",\"--default-backend-service=$(POD_NAMESPACE)/default-http-backend\",\"--configmap=$(POD_NAMESPACE)/nginx-configuration\",\"--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services\",\"--udp-services-configmap=$(POD_NAMESPACE)/udp-services\",\"--publish-service=$(POD_NAMESPACE)/ingress-nginx\",\"--annotations-prefix=nginx.ingress.kubernetes.io\"],\"env\":[{\"name\":\"POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"POD_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.namespace\"}}}],\"image\":\"quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.17.1\",\"livenessProbe\":{\"failureThreshold\":3,\"httpGet\":{\"path\":\"/healthz\",\"port\":10254,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":10,\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"name\":\"nginx-ingress-controller\",\"ports\":[{\"containerPort\":80,\"name\":\"http\"},{\"containerPort\":443,\"name\":\"https\"}],\"readinessProbe\":{\"failureThreshold\":3,\"httpGet\":{\"path\":\"/healthz\",\"port\":10254,\"scheme\":\"HTTP\"},\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"securityContext\":{\"capabilities\":{\"add\":[\"NET_BIND_SERVICE\"],\"drop\":[\"ALL\"]},\"runAsUser\":33}}],\"serviceAccountName\":\"nginx-ingress-serviceaccount\"}}}}\n"}},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"ingress-nginx"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"ingress-nginx"},"annotations":{"prometheus.io/port":"10254","prometheus.io/scrape":"true"}},"spec":{"containers":[{"name":"nginx-ingress-controller","image":"quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.17.1","args":["/nginx-ingress-controller","--default-backend-service=$(POD_NAMESPACE)/default-http-backend","--configmap=$(POD_NAMESPACE)/nginx-configuration","--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services","--udp-services-configmap=$(POD_NAMESPACE)/udp-services","--publish-service=$(POD_NAMESPACE)/ingress-nginx","--annotations-prefix=nginx.ingress.kubernetes.io"],"ports":[{"name":"http","containerPort":80,"protocol":"TCP"},{"name":"https","containerPort":443,"protocol":"TCP"}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}}],"resources":{},"livenessProbe":{"httpGet":{"path":"/healthz","port":10254,"scheme":"HTTP"},"initialDelaySeconds":10,"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/healthz","port":10254,"scheme":"HTTP"},"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"add":["NET_BIND_SERVICE"],"drop":["ALL"]},"runAsUser":33}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"nginx-ingress-serviceaccount","serviceAccount":"nginx-ingress-serviceaccount","securityContext":{},"schedulerName":"default-scheduler"}},"strategy":{"type":"RollingUpdate","rollingUpdate":{"maxUnavailable":1,"maxSurge":1}},"revisionHistoryLimit":10,"progressDeadlineSeconds":600},"status":{}}
  deployment.extensions "nginx-ingress-controller" created

After removing the probes, the status is now Running but it keeps restarting. I also tried to hardcode the variables $POD_NAMESPACE but it made no difference. I am exposing nginx pods with CluserIP service, through the internal network so firewall is not a problem.

One thing I noticed is that nginx.conf is completely empty. Seems that it doesn't pick up the configuration from the configmap, neither from the ingresses which are created? There is nothing coming out of access.log or error.log.

That said even if I curl nginx service through the internal kubernetes network I still do not see any anything on the logs apart from what I pasted on previous post.
The exit code 255 seems very generic it doesn't help much...

 docker ps -a | grep -i nginx    WARNING: Error loading config file:/root/.docker/config.json - read /root/.docker/config.json: is a directory    
 0a5d6b7f77af        
 8410cbcd825d                                                                                                                                      
 "/usr/bin/dumb-ini..."   About a minute ago   Up 59 seconds                                         
 k8s_nginx-ingress-controller_nginx-ingress-controller-6f74bf857c-8v7kf_ingress-nginx_f1ed0968-8c1b-11e8-b62c-02e079376de8_4    57c2e2f009e1        
 8410cbcd825d                                                                                                                                      
 "/usr/bin/dumb-ini..."   8 minutes ago        Exited (255) About a minute ago                       
 k8s_nginx-ingress-controller_nginx-ingress-controller-6f74bf857c-8v7kf_ingress-nginx_f1ed0968-8c1b-11e8-b62c-02e079376de8_3

www-data@nginx-ingress-controller-6f74bf857c-8v7kf:/var/log/nginx$ curl -v localhost:80
* Rebuilt URL to: localhost:80/
*   Trying ::1...
* TCP_NODELAY set
* connect to ::1 port 80 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* connect to 127.0.0.1 port 80 failed: Connection refused
* Failed to connect to localhost port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 80: Connection refused
www-data@nginx-ingress-controller-6f74bf857c-8v7kf:/var/log/nginx$ curl -v localhost:443
* Rebuilt URL to: localhost:443/
*   Trying ::1...
* TCP_NODELAY set
* connect to ::1 port 443 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* connect to 127.0.0.1 port 443 failed: Connection refused
* Failed to connect to localhost port 443: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 443: Connection refused
www-data@nginx-ingress-controller-6f74bf857c-8v7kf:/var/log/nginx$ command terminated with exit code 137

That's awkward as this is the only deployment in the cluster that has such issues with the internal network, not sure what else to check...

Does it expect any specific kernel modules to be in place?

alogoc on 20 Jul 2018

@alogoc if the ingress controller cannot reach the apiserver the configuration file will be empty. I suggest you run kubectl exec <ingress pod> -- curl -k -v https://10.43.0.1:443 but again, you have some connectivity issue between the pod and the apiserver

aledbf on 20 Jul 2018

👍2

alogoc on 21 Jul 2018

👍7

Was this page helpful?

0 / 5 - 0 ratings