Ingress-nginx: unexpected error storing fake SSL Cert: could not create PEM certificate file /etc/ingress-controller/ssl/default-fake-certificate.pem: open /etc/ingress-controller/ssl/default-fake-certificate.pem: permission denied

[puzzledz]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
FEATURE REQUEST

NGINX Ingress controller version:
0.24.1

Kubernetes version (use kubectl version):
v1.14.1

Environment:

• Cloud provider or hardware configuration: vmware evironment
• OS (e.g. from /etc/os-release): ubuntu 18.04.1
• Kernel (e.g. uname -a): 4.15.0-47-generic
• Install tools:
• Others:

What happened:
I deploy ingress-nginx,then it is wrong
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-ingress-controller-5694ccb578-csn72 0/1 CrashLoopBackOff 8 20m 10.244.1.141 node2 <none> <none>

this is a part of log
W0503 09:23:37.549224 1 flags.go:214] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false) nginx version: nginx/1.15.6 W0503 09:23:37.559659 1 client_config.go:549] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0503 09:23:37.564858 1 main.go:205] Creating API client for https://10.96.0.1:443 I0503 09:23:37.618028 1 main.go:249] Running in Kubernetes cluster version v1.14 (v1.14.1) - git (clean) commit b7394102d6ef778017f2ca4046abbaa23b88c290 - platform linux/amd64 F0503 09:23:37.925586 1 main.go:121] unexpected error storing fake SSL Cert: could not create PEM certificate file /etc/ingress-controller/ssl/default-fake-certificate.pem: open /etc/ingress-controller/ssl/default-fake-certificate.pem: permission denied
What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know:
node1 is master, node2 is generic,I apply yaml file,then ingress-nginx pod was deploy in node2,it is relative to host permission?

[puzzledz]

i have checked it

    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 33
            runAsUser: 33
          env:
            - name: POD_NAME

[puzzledz]

root@node3:/home/z/kubeadm/ingress-nginx# kubectl get pods -n ingress-nginx                                                     NAME                                        READY   STATUS             RESTARTS   AGE
nginx-ingress-controller-5694ccb578-mzb4v   0/1     CrashLoopBackOff   6          6m57s

root@node3:/home/z/kubeadm/ingress-nginx# kubectl logs nginx-ingress-controller-5694ccb578-mzb4v -n ingress-nginx
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.24.1
  Build:      ce418168f
  Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------

W0515 14:31:45.066437       1 flags.go:214] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
nginx version: nginx/1.15.6
W0515 14:31:45.095746       1 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0515 14:31:45.096599       1 main.go:205] Creating API client for https://10.96.0.1:443
I0515 14:31:45.195979       1 main.go:249] Running in Kubernetes cluster version v1.14 (v1.14.1) - git (clean) commit b7394102d6ef778017f2ca4046abbaa23b88c290 - platform linux/amd64
F0515 14:31:46.356556       1 main.go:121] unexpected error storing fake SSL Cert: could not create PEM certificate file /etc/ingress-controller/ssl/default-fake-certificate.pem: open /etc/ingress-controller/ssl/default-fake-certificate.pem: permission denied

[NicolasT]

For what it's worth, I ran into the very same issue using containerd from EPEL on CentOS 7.6 (containerd-1.2.1-1.el7). Before that, I ran into an issue with nginx being denied to bind to 0.0.0.0:80 which I could resolve by running the process as UID 0.

All of this hinted at issues with ACLs or xattrs on the binary, the cert directory,... so I ran a Google query and bumped into https://github.com/containerd/containerd/issues/2942

Indeed, removing the images from the system, then upgrading to containerd-1.2.4-1.fc30 (it's a static Go binary after all...) made the controller container start just fine after (re)pulling the image.

So, if your environment is using containerd (could be as part of Docker? We don't run Docker, plain CRI to containerd) affected by that bug, you may want to upgrade and try again.

[puzzledz]

yes,I ran ingress-nginx on docker,but the containerd version is 1.2.5,what is CRI?Suggest to upgrade containerd to containerd-1.2.4-1.fc30?

root@node3:/home/z# containerd -v
containerd github.com/containerd/containerd 1.2.5 bb71b10fd8f58240ca47fbb579b9d1028eea7c84

[NicolasT]

I am not suggesting anything at all, just providing some more information for the devs to dig into this issue, if applicable.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mcambal]

In my case I used helm upgrade command without specifying the chart version, which caused that I used chart for nginx-ingress version 0.27.x with image version 0.26.2.

There is a breaking change in the default of runAsUser attribute due to migration to Alpine linux.

[dignajar]

Solution: Change runAsUser: 33 to runAsUser: 101.

[alter]

and this solution doesn't work with

helm ls |grep ingress
test-ingress    1           Tue Mar 17 14:08:06 2020    DEPLOYED    nginx-ingress-1.33.5        0.30.0      test

Check user:

k -n test get deployment.apps/test-ingress-nginx-ingress-controller -o yaml |grep runAsUser
runAsUser: 101

[muenchdo]

@alter can you double check that your ingress controller pods are running a version >= 0.27.0? I had the same issue but figured out that I had overwritten the image tag in my Helm values.

[Cultuzz-India]

I have the same problem with 0.32.0. The default runAsUser is already set to 101.

Interestingly, it works well in one node but not in another in the cluster. I'm running it in daemonset mode.

[kylegalea]

I have the same problem with 0.32.0. The default runAsUser is already set to 101.

Interestingly, it works well in one node but not in another in the cluster. I'm running it in daemonset mode.

Have the same issue,
Tried to changed imagepullpolicy to always to get new image but issue persisted.

I then proceeded to change the ingress controller version to 0.30.0, manually deleted the 0.32.0 image on the worker node and changed back the version to 0.32.0 Seems its working however lets see how long it will take.