Ingress-nginx: is there a way to run ingress-nginx as non root?

[TheKangaroo]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Feature Request

We manage an on-prem cluster for many diffrent customers, each of them in their own namespaces. We deploy a default nginx ingress for any of them to provide an easy onboarding process.
Now we have pod security policies in place where the customers (and therefore our default ingress in the namespaces too) are no longer allowed to run container as root.
I wonder if there is any way to start the nginx ingress pod without requiring root permissions.

NGINX Ingress controller version:

0.17.1

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T15:22:13Z", GoVersion:"go1.9.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:00:59Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: on-prem (VMware/baremetal)
• OS (e.g. from /etc/os-release): Container Linux by CoreOS 1745.7.0
• Kernel (e.g. uname -a):Linux k8s-master-01.example.com 4.14.48-coreos-r2 #1 SMP Thu Jun 14 08:23:03 UTC 2018 x86_64 Intel(R) Xeon(R) Gold 6152 CPU @ 2.10GHz GenuineIntel GNU/Linux
• Install tools: vanilla kubernetes deployed with ContainerLinux ignition files

What happened:
ingress won't start up with psp's in place disallowing run as root.
The pods throw:

Error: container has runAsNonRoot and image has non-numeric user (www-data), cannot verify user is non-root

What you expected to happen:

Any way to start nginx ingress even though I forbid run as root which is best practice for container use.

How to reproduce it (as minimally and precisely as possible):

deploy a psp with the following specs and bind it to the servicesaccounts in the ingress namespace:

apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
  name: default
spec:
  allowPrivilegeEscalation: false
  fsGroup:
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  requiredDropCapabilities:
  - ALL
  runAsUser:
    rule: MustRunAsNonRoot
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  volumes:
  - configMap
  - emptyDir
  - projected
  - secret
  - downwardAPI
  - persistentVolumeClaim

Anything else we need to know:

[aledbf]

@TheKangaroo this is not possible. Even worse, because we received multiple reports with AUFS and Containerd not supporting setcap (https://github.com/kubernetes/ingress-nginx/issues/2781) we are removing the securityContext in the next version too.

[TheKangaroo]

Hi @aledbf , as far as I can see, the only reason why we need to run as root is the binding to system ports 80 and 443, right?
What if we implement a check if uid == 0 and then decide if we bind to 80, 443 or 8080,8443?
This way we could support to run as non-root without a breaking change.
I'm not very familiar with the code, but would try to implement it and send a PR if this is an acceptable solution :)

[aiman-alsari]

This is possible using the www-data user. It has a user id of 33 with the appropriate permissions in /etc/nginx.

Here is a little diff on a default spec which should get it to work:

     spec:
       serviceAccountName: nginx-ingress-serviceaccount
       terminationGracePeriodSeconds: 60
+      securityContext:
+        runAsUser: 33
+        runAsGroup: 33
       containers:
       - image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.18.0
         name: ingress-controller
         ports:
-        - containerPort: 80
+        - containerPort: 8080
         livenessProbe:
           httpGet:
             path: /healthz
-            port: 80
+            port: 8080
         readinessProbe:
           httpGet:
             path: /healthz
-            port: 80
+            port: 8080
         env:
           - name: POD_NAME
             valueFrom:
@@ -38,8 +41,10 @@ spec:
                 fieldPath: metadata.namespace
         args:
         - /nginx-ingress-controller
         - --default-backend-service=$(POD_NAMESPACE)/default-backend
         - --watch-namespace=$(POD_NAMESPACE)
+        - --http-port=8080
+        - --https-port=8443

[aledbf]

@TheKangaroo since 0.18.0 we removed the root requirement and we now use authbind to allow bind to privileged ports.
Please check https://github.com/kubernetes/ingress-nginx/blob/master/deploy/mandatory.yaml#L254-L261

[TheKangaroo]

Thank you @aledbf and @aiman-alsari
this worked for me and solved my problem :)

P.s. @aledbf you have a typo in the release notes image path
quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.18.
there's a "0" missing at the end.