Kubebuilder: Deploying manager pod as non-root

[harpratap]

I tried deploying the manager on our K8s platform that doesn't allow root by default but ran into following issues even though there's nothing really in the manager that requires root
I see the following two errors -

1. Error: container has runAsNonRoot and image will run as root
2. Error: container has runAsNonRoot and image has non-numeric user (nonroot), cannot verify user is non-root

We use the following podsecuritypolicy to ensure that pods are not running as root -

allowPrivilegeEscalation: true
  allowedCapabilities:
  - NET_RAW
  - NET_BIND_SERVICE
  fsGroup:
    rule: RunAsAny
  requiredDropCapabilities:
  - CHOWN
  - DAC_OVERRIDE
  - SETPCAP
  - MKNOD
  - AUDIT_WRITE
  - FOWNER
  - FSETID
  - KILL
  - SETGID
  - SETUID
  - SYS_CHROOT
  - SETFCAP
  - SYS_MODULE
  - SYS_RAWIO
  - SYS_PACCT
  - SYS_ADMIN
  - SYS_NICE
  - SYS_RESOURCE
  - SYS_TIME
  - SYS_TTY_CONFIG
  - AUDIT_CONTROL
  - MAC_OVERRIDE
  - MAC_ADMIN
  - NET_ADMIN
  - SYSLOG
  - DAC_READ_SEARCH
  - LINUX_IMMUTABLE
  - NET_BROADCAST
  - IPC_LOCK
  - IPC_OWNER
  - SYS_PTRACE
  - SYS_BOOT
  - LEASE
  - WAKE_ALARM
  - BLOCK_SUSPEND
  runAsUser:
    rule: MustRunAsNonRoot
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - configMap
  - emptyDir
  - projected
  - secret
  - downwardAPI
  - persistentVolumeClaim

I believe this is happening because of two reasons -

1. The kube-rbac-proxy uses root by default, since it is using distroless/static as it's base image - https://github.com/brancz/kube-rbac-proxy/blob/master/Dockerfile#L1 I got this from this comment - https://github.com/GoogleContainerTools/distroless/issues/235#issuecomment-422099277

1. The manager image is using nonroot:nonroot user - https://github.com/kubernetes-sigs/kubebuilder/blob/master/pkg/plugin/v2/scaffolds/internal/templates/dockerfile.go#L65

I have made some changes in #1635 which made it work in our cluster with the nonroot accounts.

What versions of software are you using? Specifically, the following are often useful:

• go version - 1.13
• kubebuilder version (kubebuilder version) and scaffolding version (check your PROJECT file) 2.3.1 and v2
• controller-runtime version (check your go.mod file) v0.6.0
• Kubernetes & kubectl versions (just run kubectl version against your API server) v1.18.6

/kind bug

[camilamacedo86]

Really thank you for raising it.

Wdyt about clone the kube-rbac-proxy project and build an image to be gen as the kb projects ones to test your operator project with a kube-rbac-proxy image that is nonroot and will use its user? Shows that we need to push it as a PR against kube-rbac-proxy as well.

Regards the changes in #1635 I think we need to check it deeper because: Is it supportable for all k8s versions? Is not it just valid for 1.18 see here? Would it affect the behaviour in order kinds of k8s cluster? How a project with will behaviour in OCP for example? Would it be the best approach to solve the problem for the majority of scenarios? Should it be introduced by default or just documented?

[harpratap]

Wdyt about clone the kube-rbac-proxy project and build an image to be gen as the kb projects ones to test your operator project with a kube-rbac-proxy image that is nonroot and will use its user?

After changing the base image to nonroot and using nonroot in kube-rbac-proxy the first error message does go away

Is it supportable for all k8s versions? Is not it just valid for 1.18 see here?

The pod.spec.securityContext.runAsUser API exists as far back as 1.12.x from what I can see

Would it affect the behaviour in order kinds of k8s cluster? How a project with will behaviour in OCP for example? Would it be the best approach to solve the problem for the majority of scenarios? Should it be introduced by default or just documented?

Personally I just found it a little confusing that the manager base image specified nonroot, so should have worked as-is on my nonroot cluster. Now it depends on whether kubebuilder wants to be secure by default at the cost of potentially more queries and issues. I think documentation on this will be needed in either case.

[camilamacedo86]

Hi @harpratap,

After changing the base image to nonroot and using nonroot in kube-rbac-proxy the first error message does go away

Could you please push the PR for kube-rbac-proxy and ping me on it? It is definitely something that we need to do in order to ensure security.

Personally I just found it a little confusing that the manager base image specified nonroot, so should have worked as-is on my nonroot cluster.

The container cannot have root access because it would be a strong security failure since containers are just processes and then, it means that would be easily possible have root access to the host.

Also, I understand that when we do not define the user the cluster it will assign a user with low permissions. Shows that if we do as proposed:

      securityContext:
        runAsUser: 1337

Then it grants more privileged pods when the goal is to be less privileged as possible by default. See the OCP doc. Regards this point in specific I think we should just add by default it if it would be a security recommendation to do be done always which shows not the case. However, please feel free to let me know if I am missing something or if you find any Kubernetes documentation which says the opposite.

[harpratap]

@camilamacedo86 Makes sense.
Found the correct UID:GID to be used from here

The change on kube-rbac-proxy is here - https://github.com/brancz/kube-rbac-proxy/pull/85
And I guess similar change should be made on kubebuilder manager too so we can get rid of having to change the security context?

[camilamacedo86]

Hi @harpratap,

Thank you for your reply. See that we should not add the fixed ID. If you look we are using the variable already (we might could remove the user indeed, it is required to check):

# Use distroless as minimal base image to package the manager binary
# Refer to https://github.com/GoogleContainerTools/distroless for more details
FROM gcr.io/distroless/static:nonroot
WORKDIR /
COPY --from=builder /workspace/manager .
USER nonroot:nonroot # See here

ENTRYPOINT ["/manager"]

Also, for further information check https://github.com/GoogleContainerTools/distroless/issues/235#issuecomment-503746132

In this way, what in POV we need to close this task:

1. Error: container has runAsNonRoot and image will run as root

We need to collab to the brancz/kube-rbac-proxy in order to this project provide an image that is distroless and rootless which means that it will be something such as:

FROM gcr.io/distroless/static:nonroot
 ...
USER nonroot:nonroot 
..
ENTRYPOINT ["/kube-rbac-proxy"]

1. Error: container has runAsNonRoot and image has non-numeric user (nonroot), cannot verify user is non-root

We need to update your PR against master in Kubebuidler to use :

  securityContext:  
      ...
    runAsNonRoot: true

As described in: https://kubernetes.io/blog/2016/08/security-best-practices-kubernetes-deployment/

[harpratap]

@camilamacedo86 Thanks for your changes!

Regarding securityContext.runAsNonRoot it automatically defaults to true in case the podSecurityPolicy defines MustRunAsNonRoot as mentioned here

MustRunAsNonRoot - Requires that the pod be submitted with a non-zero runAsUser or have the USER directive defined (using a numeric UID) in the image. Pods which have specified neither runAsNonRoot nor runAsUser settings will be mutated to set runAsNonRoot=true, thus requiring a defined non-zero numeric USER directive in the container. No default provided.

So if we are defining securityContext.runAsUser, runAsNonRoot is redundant

[camilamacedo86]

HI @harpratap,

Really thank you for your collaboration and patient with. See that I could finish this one: https://github.com/brancz/kube-rbac-proxy/pull/86 and I added the comments for we address your suggestion here in Kube as well. See; https://github.com/kubernetes-sigs/kubebuilder/pull/1635.

It is great and you had an excellent spot on this one.

[harpratap]

@camilamacedo86 Thanks so much!

[bsctl]

@camilamacedo86 Thanks for working on this. Waiting for update to a rootless image.

[camilamacedo86]

I am closing this one because all changes and fixes required were done. We only need to know is have an kube-rbac-proxy image that we can use in the projects. It is tracked already in https://github.com/kubernetes-sigs/kubebuilder/issues/1785.

As workaround users are able to update their projects manually to use the images provided by https://github.com/brancz/kube-rbac-proxy directly.