Cluster-api: Containers still run as root due to non-numeric user in Docker image

[invidian]

What steps did you take and what happened:

Deploying CAPI using clusterctl init on cluster with the following PSP:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: zz-minimal
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default,runtime/default'
    seccomp.security.alpha.kubernetes.io/defaultProfileName:  'docker/default'
spec:
  privileged: false
  # Required to prevent escalations to root.
  allowPrivilegeEscalation: false
  # This is redundant with non-root + disallow privilege escalation,
  # but we can provide it for defense in depth.
  requiredDropCapabilities:
  - KILL
  - MKNOD
  - SETUID
  - SETGID
  # Allow core volume types.
  volumes:
  - 'configMap'
  - 'emptyDir'
  - 'projected'
  - 'secret'
  - 'downwardAPI'
  # Assume that persistentVolumes set up by the cluster admin are safe to use.
  - 'persistentVolumeClaim'
  hostNetwork: false
  hostIPC: false
  hostPID: false
  runAsUser:
    # Require the container to run without root privileges.
    rule: 'MustRunAsNonRoot'
  seLinux:
    rule: 'RunAsAny'
  supplementalGroups:
    rule: 'MustRunAs'
    ranges:
    # Forbid adding the root group.
    - min: 1
      max: 65535
  fsGroup:
    rule: 'MustRunAs'
    ranges:
    # Forbid adding the root group.
    - min: 1
      max: 65535
  readOnlyRootFilesystem: false

Which enforces containers to run as non-root user still makes CAPI containers to not converge. This is despite having user defined in Dockerfile: https://github.com/kubernetes-sigs/cluster-api/blob/995597dc195324c5cd3ab95750f9eb9e8beda6c2/Dockerfile#L59

Events observed:

  Warning  Failed     10m (x5 over 11m)   kubelet            Error: container has runAsNonRoot and image has non-numeric user (nobody), cannot verify user is non-root
  Warning  Failed     10m (x6 over 11m)   kubelet            Error: container has runAsNonRoot and image will run as root

What did you expect to happen:
CAPI containers should not run as root and should converge.

Anything else you would like to add:
Downstream issue: https://github.com/kinvolk/lokomotive/issues/1265

Environment:

• Cluster-api version:
• Minikube/KIND version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]

[vincepri]

@shysank Do you have some time to look into this?

/milestone v0.4.0
/priority important-soon

We should also potentially backport if there is a fix, wdyt @CecileRobertMichon?

[CecileRobertMichon]

We should also potentially backport if there is a fix, wdyt @CecileRobertMichon Cecile Robert-Michon FTE?

Yes, let's first merge the fix in the main branch and, if it's non-breaking (I don't expect it would be), then we can consider a backport.

[shysank]

/assign

[shysank]

[vincepri]