Kind: Cluster created won't work properly if PSP is enabled

[brightzheng100]

What happened:
Enabled PSP by using kind-config.yaml in:

kind create cluster --name kind-cluster --config kind-cofig.yaml --loglevel debug

Everything looked good but the CNI and cube-proxy won't work:

kubectl get ds,deploy,po -n kube-system
NAME                              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
daemonset.extensions/kindnet      4         4         0       4            0           <none>                        31m
daemonset.extensions/kube-proxy   0         0         0       0            0           beta.kubernetes.io/os=linux   31m

NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.extensions/coredns   0/2     0            0           31m

NAME                READY   STATUS    RESTARTS   AGE
pod/kindnet-69ddz   0/1     Blocked   0          30m
pod/kindnet-cbc8k   0/1     Blocked   0          31m
pod/kindnet-ps6pk   0/1     Blocked   0          30m
pod/kindnet-xphlc   0/1     Blocked   0          30m

What you expected to happen:

It should detect it and apply properly PSP objects preferably, or at least print some suggestions or simply error out with helpful messages.

How to reproduce it (as minimally and precisely as possible):

$ cat > kind-cofig.yaml <<EOF
# this config file contains all config fields with comments
kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
# patch the generated kubeadm config with some extra settings
kubeadmConfigPatches:
- |
  apiVersion: kubeadm.k8s.io/v1beta2
  kind: ClusterConfiguration
  metadata:
    name: config
  apiServer:
    extraArgs:
      enable-admission-plugins: NodeRestriction,PodSecurityPolicy
      "feature-gates": "DynamicAuditing=true"
  scheduler:
    extraArgs:
      "feature-gates": "DynamicAuditing=true"
  controllerManager:
    extraArgs:
      "feature-gates": "DynamicAuditing=true"
- |
  apiVersion: kubeadm.k8s.io/v1beta2
  kind: InitConfiguration
  metadata:
    name: config
  nodeRegistration:
    kubeletExtraArgs:
      "feature-gates": "DynamicAuditing=true"
# 1 control plane node and 3 workers
nodes:
# the control plane node config
- role: control-plane
# the three workers
- role: worker
- role: worker
- role: worker
EOF

$ kind create cluster --name kind-cluster --config kind-cofig.yaml --loglevel debug

Anything else we need to know?:

Environment:

• kind version: (use kind version): v0.5.1

• Kubernetes version: (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:18:23Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
  Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-20T18:57:36Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

• Docker version: (use docker info):
  Client:
  Debug Mode: false

Server:
Containers: 11
Running: 8
Paused: 0
Stopped: 3
Images: 5
Server Version: 19.03.2
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.184-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 1.951GiB
Name: docker-desktop
ID: PIZU:RIW4:PI73:4GL4:YXJ5:3R5D:FQ4X:QJSG:2O6M:XRR6:GF7B:EXHH
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 48
Goroutines: 59
System Time: 2019-10-20T05:06:54.583473233Z
EventsListeners: 2
HTTP Proxy: gateway.docker.internal:3128
HTTPS Proxy: gateway.docker.internal:3129
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

• OS (e.g. from /etc/os-release): MacBook Pro

[BenTheElder]

PSP in kind docker nodes is not really possible on arbitrary hosts unfortunately -- it depends on the host kernel to have seccomp / apparmor 😞

That said: the issue here is likely that a default PSP needs to be installed, you can't just enable the admission controller and kubeadm doesn't handle this out of the box currently. If you install one it will probably work.

See #725 for more discussion. We're tracking the trade offs of having a higher level option around PSP there.

It should detect it and apply properly PSP objects preferably, or at least print some suggestions or simply error out with helpful messages.

Config patches are a sort of open "backdoor" to try advanced features beyond what kind natively supports. We will try to use any patch but it may not work and make no guarantees. These are for power users to experiment.

Closing as patches are best effort and the PSP being enabled not working is a kubeadm flaw imo.

We have an ongoing effort to document the config more in https://github.com/kubernetes-sigs/kind/pull/832

[BenTheElder]

cc @neolit123 I can't find anything official documenting the intersection of kubeadm and PSP, would this be a bug k8s.io/kubeadm would be willing to handle? If so I will file one.

[neolit123]

@brightzheng100

Everything looked good but the CNI and cube-proxy won't work:

possibly your policy is not configured correctly.

/triage support
/remove kind-bug

It should detect it and apply properly PSP objects preferably, or at least print some suggestions or simply error out with helpful messages.

that's a kubernetes problem kubeadm or kind cannot help much there.
also please mind that PSP is being deprecated in the future as per discussions by sig-auth.

  enable-admission-plugins: NodeRestriction,PodSecurityPolicy

adding PSP to a already running kubeadm cluster works fine.
try that instead of starting the cluster with PSP right away.

see a tutorial here:
https://youtu.be/zErhwjPRKn8?t=2066

pod/kindnet-69ddz 0/1 Blocked 0 30m

once you enable the PSP controller your api-server will not restart.
you need to apply your correct policies that will unblock it.
try restarting pods to see if the policy blocks them.

@BenTheElder

That said: the issue here is likely that a default PSP needs to be installed, you can't just enable the admission controller and kubeadm doesn't handle this out of the box currently. If you install one it will probably work.

the recommended steps are - 1) install your PSPs, 2) add the controller to the api-server manifest.
its unlikely that kubeadm will install a default PSP policy given the complications around its UX and given it will be deprecated in the feature.

cc @neolit123 I can't find anything official documenting the intersection of kubeadm and PSP, would this be a bug k8s.io/kubeadm would be willing to handle? If so I will file one.

we consider it an advanced feature that power users already know how to setup.
there is nothing special on the kubeadm side.

[BenTheElder]

ACK on deprecation, forgot about that...

[brightzheng100]

I think it's not because of kubeadm, it's all about the way how we provision addons -- in this case it's about kindnet and kube-proxy.

We may consider two potential approaches:
1. Make them patch-able and as static pods (in /etc/kubernetes/maninfests/), like what we do for below components;
- kube-apiserver.yaml
- kube-controller-manager.yaml
- kube-scheduler.yaml
2. Or simply leave them to user who can install them after kind bootstrapping and provisioning.

What do you think?

[aojea]

@neolit123 where can I find more info about this?

also please mind that PSP is being deprecated in the future as per discussions by sig-auth.

[BenTheElder]

I think it's not because of kubeadm, it's all about the way how we provision addons

From your example, a default PSP is not configured though?

kindnet already has an appropriate PSP for itself bundled, for the sake of completeness

Make them patch-able and as static pods (in /etc/kubernetes/maninfests/), like what we do for below components;

they're daemonsets, you can just patch the daemonset.

but also:

also please mind that PSP is being deprecated in the future as per discussions by sig-auth.

[BenTheElder]

@aojea https://github.com/kubernetes/enhancements/issues/5

[alexellis]

@neolit123 where is the official / latest guidance stating that PSP will definitely / or very likely be deprecated? I want to cover PSP at KubeCon, but I'm struggling to find that info. cc @stefanprodan

[neolit123]

@alexellis the only source of this information is the following meeting where "fixing PSP in a new interation" was the less preferred option: https://www.youtube.com/watch?v=WP0-uR-BQC0