Kubeadm: 1.6.0 broke selinux again

[coeki]

BUG REPORT

Versions

kubeadm version (use kubeadm version):

kubeadm version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-03-28T16:24:30Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-03-28T16:36:33Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-03-28T16:24:30Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

What happened?

On the master, where you run kubeadm init, setenforce 0 is still needed as it will be denied to read the certs.

ausearch -m avc

```time->Thu Mar 30 16:47:22 2017

type=AVC msg=audit(1490892442.272:1130): avc: denied { open } for pid=5137 comm="kube-controller" path="/etc/kubernetes/pki/sa.key" dev="sda1" ino=1573694 scontext=system_u:system_r:container_t:s0:c483,c527 tcontext=unconfined_u:object_r:cert_t:s0 tclass=file permissive=1

time->Thu Mar 30 16:47:49 2017

type=AVC msg=audit(1490892469.704:1170): avc: denied { read } for pid=7452 comm="kube-apiserver" name="apiserver.crt" dev="sda1" ino=1573691 scontext=system_u:system_r:container_t:s0:c11,c526 tcontext=unconfined_u:object_r:cert_t:s0 tclass=file permissive=1

time->Thu Mar 30 16:47:49 2017

type=AVC msg=audit(1490892469.704:1171): avc: denied { open } for pid=7452 comm="kube-apiserver" path="/etc/kubernetes/pki/apiserver.crt" dev="sda1" ino=1573691 scontext=system_u:system_r:container_t:s0:c11,c526 tcontext=unconfined_u:object_r:cert_t:s0 tclass=file permissive=1

time->Thu Mar 30 16:47:52 2017

type=AVC msg=audit(1490892472.814:1173): avc: denied { open } for pid=7326 comm="kube-controller" path="/etc/kubernetes/pki/sa.key" dev="sda1" ino=1573694 scontext=system_u:system_r:container_t:s0:c391,c896 tcontext=unconfined_u:object_r:cert_t:s0 tclass=file permissive=1

time->Thu Mar 30 16:47:52 2017
type=AVC msg=audit(1490892472.814:1172): avc: denied { read } for pid=7326 comm="kube-controller" name="sa.key" dev="sda1" ino=1573694 scontext=system_u:system_r:container_t:s0:c391,c896 tcontext=unconfined_u:object_r:cert_t:s0 tclass=file permissive=1
```

What you expected to happen?

Seems similar to #107, and expected to run with setenforce 1

How to reproduce it (as minimally and precisely as possible)?

Every time

Anything else we need to know?

[timothysc]

/cc @dgoodwin @pmorie

[dgoodwin]

Not able to reproduce immediately with a fully up to date CentOS 7 vm, how does this compare to your environment @coeki?

(root@centos1 ~) $ rpm -qa | grep selinux
libselinux-python-2.5-6.el7.x86_64
selinux-policy-3.13.1-102.el7_3.16.noarch
libselinux-2.5-6.el7.x86_64
libselinux-utils-2.5-6.el7.x86_64
selinux-policy-targeted-3.13.1-102.el7_3.16.noarch
container-selinux-2.9-4.el7.noarch

(root@centos1 ~) $ rpm -qa | grep kube
kubernetes-cni-0.5.1-0.x86_64
kubelet-1.6.1-0.x86_64
kubectl-1.6.1-0.x86_64
kubeadm-1.6.1-0.x86_64

(root@centos1 ~) $ kubeadm version
kubeadm version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

(root@centos1 ~) $ ls -lZ /etc/kubernetes/pki
-rw-r--r--. root root unconfined_u:object_r:cert_t:s0  apiserver.crt
-rw-------. root root unconfined_u:object_r:cert_t:s0  apiserver.key
-rw-r--r--. root root unconfined_u:object_r:cert_t:s0  apiserver-kubelet-client.crt
-rw-------. root root unconfined_u:object_r:cert_t:s0  apiserver-kubelet-client.key
-rw-r--r--. root root unconfined_u:object_r:cert_t:s0  ca.crt
-rw-------. root root unconfined_u:object_r:cert_t:s0  ca.key
-rw-r--r--. root root unconfined_u:object_r:cert_t:s0  front-proxy-ca.crt
-rw-------. root root unconfined_u:object_r:cert_t:s0  front-proxy-ca.key
-rw-r--r--. root root unconfined_u:object_r:cert_t:s0  front-proxy-client.crt
-rw-------. root root unconfined_u:object_r:cert_t:s0  front-proxy-client.key
-rw-------. root root unconfined_u:object_r:cert_t:s0  sa.key
-rw-------. root root unconfined_u:object_r:cert_t:s0  sa.pub

kubeadm init completed successfully for me and no denials in audit.log.

[coeki]

@dgoodwin Sorry, since I use vagrant, and some ansible, that particular environment is long gone, which was based on the just released kubeadm/kubernetes 1.6.0 (not alpha).
But I set my ansible script to just setenforce 0, since I thought it was broken, haven't tested it proper again.

I have tested it just now with a fresh environment, an updated centos7, and kubeadm/kubernetes 1.6.1, and can confirm it works with setenforce 1.

So I'll close this one out, it must have been a fluke.

Thanks for having a look.

[coeki]

Hi @dgoodwin So I refactored my vagrant dev pipeline, and it broke on fedora 25 (updated specs below), which is weird since the rpm's are build on fedora. Tested with weave plugin.

centos/7 (updated):

```[root@Centos1 vagrant]# rpm -qa |grep selinux
libselinux-2.5-6.el7.x86_64
libselinux-utils-2.5-6.el7.x86_64
container-selinux-2.10-2.el7.noarch
selinux-policy-3.13.1-102.el7_3.16.noarch
selinux-policy-targeted-3.13.1-102.el7_3.16.noarch
libselinux-python-2.5-6.el7.x86_64

fedora 25 (updated):

libselinux-python-2.5-13.fc25.x86_64
selinux-policy-targeted-3.13.1-225.13.fc25.noarch
libselinux-utils-2.5-13.fc25.x86_64
libselinux-python3-2.5-13.fc25.x86_64
rpm-plugin-selinux-4.13.0.1-1.fc25.x86_64
container-selinux-2.10-1.fc25.noarch
selinux-policy-3.13.1-225.13.fc25.noarch
libselinux-2.5-13.fc25.x86_64

docker inspect on Centos/7:

[root@Centos1 vagrant]# docker inspect 9f79beb28ce4 |grep -i -A 6 'mountlabel\|processlabel\|SecurityOpt'
"MountLabel": "system_u:object_r:svirt_sandbox_file_t:s0:c27,c200",
"ProcessLabel": "system_u:system_r:svirt_lxc_net_t:s0:c27,c200",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/etc/kubernetes:/etc/kubernetes/:ro",

"/etc/ssl/certs:/etc/ssl/certs",

        "SecurityOpt": [
            "seccomp=unconfined",
            "label=user:system_u",
            "label=role:system_r",
            "label=type:svirt_lxc_net_t",
            "label=level:s0:c27,c200"

        ],

on fedora 25:

[root@master-01 vagrant]# docker inspect 362a29628561 |grep -i -A 6 'mountlabel\|processlabel\|SecurityOpt'
"MountLabel": "system_u:object_r:container_file_t:s0:c815,c881",
"ProcessLabel": "system_u:system_r:container_t:s0:c815,c881",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/etc/kubernetes:/etc/kubernetes/:ro",
"/etc/ssl/certs:/etc/ssl/certs",

            "SecurityOpt": [
                "seccomp=unconfined",
                "label=user:system_u",
                "label=role:system_r",
                "label=type:container_t",
                "label=level:s0:c815,c881"
            ],

after adjusting, so everything with volume mount spc_t:

[root@master-01 vagrant]# docker inspect 8cc809a58e0c |grep -i -A 6 'mountlabel\|processlabel\|SecurityOpt' "MountLabel": "system_u:object_r:container_file_t:s0:c81,c416",
"ProcessLabel": "system_u:system_r:spc_t:s0:c81,c416",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/etc/kubernetes:/etc/kubernetes/:ro",

"/etc/ssl/certs:/etc/ssl/certs",

        "SecurityOpt": [
            "label=type:spc_t",
            "seccomp=unconfined",
            "label=user:system_u",
            "label=role:system_r",
            "label=type:spc_t",
            "label=level:s0:c81,c416"

So why does it work on Centos/7 and not on fedora 25 out of the box? It's ridiculous, the rpm's are build on fedora, centos and fedora should have similar selinux policies, and I think in general hostPath and selinux clash whatever. Sorry to rant but his is just not ok.

cc @rhatdan @dgoodwin @jbeda @luxas

[dgoodwin]

I can understand the frustration as this has seemingly broke out from underneath us several times now, but first lets clarify exactly what broke? Are there denials in /var/log/audit/audit.log? Are there any other errors in docker's journald? What containers were running at the time? (docker ps) Were any containers failing, (docker ps -a) and if so, what was in their docker logs?

I tried to reproduce:

• vagrant init fedora/25-cloud-base; vagrant up --provider libvirt
• full dnf update and reboot
• setenforce 1
• follow kubeadm instructions: https://kubernetes.io/docs/getting-started-guides/kubeadm/ (minus the setenforce 0)

It may be worth noting Fedora is not listed on this getting started guide, I know when we created the rpms we discussed that as far as we knew, they should work on Fedora, but CentOS was definitely considered the more useful target. You will probably have a smoother experience there, but we can still try to keep Fedora operational.

For my test I did find kube-apiserver pod crashing, and the following in audit.log:

type=AVC msg=audit(1493643899.467:671): avc:  denied  { read } for  pid=7848 comm="kube-apiserver" name="apiserver.crt" dev="vda1" ino=1573656 scontext=system_u:system_r:container_t:s0:c67,c939 tcontext=unconfined_u:object_r:cert_t:s0 tclass=file permissive=0
type=AVC msg=audit(1493643911.826:674): avc:  denied  { read } for  pid=8149 comm="kube-apiserver" name="apiserver.crt" dev="vda1" ino=1573656 scontext=system_u:system_r:container_t:s0:c67,c939 tcontext=unconfined_u:object_r:cert_t:s0 tclass=file permissive=0
type=AVC msg=audit(1493643940.435:678): avc:  denied  { read } for  pid=8418 comm="kube-apiserver" name="apiserver.crt" dev="vda1" ino=1573656 scontext=system_u:system_r:container_t:s0:c67,c939 tcontext=unconfined_u:object_r:cert_t:s0 tclass=file permissive=0

All other pods seem to come up so this would appear to be a relatively isolated issue with the new certs.

[coeki]

@dgoodwin, I'm sorry for my rant, I admit I was frustrated, cause last time I tested it on fedora/25-cloud-base, it seemed to work. But this was in one of the 1.6.alpha releases, where I build the binaries myself.

I know Fedora is officially not supported, but well centos and fedora having worked smiliar in the past, I still run it.

Probably something changed policy wise, at least the selinux labels differ between centos/7 and fedora/25, which might be the issue.

So I did some more investigations and trying to piece it together.

Fedora/25 uses the newer scheme of container_t for the process label and container_file_t for the mount label. Centos/7 uses svirt_lxc_net_t for process and svirt_sandbox_file_t for the mount label. Not sure if this matters though, they should be the same basically.

BTW I'm seeing the same avc's for kube-apiserver on fedora/25:

type=AVC msg=audit(1493669542.347:1525): avc: denied { read } for pid=7005 comm="kube-apiserver" name="apiserver.crt" dev="sda1" ino=1573608 scontext=system_u:system_r:container_t:s0:c400,c673 tcontext=unconfined_u:object_r:cert_t:s0 tclass=file permissive=0

After some tries the kube-apiserver dies, not even running in docker. Now if I manually adjust the manifest for kube-apiserver and add:

  securityContext:
    seLinuxOptions:
      type: spc_t

It runs the kube-apiserver, but if the edit was not quick enough, you might have to restart kubelet. Now it works with setenforce 1. This also affects kube-controller-manager.

type=AVC msg=audit(1493673979.866:3337): avc: denied { read } for pid=23013 comm="kube-controller" name="sa.key" dev="sda1" ino=1573672 scontext=system_u:system_r:container_t:s0:c535,c704 tcontext=unconfined_u:object_r:cert_t:s0 tclass=file permissive=0

Fixed that manifest like kube-apiserver and then the cluster runs again.

So I can't put my finger on it.
Conceptually conflicting selinux labels on hostPath should raise an AVC, that's why we introduced the spc_t container type for etcd, especially for write. So all containers/pods using a hostPath mount for writing, should be run as spc_t (weave-1.6 plugin also does that now).
In this case however reading the hostPath, /etc/kubernetes/, is prohibited on fedora/25, but not Centos/7.

So what is broken? That remains a good point.

Thanks for your patience ;)

[dgoodwin]

Not a problem, it is frustrating keeping this functional. I'm not great with selinux but this is good info above, I will try to find some time to debug soon. We could perhaps just go spc_t for the apiserver it's just unfortunate to have to open that one up as well. (we do this for etcd so it can write to host disk, but thus far that was it)

[coeki]

[dgoodwin]

[rhatdan]

[rhatdan]

[luxas]

[coeki]

[luxas]

[jamiehannaford]

[kargakis]

[jamiehannaford]