Kubevirt: Virt-handler error when running in Fedora CoreOS due to semanage package missing

[alosadagrande]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug
/kind enhancement

What happened:
Installing KubeVirt on OKD 4.3 following the manual procedure: install kubevirt-operator and then kubevirt-crds (since I could not find anymore the KubeVirt community operator) found that virt-handler does not start due to semanage (core-policyutils package) is missing in the installation of Fedora CoreOS 31 (probably older versions as well).

"component":"virt-handler","level":"info","msg":"STARTING informer CRDInformer","pos":"virtinformers.go:223","timestamp":"2020-02-28T15:02:41.152269Z"}
panic: failed to list labels: out: "", error: could not find binary semanage
goroutine 1 [running]:
main.(*virtHandlerApp).Run(0xc000130400)
    cmd/virt-handler/virt-handler.go:320 +0x1f3e
main.main()
    cmd/virt-handler/virt-handler.go:475 +0x6e

What you expected to happen:
virt-handler to run successfully and apply if needed the proper SELinux label.

How to reproduce it (as minimally and precisely as possible):

1. Install OKD 4.3 (it comes with FCOS as OS)
2. Install KubeVirt on top

Anything else we need to know?:

Environment:

• KubeVirt version (use virtctl version): 0.26.1
• Kubernetes version (use kubectl version): 1.16.2 (OKD 4.3)
• OKD

NAME                           STATUS   ROLES    AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                                  KERNEL-VERSION           CONTAINER-RUNTIME
okd-master-0.okd.okdlabs.com   Ready    master   10d   v1.16.2   192.168.17.228   <none>        Fedora CoreOS preview 31.20191115.dev.1   5.3.11-300.fc31.x86_64   cri-o://1.16.0-2.rhaos4.3.git8b6bfb5.el8
okd-worker-0.okd.okdlabs.com   Ready    worker   10d   v1.16.2   192.168.17.106   <none>        Fedora CoreOS preview 31.20191115.dev.1   5.3.11-300.fc31.x86_64   cri-o://1.16.0-2.rhaos4.3.git8b6bfb5.el8
okd-worker-1.okd.okdlabs.com   Ready    worker   10d   v1.16.2   192.168.17.168   <none>        Fedora CoreOS preview 31.20191115.dev.1   5.3.11-300.fc31.x86_64   cri-o://1.16.0-2.rhaos4.3.git8b6bfb5.el8
okd-worker-2.okd.okdlabs.com   Ready    worker   10d   v1.16.2   192.168.17.210   <none>        Fedora CoreOS preview 31.20191115.dev.1   5.3.11-300.fc31.x86_64   cri-o://1.16.0-2.rhaos4.3.git8b6bfb5.el8

• VM or VMI specifications:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

[root@okd-worker-0 ~]# cat /etc/os-release 
NAME=Fedora
VERSION="31.20191115.dev.1 (CoreOS preview)"
ID=fedora
VERSION_ID=31
VERSION_CODENAME=""
PLATFORM_ID="platform:f31"
PRETTY_NAME="Fedora CoreOS preview 31.20191115.dev.1"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:31"
HOME_URL="https://getfedora.org/coreos/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/"
SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=31
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=31
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="CoreOS preview"
VARIANT_ID=coreos
OSTREE_VERSION='31.20191115.dev.1'

• Kernel (e.g. uname -a):

Linux okd-worker-0.okd.okdlabs.com 5.3.11-300.fc31.x86_64 #1 SMP Tue Nov 12 19:08:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:

• Others:
  Python is included in the virt-handler image, but semanage does not.

• Workaround
  Set SElinux to permissive. IMHO it is a no-go
  Install the policy-coreutils package and reboot the node. However, AFAIK this can lead to problems during OKD updates since it manages the OS life-cycle as well, so it can upgrade the OS and face the semanage issue again since it does not come out of the box.

[root@okd-worker-0 ~]# rpm-ostree upgrade 
Pinned to commit by custom origin: Managed by machine-config-operator

[root@okd-worker-0 ~]# crictl ps | grep handler
23de930a920f5       0a1968ebd59f60fa4c3d58ccda08112935ead5c054d2d4f3d56726ba25fb6688                                                                                5 days ago          Running             virt-handler            0                   92f69afc2d56f

[root@okd-worker-0 ~]# crictl exec -it 23de930a920f5 bash
[root@virt-handler-gkcq2 /]# python3 --version
Python 3.7.4
[root@virt-handler-gkcq2 /]# semanage 
sh: semanage: command not found

• Similar issues and PRs stand-by

2664

2694

[vladikr]

Thanks for opening this issue.

There is an ongoing discussion with openshift/okd to include the core-policyutils package.
https://github.com/openshift/okd/issues/77

However, perhaps virt-handler can include it, at least until okd does?
/cc @rmohr @fabiand

I'm not sure if we need to use rpm-ostree to install the package. The selinux-operator project installs policies[1] successfully without restarting, probably virt-handler can too. The selinux-operator project has these packages[2] in its image, but we probably don't need all of it to install our policy.

What do you think?

[1] https://github.com/JAORMX/selinux-operator/blob/master/pkg/controller/configmap/configmap_controller.go#L154

[2] https://github.com/JAORMX/selinux-operator/blob/master/images/udica/Dockerfile

[fabiand]

I'd say that it depends on the impact/size.

Do we know how much difference it would make?

[LorbusChris]

just a note: policycoreutils and with it semodule (but not semanage, which is included in policycoreutils-python-utils) is by default included in RHCOS and FCOS, so https://github.com/kubevirt/kubevirt/pull/2694 would solve this at least on these OSes afaict.

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[dustymabe]

I hit this today. Is virt-handler really trying to run semanage from the host?

[rmohr]

I hit this today. Is virt-handler really trying to run semanage from the host?

Yes. I will tackle it this week. Let' me try to finally resolve that in https://github.com/kubevirt/kubevirt/pull/2694.

[rmohr]

Finally merged. Should be available in the next developer build: https://kubevirt.io/user-guide/#/installation/installation?id=installing-the-daily-developer-builds

[dustymabe]

Seems to be working for me!

[kubevirt-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

[rmohr]

Fixed.

/close

[kubevirt-bot]

@rmohr: Closing this issue.

In response to this:

Fixed.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.