Crc: [BUG] kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

[drpaneas]

General information

• OS: Fedora 31 Workstation Edition
• Hypervisor: KVM
• Did you run crc setup before starting it (Yes/No)? Yes

CRC version

crc version: 1.5.0+e1c8fb8
OpenShift version: 4.2.14 (embedded in binary)

CRC status

CRC VM:          Stopped
OpenShift:       Stopped
Disk Usage:      0B of 0B (Inside the CRC VM)
Cache Usage:     11.98GB
Cache Directory: /home/drpaneas/.crc/cache

CRC config

# Nothing

Host Operating System

NAME=Fedora
VERSION="31 (Workstation Edition)"
ID=fedora
VERSION_ID=31
VERSION_CODENAME=""
PLATFORM_ID="platform:f31"
PRETTY_NAME="Fedora 31 (Workstation Edition)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:31"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f31/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=31
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=31
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Workstation Edition"
VARIANT_ID=workstation

Steps to reproduce

1. crc setup
2. crc start

Expected

To work without ERRO

Actual

ERRO Failed to connect to the CRC VM with SSH

Issue

I couldn't connect to the machine, simply because the machine (VM) was having a kernel panic. So it make sense to not being able to ssh into it. However, I would expect maybe a better log message, e.g. trying to ping first, rather than ssh first. Anyways, here's the kernel panic from inside the VM:

---

https://youtu.be/u--p8i9PlV0?t=2

Message: kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

Note: use frame stepping , and . to see the actual message

---

I've noticed that there was no information about the disk of this VM:

virt-manager and going through the VM details:

* CPU: 4
* Memory: 7812 MB
* VirtO Disk1: /home/drpaneas/.crc/machines/crc/crc
  --> Storage: Unknown

Problematic XML says:

<disk type="file" device="disk">
  <driver name="qemu" type="qcow2" io="threads"/>
  <source file="/home/drpaneas/.crc/machines/crc/crc"/>
  <target dev="vda" bus="virtio"/>
  <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/>
</disk>

Debugging

Digging deeper, I realised there was something wrong with the disk crc was allocating to the VM. See:

-rw-------. 1 drpaneas drpaneas unconfined_u:object_r:user_home_t:s0 9679011840 Feb 16 16:45 crc

This happened due my custom settings:

# grep ^[^#] /etc/libvirt/qemu.conf

security_default_confined = 0
user = "drpaneas"   # <--- put your normal user username
group = "libvirt"
dynamic_ownership = 1

After removing this configuration, then the listing was different:

-rw-------. 1 qemu     qemu     system_u:object_r:svirt_image_t:s0:c223,c1015 9973268480 Feb 16 17:41 crc

Also the XML was different. A backingStore and and alias that where missing before, they are now present:

Working XML:
<disk type="file" device="disk">
  <driver name="qemu" type="qcow2" io="threads"/>
  <source file="/home/drpaneas/.crc/machines/crc/crc"/>
  <backingStore/>
  <target dev="vda" bus="virtio"/>
  <alias name="virtio-disk0"/>
  <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/>
</disk>

And also crc status is reporting the size this time:

CRC VM:          Stopped
OpenShift:       Stopped
Disk Usage:      0B of 0B (Inside the CRC VM)
Cache Usage:     11.98GB
Cache Directory: /home/drpaneas/.crc/cache

Workaround

Disable dynamic_ownership in KVM. I filling this as bug, mostly because I think you could add this to the documentation as part of the troubleshooting section. In case there is a kernel panic it most likely has to do with dynamic_onwership -- at least that was in my case.

[praveenkumar]

@zeenix , @cfergeau should we add this in the document if that is known from the libvirt side or we can do something in the machine-driver side to prevent it?

[cfergeau]

There are probably plenty of local changes that can break crc expectations, I don't think we can/should document all of them.
I'm surprised virsh start does not fail on permission issues though, rather than starting qemu and having a broken vm eventually.

[cfergeau]

I can't reproduce with

# grep ^[^#] /etc/libvirt/qemu.conf

security_default_confined = 0
user = "drpaneas"   # <--- put your normal user username
group = "libvirt"
dynamic_ownership = 1

in my qemu.conf:

$ ls -alZ ~/.crc/machines/crc/crc 
-rw-------. 1 teuf libvirt unconfined_u:object_r:user_home_t:s0 9653911552 18 févr. 11:57 /home/teuf/.crc/machines/crc/crc

I don't get a kernel panic with these settings, crc starts fine.

[drpaneas]

Please try this:

Let’s make sure that any normal user is able to connect and interact with libvirt, without any need for root priviledges. First off, we configure the sockets to be owned and be accessible by a certain group (e.g. libvirt group). Before doing that, make sure that the group libvirt already exists in your system:

getent group | grep libvirt
libvirt:x:462

If it doesn’t exist, then just create it:

root # groupadd libvirt

Now, make your normal user is part of libvirt group:

$ sudo usermod --append --groups libvirt $(whoami)
# Verification:
grep $(whoami) /etc/group | grep libvirt
libvirt:x:462:drpaneas

Also, we need to change the configuration which is related to the access of the Unix socket. The group ownership should be libvirt, the permissions for the socket should be srwxrwx--- and lastly disable other authentication methods in order to handle this solely by the socket permissions itself. To do all of those, change the configuration in /etc/libvirt/libvirtd.conf as follows:

# grep  ^[^#]  /etc/libvirt/libvirtd.conf

unix_sock_group = "libvirt"
unix_sock_dir = "/var/run/libvirt"
auth_unix_rw = "none"

After configuring the socket. Let’s do the same for qemu. Edit /etc/libvirt/qemu.conf and change the configuration as follows:

# grep ^[^#] /etc/libvirt/qemu.conf

security_default_confined = 0
user = "drpaneas"   # <--- put your normal user username
group = "libvirt"
dynamic_ownership = 1

Then, make your normal user a member of the group kvm:

usermod --append --groups kvm $(whoami)

# Verification
grep $(whoami) /etc/group | grep kvm
kvm:x:484:qemu,drpaneas

This step is needed to grant access to /dev/kvm, which is required to start VM Guests as drpaneas which is the username of my normal user.

To take the changes into effect, restart libvirtd:

root # systemctl restart libvirtd

# Verification
root # systemctl is-active libvirtd
active

From now on, drpaneas is able to communicate with virsh. You can test this, to make sure of it:

drpaneas@localhost:~> virsh -c 'qemu:///system' list
 Id    Name                           State
----------------------------------------------------

Ok, now delete the whole /.crc directory and start over again. If you still cannot reproduce it, then close this as invalid.

[gbraad]

However, I would expect maybe a better log message, e.g. trying to ping first, rather than ssh first

Using ping would conclude the same as we do with ssh; the host is unreachable. The reason we do not use ping is that the test concludes the network stack is up, while ssh actually concludes that the dependency we rely is available. We rather not add a ton of checks that do not provide a lot of insights to the user. Probably changing the message a little to Failed to connect to the CRC VM with SSH. Host might be unreachable. But you can see the issue here... even if we add Check network settings or else, it would not help the user to solve the problem. WDYT?

[gbraad]

https://github.com/kubernetes/minikube/issues/4467

This also occurs on the minikube driver.

Check /etc/libvirt/qemu.conf

# Whether libvirt should dynamically change file ownership
# to match the configured user/group above. Defaults to 1.
# Set to 0 to disable file ownership changes.
dynamic_ownership = 0  # override

However, I am not sure if we can just change the default. This is also not something that got reported before... It might be related to using this in combination with SELinux. Perhaps that could be checked for?

[drpaneas]

Probably changing the message a little to Failed to connect to the CRC VM with SSH. Host might be unreachable. But you can see the issue here... even if we add Check network settings or else, it would not help the user to solve the problem. WDYT?

That would be sufficient for me, many thanks :) I've prepared a PR https://github.com/code-ready/crc/pull/1026 (feel free to review).