Crc: Could not open '...crc/cache/crc_libvirt_4.5.14/crc.qcow2': Permission denied'

[madhu-liferay]

General information

• OS: Ubuntu 18.04
• Hypervisor: KVM
• Did you run crc setup before starting it (Yes/No)? Yes

CRC version

# Put the output of `crc version`
CodeReady Containers version: 1.17.0+99f5c87
OpenShift version: 4.5.14 (embedded in binary)

CRC status

# Put the output of `crc status`
CRC VM:          Stopped
OpenShift:       Stopped
Disk Usage:      0B of 0B (Inside the CRC VM)
Cache Usage:     13.04GB
Cache Directory: /home/madhu/.crc/cache

CRC config

# Put the output of `crc config view`
BLANK

Host Operating System

# Put the output of `cat /etc/os-release` in case of Linux
# put the output of `sw_vers` in case of Mac
# Put the output of `systeminfo` in case of Windows
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Steps to reproduce

1. ./crc setup
2. ./crc start

Expected

Starts successfully

Actual

Error starting stopped VM: virError(Code=1, Domain=10, Message='internal error: process exited while connecting to monitor: 2020-10-07T07:50:41.287023Z qemu-system-x86_64: -drive file=/home/madhu/.crc/machines/crc/crc.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,aio=threads: Could not open backing file: Could not open '/home/madhu/.crc/cache/crc_libvirt_4.5.14/crc.qcow2': Permission denied')
(reverse-i-search)`chown': sudo ^Cown madhu:madhu /home/madhu/.crc/machines/crc/crc.qcow2

Logs

You can start crc with crc start --log-level debug to collect logs.
Please consider posting this on http://gist.github.com/ and post the link in the issue.

https://gist.github.com/madhu-liferay/75103587ffef6dfea511159452eacbf3

[cfergeau]

Can you post the output of crc setup --log-level debug and ls -al ~/.crc/cache and ls -al /home/madhu/.crc/cache/crc_libvirt_4.5.14?

[madhu-liferay]

Its here https://gist.github.com/madhu-liferay/6eaba31103c436bcfd9fbeb97aa9b43d

FYI, I tried the below and error persists
sudo chmod 775 -Rvf ~/.crc/cache/
sudo chown madhu:madhu ~/.crc/machines/crc/crc.qcow2

[madhu-liferay]

To answer about

'any changes to /etc/libvirtd configuration files (in particular libvirtd.conf and qemu.conf), or is this the default libvirt setup on ubuntu?'

Those are the default ones.

madhu@HF1DZF2:/etc/libvirt$ ls -lhrt
total 108K
-rw-r--r-- 1 root root 2.1K May 16  2017 virtlockd.conf
-rw-r--r-- 1 root root 2.2K May 16  2017 qemu-lockd.conf
-rw-r--r-- 1 root root 2.2K May 16  2017 libxl-lockd.conf
-rw-r--r-- 1 root root  547 May 16  2017 libvirt.conf
-rw-r--r-- 1 root root  450 May 16  2017 libvirt-admin.conf
-rw-r--r-- 1 root root 1.9K Aug 28  2018 virt-login-shell.conf
-rw-r--r-- 1 root root 2.0K Aug 28  2018 virtlogd.conf
-rw-r--r-- 1 root root 2.5K Aug 28  2018 qemu-sanlock.conf
-rw------- 1 root root  29K Aug 28  2018 qemu.conf
-rw-r--r-- 1 root root 1.2K Aug 28  2018 lxc.conf
-rw-r--r-- 1 root root 1.9K Aug 28  2018 libxl.conf
-rw-r--r-- 1 root root  16K Aug 28  2018 libvirtd.conf
drwxr-xr-x 2 root root 4.0K Aug 28  2018 hooks
drwx------ 2 root root 4.0K Nov 20  2018 secrets
drwxr-xr-x 2 root root 4.0K Jul 17 09:06 nwfilter
drwxr-xr-x 3 root root 4.0K Oct  7 13:00 storage
drwxr-xr-x 3 root root 4.0K Oct  7 13:01 qemu
madhu@HF1DZF2:/etc/libvirt$ ls -lhrt qemu
total 16K
drwxr-xr-x 3 root root 4.0K Oct  7 12:23 networks
-rw------- 1 root root 2.9K Oct  7 12:30 crc.xml
-rw------- 1 root root 4.1K Oct  7 13:01 alpine.xml
madhu@HF1DZF2:/etc/libvirt$ 

btw whenever i change the ownership of below file (tried madhu:madhu and libvirt-qemu:kvm) and do 'crc start', it reverts to root:root
~/.crc/machines/crc/crc.qcow2

[guillaumerose]

It might be due to AppArmor. This serverfault link suggests we would need to add /home/madhu/.crc/cache/crc_libvirt_4.5.14/crc.qcow2 in the libvirt profile.

Can you give it a try ? I will try spin up an Ubuntu machine on my side.

[madhu-liferay]

didn't help... tried both entries one by one and together, and an OS restart

ERRO Error starting stopped VM: virError(Code=1, Domain=10, Message='internal error: process exited while connecting to monitor: 2020-10-08T05:06:01.220062Z qemu-system-x86_64: -drive file=/home/madhu/.crc/machines/crc/crc.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,aio=threads: Could not open backing file: Could not open '/home/madhu/.crc/cache/crc_libvirt_4.5.14/crc.qcow2': Permission denied') 
Error starting stopped VM: virError(Code=1, Domain=10, Message='internal error: process exited while connecting to monitor: 2020-10-08T05:06:01.220062Z qemu-system-x86_64: -drive file=/home/madhu/.crc/machines/crc/crc.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,aio=threads: Could not open backing file: Could not open '/home/madhu/.crc/cache/crc_libvirt_4.5.14/crc.qcow2': Permission denied')
madhu@HF1DZF2:~/downloads/devops/liferay-stack-openshift-nginx/kubernetes$ cat /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper | grep madhu
  /home/madhu/.crc/cache/crc_libvirt_4.5.14/crc.qcow2 rw,
  /home/madhu/.crc/machines/crc/crc.qcow2 rw,
madhu@HF1DZF2:~/downloads/devops/liferay-stack-openshift-nginx/kubernetes$ sudo apparmor_parser -r /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper 
madhu@HF1DZF2:~/downloads/devops/liferay-stack-openshift-nginx/kubernetes$ 

[guillaumerose]

I edited /etc/apparmor.d/libvirt/TEMPLATE.qemu and it went further.

$ cat /etc/apparmor.d/libvirt/TEMPLATE.qemu
#
# This profile is for the domain whose UUID matches this file.
#

#include <tunables/global>

profile LIBVIRT_TEMPLATE flags=(attach_disconnected) {
  #include <abstractions/libvirt-qemu>
  /home/guillaumerose/.crc/cache/crc_libvirt_4.5.14/crc.qcow2
}

[madhu-liferay]

It goes to the next step and fails. Tried crc cleanup this time, just in case, didnt help

INFO Starting CodeReady Containers VM for OpenShift 4.5.14... 
DEBU Updating CRC VM configuration                
ERRO Error starting stopped VM: virError(Code=1, Domain=10, Message='internal error: Process exited prior to exec: libvirt:  error : unable to set AppArmor profile 'libvirt-088e9640-67b5-49a6-99de-dca883ec9521' for '/usr/bin/kvm-spice': No such file or directory') 
Error starting stopped VM: virError(Code=1, Domain=10, Message='internal error: Process exited prior to exec: libvirt:  error : unable to set AppArmor profile 'libvirt-088e9640-67b5-49a6-99de-dca883ec9521' for '/usr/bin/kvm-spice': No such file or directory')
madhu@HF1DZF2:~/downloads/devops/liferay-stack-openshift-nginx/kubernetes$ ls -lhrt /usr/bin/kvm*
lrwxrwxrwx 1 root root  3 Sep 15 19:35 /usr/bin/kvm-spice -> kvm
-rwxr-xr-x 1 root root 51 Sep 15 19:35 /usr/bin/kvm

madhu@HF1DZF2:~/downloads/devops/liferay-stack-openshift-nginx/kubernetes$ cat /etc/apparmor.d/libvirt/libvirt-088e9640-67b5-49a6-99de-dca883ec9521
#
# This profile is for the domain whose UUID matches this file.
#

#include <tunables/global>

profile libvirt-088e9640-67b5-49a6-99de-dca883ec9521 flags=(attach_disconnected) {
  #include <abstractions/libvirt-qemu>
  /home/madhu/.crc/cache/crc_libvirt_4.5.14/crc.qcow2
  #include <libvirt/libvirt-088e9640-67b5-49a6-99de-dca883ec9521.files>

}

[guillaumerose]

Yes same for me.

Until we found out why, I will open a PR in https://github.com/code-ready/machine-driver-libvirt that do a full copy of the image like before when virsh capabilities reports secmodel=apparmor.

[vickas5221]

This looks like to happen only with crc 4.5 version , however any progress made on it ? @guillaumerose @madhu-liferay

[cfergeau]

Yes this is only an issue with crc 1.17 as this is caused by this change https://github.com/code-ready/machine-driver-libvirt/pull/57 which is new in the 1.17 release.

[guillaumerose]

With crc 1.17, I implemented above strategy and it's the same error at the end.

When I try with crc 1.16, I have a very similar error.

(crc) Failed to start: virError(Code=1, Domain=10, Message='internal error: Process exited prior to exec: libvirt:  error : unable to set AppArmor profile 'libvirt-f6b0b451-2dec-4815-a30a-5f6674afe1bf' for '/usr/bin/kvm-spice': No such file or directory')

[nachomillangarcia]

Got it working in ubuntu 20.04, by adding the path to /etc/apparmor.d/libvirt/TEMPLATE.qemu

profile LIBVIRT_TEMPLATE flags=(attach_disconnected) {
  #include <abstractions/libvirt-qemu>
  /home/............./.crc/cache/crc_libvirt_4.5.14/crc.qcow2 rk,

}

Found the solution here

@guillaumerose I think you lacked the rk, to set the permissions to read and lock.

Anyway I'm still facing other issue to get it working, but crc start finished successfully

[madhu-liferay]

Adding the rk option worked for me and it moves to next step. Closing this issue

had to do cleanup,delete,setup,start after this fora successful startup

[ArthurSens]

With crc 1.17, I implemented above strategy and it's the same error at the end.

When I try with crc 1.16, I have a very similar error.

(crc) Failed to start: virError(Code=1, Domain=10, Message='internal error: Process exited prior to exec: libvirt:  error : unable to set AppArmor profile 'libvirt-f6b0b451-2dec-4815-a30a-5f6674afe1bf' for '/usr/bin/kvm-spice': No such file or directory')

Hi @guillaumerose, I'm facing this same issue. Did you manage to solve it somehow?