Crc: [BUG] Start fails after cheking libvirtd

Created on 4 Feb 2020  ·  14Comments  ·  Source: code-ready/crc

General information

  • OS: Linux / macOS / Windows
  • Hypervisor: KVM
  • Did you run crc setup before starting it? Yes

CRC version

# crc version    
crc version: 1.5.0+e1c8fb8
OpenShift version: 4.2.14 (embedded in binary)

CRC status

# crc status                 
CRC VM:          Stopped
OpenShift:       Stopped
Disk Usage:      0B of 0B (Inside the CRC VM)
Cache Usage:     11.98GB
Cache Directory: /home/b1zzu/.crc/cache

CRC config

Doesn't print anything

# crc config view

Host Operating System

NAME=Fedora
VERSION="31 (Workstation Edition)"
ID=fedora
VERSION_ID=31
VERSION_CODENAME=""
PLATFORM_ID="platform:f31"
PRETTY_NAME="Fedora 31 (Workstation Edition)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:31"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f31/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=31
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=31
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Workstation Edition"
VARIANT_ID=workstation

Steps to reproduce

  1. crc setup
  2. crc start On the first time it works
  3. crc stop
  4. crc start

Expected

To start OpenShift

Actual

It doesn't start

Logs

# crc start --log-level debug
DEBU No new version available. The latest version is 1.5.0 
INFO Checking if oc binary is cached              
DEBU oc binary already cached                     
INFO Checking if running as non-root              
INFO Checking if Virtualization is enabled        
DEBU Checking if the vmx/svm flags are present in /proc/cpuinfo 
DEBU CPU virtualization flags are good            
INFO Checking if KVM is enabled                   
DEBU Checking if /dev/kvm exists                  
DEBU /dev/kvm was found                           
INFO Checking if libvirt is installed             
DEBU Checking if 'virsh' is available             
DEBU 'virsh' was found in /usr/bin/virsh          
INFO Checking if user is part of libvirt group    
DEBU Checking if current user is part of the libvirt group 
DEBU Current user is already in the libvirt group 
INFO Checking if libvirt is enabled               
DEBU Checking if libvirtd.service is enabled      
DEBU libvirtd.service is already enabled          
INFO Checking if libvirt daemon is running        
DEBU Checking if libvirtd.service is running      
DEBU exit status 3 :                              
FATA exit status 3 :

I would like to add more but this is really everything I have

kinbug
Source
picture b1zzu

Most helpful comment

It is hard to provide an error for a situation that might be caused by a daemon that perhaps failed during usage. I think it crashed during an api call... but this is merely guessing.

DEBU Checking if libvirtd.service is running      
DEBU Running `systemctl is-active libvirtd`                              
DEBU command failed with exit status 3
FATA libvirtd.service is not running

would be much nicer than

DEBU Checking if libvirtd.service is running      
DEBU exit status 3 :                              
FATA exit status 3 :
picture cfergeau on 6 Feb 2020
❤1 👍1

All 14 comments

It looks like systemctl is-active libvirtd is failing on the second run? Relevant code is https://github.com/code-ready/crc/blob/master/pkg/crc/preflight/preflight_checks_linux.go#L211

cfergeau picture cfergeau on 4 Feb 2020

Thanks, @cfergeau, I've found out that libvirtd was not running. But as an imrpovement it would be good to log a better error message. Feel free to close this bug as I've manged to solve the problem

b1zzu picture b1zzu on 4 Feb 2020

it should be able to restart the service, but something here prevented it?

$ crc setup would start it... but why do you say the second time executing crc start fails? our crc stop does not stop libvirt, so likely something else might be misconfigured or failing here that might not relate to CRC.

gbraad picture gbraad on 5 Feb 2020

Yes, crc setup should start libvirtd, and ensure it starts automatically on reboots. So it's surprising if it stops between runs.

cfergeau picture cfergeau on 5 Feb 2020

it's surprising if it stops between runs

@b1zzu can you provide feedback on this?

gbraad picture gbraad on 6 Feb 2020

@gbraad @cfergeau I'm not running crc as sudo/root so I don't see how it could start libvirtd which is a systemd service. If the tool is supposed to ask for root password that the fact that I'm using the zsh shell could be the problem maybe.

I tried to go again through the steps crc start, crc stop, and crc start but I can't reproduce the issue probably because the first time libvirtd stopped for some other reasons.

I think here the problem is not the behavior of crc but just the fact that the error was not helping at all

b1zzu picture b1zzu on 6 Feb 2020

It is hard to provide an error for a situation that might be caused by a daemon that perhaps failed during usage. I think it crashed during an api call... but this is merely guessing.

We do a specific check to see if libvirtd is running and fail accordingly if not. I believe what happened here is that the vm driver returned a null as the value wasn't able to be determined.

gbraad picture gbraad on 6 Feb 2020

I'm not running crc as sudo/root so I don't see how it could start libvirtd which is a systemd service. If the tool is supposed to ask for root password that the fact that I'm using the zsh shell could be the problem maybe.

crc setup is taking care of starting libvirtd, asking for elevated privileges if needed. You are correct that crc start should not be run with sudo/as root, and that it won't try to start libvirtd. It will only check if it's running.

cfergeau picture cfergeau on 6 Feb 2020

It is hard to provide an error for a situation that might be caused by a daemon that perhaps failed during usage. I think it crashed during an api call... but this is merely guessing.

DEBU Checking if libvirtd.service is running      
DEBU Running `systemctl is-active libvirtd`                              
DEBU command failed with exit status 3
FATA libvirtd.service is not running

would be much nicer than

DEBU Checking if libvirtd.service is running      
DEBU exit status 3 :                              
FATA exit status 3 :
cfergeau picture cfergeau on 6 Feb 2020
❤1 👍1

Hi, just adding a quick feedback here after a fresh fedora 32 installation and system update sudo dnf update:

[hf@localhost ~]$ crc start
INFO Checking if oc binary is cached              
INFO Checking if podman remote binary is cached   
INFO Checking if running as non-root              
INFO Checking if Virtualization is enabled        
INFO Checking if KVM is enabled                   
INFO Checking if libvirt is installed             
INFO Checking if user is part of libvirt group    
INFO Checking if libvirt is enabled               
INFO Checking if libvirt daemon is running        
FATA Failed to check if libvirtd service is active

In other terminal:

[hf@localhost ~]$ sudo systemctl is-active libvirtd
inactive
[hf@localhost ~]$ sudo systemctl start libvirtd
[hf@localhost ~]$ sudo systemctl is-active libvirtd
active

crc start again (now working):

[hf@localhost ~]$ crc start
INFO Checking if oc binary is cached              
INFO Checking if podman remote binary is cached   
INFO Checking if running as non-root              
INFO Checking if Virtualization is enabled        
INFO Checking if KVM is enabled                   
INFO Checking if libvirt is installed             
INFO Checking if user is part of libvirt group    
INFO Checking if libvirt is enabled               
INFO Checking if libvirt daemon is running        
INFO Checking if a supported libvirt version is installed 
INFO Checking if crc-driver-libvirt is installed  
INFO Checking if libvirt 'crc' network is available 
INFO Checking if libvirt 'crc' network is active  
INFO Checking if NetworkManager is installed      
INFO Checking if NetworkManager service is running 
INFO Checking if /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf exists 
INFO Checking if /etc/NetworkManager/dnsmasq.d/crc.conf exists 
? Image pull secret [? for help] 
...

INFO Extracting bundle: crc_libvirt_4.4.3.crcbundle ... 
INFO Checking size of the disk image /home/hf/.crc/cache/crc_libvirt_4.4.3/crc.qcow2 ... 
INFO Creating CodeReady Containers VM for OpenShift 4.4.3... 
INFO CodeReady Containers VM is running           
INFO Verifying validity of the cluster certificates ... 
INFO Check internal and public DNS query ...      
INFO Check DNS query from host ...                
INFO Generating new SSH key                       
INFO Copying kubeconfig file to instance dir ...  
INFO Starting OpenShift kubelet service           
INFO Configuring cluster for first start          
INFO Adding user's pull secret ...                
INFO Updating cluster ID ...                      
INFO Starting OpenShift cluster ... [waiting 3m] 

$ cat /etc/redhat-release 
Fedora release 32 (Thirty Two)

$ crc version
crc version: 1.10.0+9025021
OpenShift version: 4.4.3 (embedded in binary)
helio-frota picture helio-frota on 27 May 2020 
FATA Failed to check if libvirtd service is active

This is unexpected as this message would mean that systemctl is-active libvirtd failed? returning inactive is not a failure

cfergeau picture cfergeau on 27 May 2020

yeah, the first time I ran crc start it failed with the message: FATA Failed to check if libvirtd service is active.

Then, in other terminal I started the libvirtd and ran crc start again and the issue gone.

But yesterday I tested same fedora 32 in other computer and everything goes correct without need to manually start libvirtd.

helio-frota picture helio-frota on 27 May 2020

I'm experiencing the same issue with Fedora 32. After manually restarting libvirtd service I was able to run crc start. :man_shrugging: 

INFO Checking if oc binary is cached              
INFO Checking if podman remote binary is cached   
INFO Checking if goodhosts binary is cached       
INFO Checking if running as non-root              
INFO Checking if Virtualization is enabled        
INFO Checking if KVM is enabled                   
INFO Checking if libvirt is installed             
INFO Checking if user is part of libvirt group    
INFO Checking if libvirt is enabled               
INFO Checking if libvirt daemon is running        
FATA Failed to check if libvirtd service is active

Then:

systemctl is-active libvirtd
inactive

Just in case, I checked the service:

sudo systemctl status libvirtd.service
● libvirtd.service - Virtualization daemon
     Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Wed 2020-06-10 09:39:05 -03; 7h ago
TriggeredBy: ● libvirtd.socket
             ● libvirtd-ro.socket
             ● libvirtd-admin.socket
       Docs: man:libvirtd(8)
             https://libvirt.org
    Process: 6621 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, status=0/SUCCESS)
   Main PID: 6621 (code=exited, status=0/SUCCESS)
        CPU: 464ms

Jun 10 09:37:08 skywalker dnsmasq[6927]: started, version 2.81 cachesize 150
Jun 10 09:37:08 skywalker dnsmasq[6927]: compile time options: IPv6 GNU-getopt DBus no-UBus no-i18n IDN2 DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify dumpfile
Jun 10 09:37:08 skywalker dnsmasq-dhcp[6927]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h
Jun 10 09:37:08 skywalker dnsmasq-dhcp[6927]: DHCP, sockets bound exclusively to interface virbr0
Jun 10 09:37:08 skywalker dnsmasq[6927]: reading /etc/resolv.conf
Jun 10 09:37:08 skywalker dnsmasq[6927]: using nameserver 127.0.0.1#53
Jun 10 09:37:08 skywalker dnsmasq[6927]: read /etc/hosts - 7 addresses
Jun 10 09:37:08 skywalker dnsmasq[6927]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Jun 10 09:37:08 skywalker dnsmasq-dhcp[6927]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Jun 10 09:39:05 skywalker systemd[1]: libvirtd.service: Succeeded.

Seems ok, right? Let's restart then:

sudo systemctl restart libvirtd.service

and...

systemctl is-active libvirtd
active

I haven't the time to debug it yet, but could be something on Fedora 32 maybe?

ricardozanini picture ricardozanini on 10 Jun 2020

It's been fixed by https://github.com/code-ready/crc/commit/6c0b5122ff29b75af0467d06f0cb9bec2cf2fead

cfergeau picture cfergeau on 11 Jun 2020
👎1 👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

mojotx picture mojotx  ·  3Comments
BrianGos picture BrianGos  ·  9Comments
skabashnyuk picture skabashnyuk  ·  3Comments
bradfordcp picture bradfordcp  ·  9Comments
omkensey picture omkensey  ·  8Comments