Crc: Error starting crc version 1.10.0+9025021. OpenShift version: 4.4.3 (embedded in binary)

Created on 1 Jun 2020  路  5Comments  路  Source: code-ready/crc

General information

  • OS: macOS
  • Hypervisor: hyperkit
  • Did you run crc setup before starting it (Yes/No)? Yes

CRC version

# Put the output of `crc version`

crc version: 1.10.0+9025021
OpenShift version: 4.4.3 (embedded in binary)

CRC status

# Put the output of `crc status`
CRC VM:          Stopped
OpenShift:       Stopped
Disk Usage:      0B of 0B (Inside the CRC VM)
Cache Usage:     12.05GB
Cache Directory: /Users/sp82299/.crc/cache

## CRC config
```bash
# Put the output of `crc config view`

No output produced

Host Operating System

# Put the output of `cat /etc/os-release` in case of Linux
# put the output of `sw_vers` in case of Mac
# Put the output of `systeminfo` in case of Windows

ProductName: Mac OS X
ProductVersion: 10.15.5
BuildVersion: 19F96

Steps to reproduce

  1. crc delete
  2. crc setup
  3. crc start --log-level debug
    4. 4.

Expected

CRC running

Actual

CRC is failing and not starting

I've done many times the process of: crc stop -> crc delete -> rm -rf ~.crc -> crc setup -> crc start. -log-level debug

Logs

crc start -p ./pull-secret.txt --log-level debug
DEBU No new version available. The latest version is 1.10.0
INFO Checking if oc binary is cached
DEBU oc binary already cached
INFO Checking if podman remote binary is cached
DEBU Currently podman remote is not supported
INFO Checking if running as non-root
INFO Checking if HyperKit is installed
DEBU Checking if hyperkit is installed
INFO Checking if crc-driver-hyperkit is installed
DEBU Checking if crc-driver-hyperkit is installed
DEBU Running '/Users/sp82299/.crc/bin/crc-driver-hyperkit version'
DEBU crc-driver-hyperkit is already installed in /Users/sp82299/.crc/bin/crc-driver-hyperkit
INFO Checking file permissions for /etc/resolver/testing
INFO Checking file permissions for /etc/hosts
Checking file: /Users/sp82299/.crc/machines/crc/.crc-exist
INFO Loading bundle: crc_hyperkit_4.4.3.crcbundle ...
INFO Checking size of the disk image /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/crc.qcow2 ...
INFO Creating CodeReady Containers VM for OpenShift 4.4.3...
Found binary path at /Users/sp82299/.crc/bin/crc-driver-hyperkit
Launching plugin server for driver hyperkit
Plugin server listening at address 127.0.0.1:50711
() Calling .GetVersion
Using API Version 1
() Calling .SetConfigRaw
() Calling .GetMachineName
(crc) Calling .GetMachineName
(crc) Calling .DriverName
Running pre-create checks...
(crc) Calling .PreCreateCheck
(crc) DBG | exe=/Users/sp82299/.crc/bin/crc-driver-hyperkit uid=0
(crc) Calling .GetConfigRaw
Creating machine...
(crc) Calling .Create
(crc) DBG | exe=/Users/sp82299/.crc/bin/crc-driver-hyperkit uid=0
(crc) Downloading /Users/sp82299/.crc/cache/crc.disk from file:///Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/crc.qcow2...
(crc) DBG | exe=/Users/sp82299/.crc/bin/crc-driver-hyperkit uid=0
(crc) DBG | clean start, hyperkit pid file doesn't exist: /Users/sp82299/.crc/machines/crc/hyperkit.pid
(crc) DBG | Using hyperkit binary from /Users/sp82299/.crc/bin/hyperkit
(crc) DBG | Using UUID c3d68012-0208-11ea-9fd7-f2189899ab08
(crc) DBG | Generated MAC e:f0:e3:c7:4f:be
(crc) DBG | Starting with cmdline: BOOT_IMAGE=(hd0,gpt1)/ostree/rhcos-ec6f097d71f19d2713779aeb6d2296122dec138ffa31e2fd0a15a0f39e39cada/vmlinuz-4.18.0-147.8.1.el8_1.x86_64 rhcos.root=crypt_rootfs console=tty0 console=ttyS0,115200n8 ignition.platform.id=qemu rd.luks.options=discard ostree=/ostree/boot.1/rhcos/ec6f097d71f19d2713779aeb6d2296122dec138ffa31e2fd0a15a0f39e39cada/0
(crc) DBG | Trying to execute /Users/sp82299/.crc/bin/hyperkit -A -u -F /Users/sp82299/.crc/machines/crc/hyperkit.pid -c 4 -m 9216M -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-net -U c3d68012-0208-11ea-9fd7-f2189899ab08 -s 2:0,virtio-blk,file:///Users/sp82299/.crc/machines/crc/crc.disk,format=qcow -s 3,virtio-rnd -l com1,autopty=/Users/sp82299/.crc/machines/crc/tty,log=/Users/sp82299/.crc/machines/crc/console-ring -f kexec,/Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/vmlinuz-4.18.0-147.8.1.el8_1.x86_64,/Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/initramfs-4.18.0-147.8.1.el8_1.x86_64.img,earlyprintk=serial BOOT_IMAGE=(hd0,gpt1)/ostree/rhcos-ec6f097d71f19d2713779aeb6d2296122dec138ffa31e2fd0a15a0f39e39cada/vmlinuz-4.18.0-147.8.1.el8_1.x86_64 rhcos.root=crypt_rootfs console=tty0 console=ttyS0,115200n8 ignition.platform.id=qemu rd.luks.options=discard ostree=/ostree/boot.1/rhcos/ec6f097d71f19d2713779aeb6d2296122dec138ffa31e2fd0a15a0f39e39cada/0
(crc) DBG | exe=/Users/sp82299/.crc/bin/crc-driver-hyperkit uid=0
(crc) DBG | hyperkit pid from json: 2102
(crc) DBG | Searching for e:f0:e3:c7:4f:be in /var/db/dhcpd_leases ...
(crc) DBG | Found 1 entries in /var/db/dhcpd_leases!
(crc) DBG | dhcp entry: {Name:crc-dv9sm-master-0 IPAddress:192.168.64.2 HWAddress:e:f0:e3:c7:4f:be ID:1,e:f0:e3:c7:4f:be Lease:0x5ed68661}
(crc) DBG | Found match: e:f0:e3:c7:4f:be
(crc) DBG | IP: 192.168.64.2
(crc) Calling .GetConfigRaw
(crc) Calling .DriverName
(crc) Calling .DriverName
Waiting for machine to be running, this may take a few minutes...
(crc) Calling .GetState
(crc) DBG | exe=/Users/sp82299/.crc/bin/crc-driver-hyperkit uid=0
(crc) DBG | hyperkit pid from json: 2102
Machine is up and running!
Machine successfully created
Created /Users/sp82299/.crc/machines/crc/.crc-exist
(crc) Calling .GetState
(crc) DBG | exe=/Users/sp82299/.crc/bin/crc-driver-hyperkit uid=0
(crc) DBG | hyperkit pid from json: 2102
Found binary path at /Users/sp82299/.crc/bin/crc-driver-hyperkit
Launching plugin server for driver hyperkit
Plugin server listening at address 127.0.0.1:50716
() Calling .GetVersion
Using API Version 1
() Calling .SetConfigRaw
() Calling .GetMachineName
DEBU Waiting until ssh is available
(crc) Calling .GetSSHHostname
(crc) Calling .GetSSHPort
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc (-r--------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -o IdentitiesOnly=yes -i /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc -p 22] /usr/bin/ssh }
DEBU About to run SSH command:
exit 0
DEBU SSH cmd err, output: :
INFO CodeReady Containers VM is running
INFO Verifying validity of the cluster certificates ...
(crc) Calling .GetSSHHostname
(crc) Calling .GetSSHPort
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc (-r--------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -o IdentitiesOnly=yes -i /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc -p 22] /usr/bin/ssh }
DEBU About to run SSH command:
date --date="$(sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -enddate | cut -d= -f 2)" --iso-8601=seconds
DEBU SSH cmd err, output: : 2020-06-01T05:40:49+00:00
(crc) Calling .GetIP
(crc) Calling .DriverName
(crc) Calling .GetSSHHostname
(crc) Calling .GetSSHPort
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc (-r--------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -o IdentitiesOnly=yes -i /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc -p 22] /usr/bin/ssh }
DEBU About to run SSH command:
echo dXNlcj1yb290CnBvcnQ9IDUzCmJpbmQtaW50ZXJmYWNlcwpleHBhbmQtaG9zdHMKbG9nLXF1ZXJpZXMKc3J2LWhvc3Q9X2V0Y2Qtc2VydmVyLXNzbC5fdGNwLmNyYy50ZXN0aW5nLGV0Y2QtMC5jcmMudGVzdGluZywyMzgwLDEwCmxvY2FsPS9jcmMudGVzdGluZy8KZG9tYWluPWNyYy50ZXN0aW5nCmFkZHJlc3M9L2FwcHMtY3JjLnRlc3RpbmcvMTkyLjE2OC42NC4yCmFkZHJlc3M9L2V0Y2QtMC5jcmMudGVzdGluZy8xOTIuMTY4LjY0LjIKYWRkcmVzcz0vYXBpLmNyYy50ZXN0aW5nLzE5Mi4xNjguNjQuMgphZGRyZXNzPS9hcGktaW50LmNyYy50ZXN0aW5nLzE5Mi4xNjguNjQuMgphZGRyZXNzPS9jcmMtZHY5c20tbWFzdGVyLTAuY3JjLnRlc3RpbmcvMTkyLjE2OC4xMjYuMTEK | base64 --decode | sudo tee /var/srv/dnsmasq.conf > /dev/null
DEBU SSH cmd err, output: :
(crc) Calling .GetSSHHostname
(crc) Calling .GetSSHPort
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc (-r--------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -o IdentitiesOnly=yes -i /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc -p 22] /usr/bin/ssh }
DEBU About to run SSH command:
sudo podman rm -f dnsmasq
DEBU SSH cmd err, output: exit status 1: Error: Failed to evict container: "": Failed to find container "dnsmasq" in state: no container with name or ID dnsmasq found: no such container
(crc) Calling .GetSSHHostname
(crc) Calling .GetSSHPort
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc (-r--------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -o IdentitiesOnly=yes -i /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc -p 22] /usr/bin/ssh }
DEBU About to run SSH command:
sudo rm -f /var/lib/cni/networks/podman/10.88.0.8
DEBU SSH cmd err, output: :
(crc) Calling .GetSSHHostname
(crc) Calling .GetSSHPort
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc (-r--------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -o IdentitiesOnly=yes -i /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc -p 22] /usr/bin/ssh }
DEBU About to run SSH command:
sudo podman run --ip 10.88.0.8 --name dnsmasq -v /var/srv/dnsmasq.conf:/etc/dnsmasq.conf -p 53:53/udp --privileged -d quay.io/crcont/dnsmasq:latest
DEBU SSH cmd err, output: : 6cb49587ab356f97e4f3c080aca43b083a2aa2673be93e23c15636607f2d7828
DEBU Updating api.crc.testing to 192.168.64.2 in /etc/hosts file
DEBU Updating oauth-openshift.apps-crc.testing to 192.168.64.2 in /etc/hosts file
INFO Network restart not needed
(crc) Calling .GetSSHHostname
(crc) Calling .GetSSHPort
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc (-r--------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -o IdentitiesOnly=yes -i /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc -p 22] /usr/bin/ssh }
DEBU About to run SSH command:
cat /etc/resolv.conf
DEBU SSH cmd err, output: : # Generated by NetworkManager
nameserver 192.168.64.1
(crc) Calling .GetSSHHostname
(crc) Calling .GetSSHPort
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc (-r--------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -o IdentitiesOnly=yes -i /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc -p 22] /usr/bin/ssh }
DEBU About to run SSH command:
echo IyBHZW5lcmF0ZWQgYnkgQ1JDCnNlYXJjaCBjcmMudGVzdGluZwpuYW1lc2VydmVyIDEwLjg4LjAuOApuYW1lc2VydmVyIDE5Mi4xNjguNjQuMQoK | base64 --decode | sudo tee /etc/resolv.conf > /dev/null
DEBU SSH cmd err, output: :
(crc) Calling .GetSSHHostname
(crc) Calling .GetSSHPort
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc (-r--------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -o IdentitiesOnly=yes -i /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc -p 22] /usr/bin/ssh }
DEBU About to run SSH command:
host -R 3 foo.apps-crc.testing
DEBU SSH cmd err, output: : foo.apps-crc.testing has address 192.168.64.2
INFO Check internal and public DNS query ...
(crc) Calling .GetSSHHostname
(crc) Calling .GetSSHPort
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc (-r--------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -o IdentitiesOnly=yes -i /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc -p 22] /usr/bin/ssh }
DEBU About to run SSH command:
host -R 3 quay.io
DEBU SSH cmd err, output: : quay.io has address 52.202.225.67
quay.io has address 18.205.55.240
quay.io has address 54.84.167.150
quay.io mail is handled by 10 aspmx.l.google.com.
quay.io mail is handled by 20 alt1.aspmx.l.google.com.
quay.io mail is handled by 20 alt2.aspmx.l.google.com.
quay.io mail is handled by 30 aspmx2.googlemail.com.
quay.io mail is handled by 30 aspmx3.googlemail.com.
INFO Check DNS query from host ...
DEBU api.crc.testing resolved to [192.168.64.2]
INFO Generating new SSH key
(crc) Calling .GetSSHHostname
(crc) Calling .GetSSHPort
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc (-r--------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -o IdentitiesOnly=yes -i /Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/id_rsa_crc -p 22] /usr/bin/ssh }
DEBU About to run SSH command:
echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPMiL2tfPDIVnFONdPACYT0POdsOgPl9d2C1EilOrEVsE1gyaX+lnhW4T0qGxr4YdRVRrxaB+TM8NY8HBc8MKag7NX/qetIQqRgWry8nj9KxZwQ0Xp4e8ib04ST6tAnl7yx3F3iFVoarIzb8rHQk5wbo1MsEeNK66zDninaN8/f9pm3EzTADAs3GC0i21tIYqo3BT40A9OniZ8+17mqV3R867npzxcn1sEtiSIih0q2X+F3C/lJINLG2Y+Sxj1dUBoaK8bIOZPcK2XCbBGIhEIbsC7dU3OMf9pHn068HxbGzU0aHeRVTfNzw7yFRwS9tWvwT1kvZAL58LG5j79b+e7
' > /home/core/.ssh/authorized_keys
DEBU SSH cmd err, output: :
INFO Copying kubeconfig file to instance dir ...
DEBU Copying '/Users/sp82299/.crc/cache/crc_hyperkit_4.4.3/kubeconfig' to '/Users/sp82299/.crc/machines/crc/kubeconfig'
INFO Cluster TLS certificates have expired, renewing them... [will take up to 5 minutes]
(crc) Calling .GetSSHHostname
(crc) Calling .GetSSHPort
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sp82299/.crc/machines/crc/id_rsa (-rw-------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -o IdentitiesOnly=yes -i /Users/sp82299/.crc/machines/crc/id_rsa -p 22] /usr/bin/ssh }
DEBU About to run SSH command:
sudo systemctl daemon-reload
DEBU SSH cmd err, output: :
(crc) Calling .GetSSHHostname
(crc) Calling .GetSSHPort
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sp82299/.crc/machines/crc/id_rsa (-rw-------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -o IdentitiesOnly=yes -i /Users/sp82299/.crc/machines/crc/id_rsa -p 22] /usr/bin/ssh }
DEBU About to run SSH command:
sudo systemctl -f start kubelet
DEBU SSH cmd err, output: :
DEBU kube-apiserver-operator image: openshift/cert-recovery
(crc) Calling .GetSSHHostname
(crc) Calling .GetSSHPort
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sp82299/.crc/machines/crc/id_rsa (-rw-------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -o IdentitiesOnly=yes -i /Users/sp82299/.crc/machines/crc/id_rsa -p 22] /usr/bin/ssh }
DEBU About to run SSH command:
sudo podman run -it --rm --network=host -v /etc/kubernetes/:/etc/kubernetes/:Z --entrypoint=/usr/bin/cluster-kube-apiserver-operator 'openshift/cert-recovery' recovery-apiserver create
DEBU SSH cmd err, output: : I0601 17:18:40.311628 1 apiserver.go:217] Recovery apiserver certificates will be valid for 168h0m0s
I0601 17:18:40.794095 1 create.go:82] To access the server.
I0601 17:18:40.794112 1 create.go:83] export KUBECONFIG=/etc/kubernetes/static-pod-resources/recovery-kube-apiserver-pod admin.kubeconfig
DEBU Recovery pod is running
DEBU Recovery KUBECONFIG: [/etc/kubernetes/static-pod-resources/recovery-kube-apiserver-pod/admin.kubeconfig]
DEBU kube-apiserver-operator: Running sudo oc get namespace kube-system --kubeconfig /etc/kubernetes/static-pod-resources/recovery-kube-apiserver-pod/admin.kubeconfig --context admin --cluster recovery
(crc) Calling .GetSSHHostname
(crc) Calling .GetSSHPort
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/sp82299/.crc/machines/crc/id_rsa (-rw-------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -o IdentitiesOnly=yes -i /Users/sp82299/.crc/machines/crc/id_rsa -p 22] /usr/bin/ssh }
DEBU About to run SSH command:
sudo oc get namespace kube-system --kubeconfig /etc/kubernetes/static-pod-resources/recovery-kube-apiserver-pod/admin.kubeconfig --context admin --cluster recovery
DEBU SSH cmd err, output: exit status 1: The connection to the server localhost:7443 was refused - did you specify the right host or port?
DEBU error: Temporary Error: ssh command error:
command : sudo oc get namespace kube-system --kubeconfig /etc/kubernetes/static-pod-resources/recovery-kube-apiserver-pod/admin.kubeconfig --context admin --cluster recovery
err : exit status 1
output : The connection to the server localhost:7443 was refused - did you specify the right host or port?

  • sleeping 1s
    DEBU retry loop 1
    DEBU kube-apiserver-operator: Running sudo oc get namespace kube-system --kubeconfig /etc/kubernetes/static-pod-resources/recovery-kube-apiserver-pod/admin.kubeconfig --context admin --cluster recovery
    (crc) Calling .GetSSHHostname
    (crc) Calling .GetSSHPort
    (crc) Calling .GetSSHKeyPath
    (crc) Calling .GetSSHKeyPath
    (crc) Calling .GetSSHUsername
    Using SSH client type: external
    Using SSH private key: /Users/sp82299/.crc/machines/crc/id_rsa (-rw-------)
    &{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -o IdentitiesOnly=yes -i /Users/sp82299/.crc/machines/crc/id_rsa -p 22] /usr/bin/ssh }
    DEBU About to run SSH command:
    sudo oc get namespace kube-system --kubeconfig /etc/kubernetes/static-pod-resources/recovery-kube-apiserver-pod/admin.kubeconfig --context admin --cluster recovery
    DEBU SSH cmd err, output: exit status 1: The connection to the server localhost:7443 was refused - did you specify the right host or port?
    DEBU error: Temporary Error: ssh command error:
    command : sudo oc get namespace kube-system --kubeconfig /etc/kubernetes/static-pod-resources/recovery-kube-apiserver-pod/admin.kubeconfig --context admin --cluster recovery
    err : exit status 1
    output : The connection to the server localhost:7443 was refused - did you specify the right host or port?
  • sleeping 1s
    DEBU retry loop 2
    DEBU kube-apiserver-operator: Running sudo oc get namespace kube-system --kubeconfig /etc/kubernetes/static-pod-resources/recovery-kube-apiserver-pod/admin.kubeconfig --context admin --cluster recovery
    (crc) Calling .GetSSHHostname
    (crc) Calling .GetSSHPort
    (crc) Calling .GetSSHKeyPath
    (crc) Calling .GetSSHKeyPath
    (crc) Calling .GetSSHUsername
    Using SSH client type: external
    Using SSH private key: /Users/sp82299/.crc/machines/crc/id_rsa (-rw-------)
    &{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -o IdentitiesOnly=yes -i /Users/sp82299/.crc/machines/crc/id_rsa -p 22] /usr/bin/ssh }
    DEBU About to run SSH command:
    sudo oc get namespace kube-system --kubeconfig /etc/kubernetes/static-pod-resources/recovery-kube-apiserver-pod/admin.kubeconfig --context admin --cluster recovery
    DEBU SSH cmd err, output: : NAME STATUS AGE
    kube-system Active 30d
    (crc) Calling .GetSSHHostname
    (crc) Calling .GetSSHPort
    (crc) Calling .GetSSHKeyPath
    (crc) Calling .GetSSHKeyPath
    (crc) Calling .GetSSHUsername
    Using SSH client type: external
    Using SSH private key: /Users/sp82299/.crc/machines/crc/id_rsa (-rw-------)
    &{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -o IdentitiesOnly=yes -i /Users/sp82299/.crc/machines/crc/id_rsa -p 22] /usr/bin/ssh }
    DEBU About to run SSH command:
    sudo podman run -it --rm --network=host -v /etc/kubernetes/:/etc/kubernetes/:Z --entrypoint=/usr/bin/cluster-kube-apiserver-operator 'openshift/cert-recovery' regenerate-certificates
    DEBU SSH cmd err, output: exit status 1: I0601 17:19:12.758555 1 certrotationcontroller.go:505] Waiting for CertRotation
    I0601 17:19:13.259131 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "AggregatorProxyClientCert"
    I0601 17:19:13.259233 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "AggregatorProxyClientCert"
    I0601 17:19:13.259244 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "KubeAPIServerToKubeletClientCert"
    I0601 17:19:13.259523 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "KubeAPIServerToKubeletClientCert"
    I0601 17:19:13.259587 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "LocalhostServing"
    I0601 17:19:13.259611 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "LocalhostServing"
    I0601 17:19:13.259626 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "ServiceNetworkServing"
    I0601 17:19:13.259929 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "ServiceNetworkServing"
    I0601 17:19:13.260079 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "ExternalLoadBalancerServing"
    I0601 17:19:13.260228 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "ExternalLoadBalancerServing"
    I0601 17:19:13.260236 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "InternalLoadBalancerServing"
    I0601 17:19:13.260251 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "InternalLoadBalancerServing"
    I0601 17:19:13.260257 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "LocalhostRecoveryServing"
    I0601 17:19:13.260262 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "LocalhostRecoveryServing"
    I0601 17:19:13.260266 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "KubeControllerManagerClient"
    I0601 17:19:13.260283 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "KubeControllerManagerClient"
    I0601 17:19:13.260288 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "KubeSchedulerClient"
    I0601 17:19:13.260462 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "KubeSchedulerClient"
    I0601 17:19:13.260492 1 certrotationcontroller.go:527] Finished waiting for CertRotation
    I0601 17:19:13.260509 1 kubecontrollermanagercertrotation.go:84] Waiting for CertRotation
    I0601 17:19:13.260529 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "CSRSigningCert"
    I0601 17:19:13.260560 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "CSRSigningCert"
    I0601 17:19:13.260685 1 kubecontrollermanagercertrotation.go:90] Finished waiting for CertRotation
    I0601 17:19:13.260768 1 regenerate_certificates.go:197] Refreshing certificates.
    I0601 17:19:13.272210 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SignerUpdateRequired' "aggregator-client-signer" in "openshift-kube-apiserver-operator" requires a new signing cert/key pair: past its latest possible time 2020-05-26 23:50:47.8 +0000 UTC
    I0601 17:19:13.703145 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'CABundleUpdateRequired' "kube-apiserver-aggregator-client-ca" in "openshift-config-managed" requires a new cert
    I0601 17:19:13.703174 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretUpdated' Updated Secret/aggregator-client-signer -n openshift-kube-apiserver-operator because it changed
    I0601 17:19:13.724121 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/kube-apiserver-aggregator-client-ca -n openshift-config-managed:
    cause by changes in data.ca-bundle.crt
    I0601 17:19:13.724943 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TargetUpdateRequired' "aggregator-client" in "openshift-kube-apiserver" requires a new target cert/key pair: past its latest possible time 2020-05-26 23:51:47.6 +0000 UTC
    I0601 17:19:14.372539 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretUpdated' Updated Secret/aggregator-client -n openshift-kube-apiserver because it changed
    I0601 17:19:14.373217 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TargetUpdateRequired' "localhost-serving-cert-certkey" in "openshift-kube-apiserver" requires a new target cert/key pair: already expired
    I0601 17:19:15.167110 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'SecretUpdateFailed' Failed to update Secret/localhost-serving-cert-certkey -n openshift-kube-apiserver: Operation cannot be fulfilled on secrets "localhost-serving-cert-certkey": the object has been modified; please apply your changes to the latest version and try again
    I0601 17:19:15.167606 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SignerUpdateRequired' "kube-control-plane-signer" in "openshift-kube-apiserver-operator" requires a new signing cert/key pair: past its refresh time 2020-06-01 04:37:54 +0000 UTC
    I0601 17:19:15.964111 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'SecretUpdateFailed' Failed to update Secret/kube-control-plane-signer -n openshift-kube-apiserver-operator: Secret "kube-control-plane-signer" is invalid: type: Invalid value: "kubernetes.io/tls": field is immutable
    I0601 17:19:16.164906 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretDeleted' Deleted Secret/kube-control-plane-signer -n openshift-kube-apiserver-operator
    [Operation cannot be fulfilled on secrets "localhost-serving-cert-certkey": the object has been modified; please apply your change to the latest version and try again, secrets "kube-control-plane-signer" already exists]
    DEBU Error running 'regenerate-certificates' command in the kube-apiserver-operator
    (crc) Calling .GetSSHHostname
    (crc) Calling .GetSSHPort
    (crc) Calling .GetSSHKeyPath
    (crc) Calling .GetSSHKeyPath
    (crc) Calling .GetSSHUsername
    Using SSH client type: external
    Using SSH private key: /Users/sp82299/.crc/machines/crc/id_rsa (-rw-------)
    &{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -o IdentitiesOnly=yes -i /Users/sp82299/.crc/machines/crc/id_rsa -p 22] /usr/bin/ssh }
    DEBU About to run SSH command:
    sudo podman run -it --rm --network=host -v /etc/kubernetes/:/etc/kubernetes/:Z --entrypoint=/usr/bin/cluster-kube-apiserver-operator 'openshift/cert-recovery' recovery-apiserver destroy
    DEBU SSH cmd err, output: : I0601 17:19:18.094402 1 apiserver.go:347] Deleting resource-dir "/etc/kubernetes/static-pod-resources/recovery-kube-apiserver-pod"
    I0601 17:19:18.094766 1 apiserver.go:353] Deleting recovery pod manifest "/etc/kubernetes/manifests/recovery-kube-apiserver- od.yaml"
    (crc) Calling .GetSSHHostname
    (crc) Calling .GetSSHPort
    (crc) Calling .GetSSHKeyPath
    (crc) Calling .GetSSHKeyPath
    (crc) Calling .GetSSHUsername
    Using SSH client type: external
    Using SSH private key: /Users/sp82299/.crc/machines/crc/id_rsa (-rw-------)
    &{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -o IdentitiesOnly=yes -i /Users/sp82299/.crc/machines/crc/id_rsa -p 22] /usr/bin/ssh }
    DEBU About to run SSH command:
    sudo systemctl -f stop kubelet
    DEBU SSH cmd err, output: :
    ERRO Failed to renew TLS certificates: ssh command error:
    command : sudo podman run -it --rm --network=host -v /etc/kubernetes/:/etc/kubernetes/:Z --entrypoint=/usr/bin/cluster-kube-apiserver-operator 'openshift/cert-recovery' regenerate-certificates
    err : exit status 1
    output : I0601 17:19:12.758555 1 certrotationcontroller.go:505] Waiting for CertRotation
    I0601 17:19:13.259131 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "AggregatorProxyClientCert"
    I0601 17:19:13.259233 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "AggregatorProxyClientCert"
    I0601 17:19:13.259244 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "KubeAPIServerToKubeletClientCert"
    I0601 17:19:13.259523 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "KubeAPIServerToKubeletClientCert"
    I0601 17:19:13.259587 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "LocalhostServing"
    I0601 17:19:13.259611 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "LocalhostServing"
    I0601 17:19:13.259626 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "ServiceNetworkServing"
    I0601 17:19:13.259929 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "ServiceNetworkServing"
    I0601 17:19:13.260079 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "ExternalLoadBalancerServing"
    I0601 17:19:13.260228 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "ExternalLoadBalancerServing"
    I0601 17:19:13.260236 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "InternalLoadBalancerServing"
    I0601 17:19:13.260251 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "InternalLoadBalancerServing"
    I0601 17:19:13.260257 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "LocalhostRecoveryServing"
    I0601 17:19:13.260262 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "LocalhostRecoveryServing"
    I0601 17:19:13.260266 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "KubeControllerManagerClient"
    I0601 17:19:13.260283 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "KubeControllerManagerClient"
    I0601 17:19:13.260288 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "KubeSchedulerClient"
    I0601 17:19:13.260462 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "KubeSchedulerClient"
    I0601 17:19:13.260492 1 certrotationcontroller.go:527] Finished waiting for CertRotation
    I0601 17:19:13.260509 1 kubecontrollermanagercertrotation.go:84] Waiting for CertRotation
    I0601 17:19:13.260529 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "CSRSigningCert"
    I0601 17:19:13.260560 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "CSRSigningCert"
    I0601 17:19:13.260685 1 kubecontrollermanagercertrotation.go:90] Finished waiting for CertRotation
    I0601 17:19:13.260768 1 regenerate_certificates.go:197] Refreshing certificates.
    I0601 17:19:13.272210 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SignerUpdateRequired' "aggregator-client-signer" in "openshift-kube-apiserver-operator" requires a new signing cert/key pair: past its latest possible time 2020-05-26 23:50:47.8 +0000 UTC
    I0601 17:19:13.703145 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'CABundleUpdateRequired' "kube-apiserver-aggregator-client-ca" in "openshift-config-managed" requires a new cert
    I0601 17:19:13.703174 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretUpdated' Updated Secret/aggregator-client-signer -n openshift-kube-apiserver-operator because it changed
    I0601 17:19:13.724121 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/kube-apiserver-aggregator-client-ca -n openshift-config-managed:
    cause by changes in data.ca-bundle.crt
    I0601 17:19:13.724943 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TargetUpdateRequired' "aggregator-client" in "openshift-kube-apiserver" requires a new target cert/key pair: past its latest possible time 2020-05-26 23:51:47.6 +0000 UTC
    I0601 17:19:14.372539 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretUpdated' Updated Secret/aggregator-client -n openshift-kube-apiserver because it changed
    I0601 17:19:14.373217 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TargetUpdateRequired' "localhost-serving-cert-certkey" in "openshift-kube-apiserver" requires a new target cert/key pair: already expired
    I0601 17:19:15.167110 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'SecretUpdateFailed' Failed to update Secret/localhost-serving-cert-certkey -n openshift-kube-apiserver: Operation cannot be fulfilled on secrets "localhost-serving-cert-certkey": the object has been modified; please apply your changes to the latest version and try again
    I0601 17:19:15.167606 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SignerUpdateRequired' "kube-control-plane-signer" in "openshift-kube-apiserver-operator" requires a new signing cert/key pair: past its refresh time 2020-06-01 04:37:54 +0000 UTC
    I0601 17:19:15.964111 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'SecretUpdateFailed' Failed to update Secret/kube-control-plane-signer -n openshift-kube-apiserver-operator: Secret "kube-control-plane-signer" is invalid: type: Invalid value: "kubernetes.io/tls": field is immutable
    I0601 17:19:16.164906 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretDeleted' Deleted Secret/kube-control-plane-signer -n openshift-kube-apiserver-operator
    [Operation cannot be fulfilled on secrets "localhost-serving-cert-certkey": the object has been modified; please apply your change to the latest version and try again, secrets "kube-control-plane-signer" already exists]
    Making call to close driver server
    (crc) Calling .Close
    (crc) DBG | Closing plugin on server side
    Successfully made call to close driver server
    Making call to close connection to plugin binary
    Making call to close driver server
    (crc) Calling .Close
    (crc) DBG | Closing plugin on server side
    Successfully made call to close driver server
    Making call to close connection to plugin binary
resolutioduplicate
Source
picture jriverol

Most helpful comment

+1 on this issue.

Getting a cert renewal issue:

I0601 18:23:02.017147       1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TargetUpdateRequired' "kube-scheduler-client-cert-key" in "openshift-config-managed" requires a new target cert/key pair: already expired
operation cannot be fulfilled on secrets "kube-scheduler-client-cert-key": the object has been modified; please apply your changes to the latest version and try again

I ensured the environment was clean by nuking ~/.crc and running crc delete and crc cleanup before starting from crc setup && crc start.

On my workstation, this is the latest 1.10 release of crc that I untar'd to /opt and copied the binary over to /usr/local/bin/ to be accessible on the $PATH.

$ md5sum /usr/local/bin/crc /opt/crc-linux-1.10.0-amd64/crc
3fe0d708ad31a22737c731093c73d23f  /usr/local/bin/crc 
3fe0d708ad31a22737c731093c73d23f  /opt/crc-linux-1.10.0-amd64/crc
$ md5sum ~/Downloads/crc-linux-amd64.tar.xz 
6bec664472006b84d77c53774890a06a  /home/tim/Downloads/crc-linux-amd64.tar.xz
picture timhdev on 1 Jun 2020
👍3

All 5 comments

+1 on this issue.

Getting a cert renewal issue:

I0601 18:23:02.017147       1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TargetUpdateRequired' "kube-scheduler-client-cert-key" in "openshift-config-managed" requires a new target cert/key pair: already expired
operation cannot be fulfilled on secrets "kube-scheduler-client-cert-key": the object has been modified; please apply your changes to the latest version and try again

I ensured the environment was clean by nuking ~/.crc and running crc delete and crc cleanup before starting from crc setup && crc start.

On my workstation, this is the latest 1.10 release of crc that I untar'd to /opt and copied the binary over to /usr/local/bin/ to be accessible on the $PATH.

$ md5sum /usr/local/bin/crc /opt/crc-linux-1.10.0-amd64/crc
3fe0d708ad31a22737c731093c73d23f  /usr/local/bin/crc 
3fe0d708ad31a22737c731093c73d23f  /opt/crc-linux-1.10.0-amd64/crc
$ md5sum ~/Downloads/crc-linux-amd64.tar.xz 
6bec664472006b84d77c53774890a06a  /home/tim/Downloads/crc-linux-amd64.tar.xz
timhdev picture timhdev on 1 Jun 2020
👍3

I tried to replicate with the 1.9.0 release found here: https://mirror.openshift.com/pub/openshift-v4/clients/crc/1.9.0/

This version works fine- I'm able to start a cluster and renew certs without issue:

$ crc version
crc version: 1.9.0+a68b5e0
OpenShift version: 4.3.10 (embedded in binary)
$ crc start
WARN A new version (1.10.0) has been published on https://cloud.redhat.com/openshift/install/crc/installer-provisioned 
INFO Checking if oc binary is cached              
INFO Checking if podman remote binary is cached   
INFO Checking if running as non-root              
INFO Checking if Virtualization is enabled        
INFO Checking if KVM is enabled                   
INFO Checking if libvirt is installed             
INFO Checking if user is part of libvirt group    
INFO Checking if libvirt is enabled               
INFO Checking if libvirt daemon is running        
INFO Checking if a supported libvirt version is installed 
INFO Checking if crc-driver-libvirt is installed  
INFO Checking if libvirt 'crc' network is available 
INFO Checking if libvirt 'crc' network is active  
INFO Checking if NetworkManager is installed      
INFO Checking if NetworkManager service is running 
INFO Checking if /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf exists 
INFO Checking if /etc/NetworkManager/dnsmasq.d/crc.conf exists 
? Copy it from https://cloud.redhat.com/openshift/install/crc/installer-provisioned
? Image pull secret [removed for brevity]
INFO Extracting bundle: crc_libvirt_4.3.10.crcbundle ... 
INFO Checking size of the disk image /home/tim/.crc/cache/crc_libvirt_4.3.10/crc.qcow2 ... 
INFO Creating CodeReady Containers VM for OpenShift 4.3.10... 
INFO Verifying validity of the cluster certificates ... 
INFO Check internal and public DNS query ...      
INFO Check DNS query from host ...                
INFO Copying kubeconfig file to instance dir ...  
INFO Cluster TLS certificates have expired, renewing them... [will take up to 5 minutes] 
INFO Adding user's pull secret ...                
INFO Updating cluster ID ...                      
INFO Starting OpenShift cluster ... [waiting 3m]  
INFO                                              
INFO To access the cluster, first set up your environment by following 'crc oc-env' instructions
# credentials follow below.

This narrows it down between tag 1.9->1.10, will have to look through the diffs to check for where the bug was introduced.

timhdev picture timhdev on 1 Jun 2020

Same issue here. Cert renewal failing in v1.10.0. But, worked fine when i tried with older version 1.9.0

purushdas picture purushdas on 2 Jun 2020

I did exactly the same this morning. I downgraded to versi贸n 1.9 and it worked without doing any change to the system. So clearly it's a problem with 1.10 version.

jriverol picture jriverol on 2 Jun 2020

Duplicate of #1242. We are aware of the issue and looking into this. A new release will be available soon to provide you with a working cluster with new certificates while we investigate why the renew process fails.

gbraad picture gbraad on 2 Jun 2020
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

rajivnathan picture rajivnathan  路  6Comments
bradfordcp picture bradfordcp  路  9Comments
skabashnyuk picture skabashnyuk  路  3Comments
omkensey picture omkensey  路  8Comments
Eyaldavid7 picture Eyaldavid7  路  8Comments