Crc: [BUG] CRC cannot start

[dobharweim]

General information

• OS: macOS
• Hypervisor: hyperkit
• Did you run crc setup before starting it (Yes/No)? Yes
• Running CRC on: Laptop

CRC version

crc version: 1.10.0+9025021
OpenShift version: 4.4.3 (embedded in binary)

CRC status

DEBU Running '/Users/daithi/.crc/bin/oc get co -ojson --kubeconfig /Users/daithi/.crc/machines/crc/kubeconfig --context admin --cluster crc'
DEBU operator-lifecycle-manager-packageserver operator not available, Reason:
DEBU operator-lifecycle-manager-packageserver operator is still progressing, Reason:
DEBU About to run SSH command:
df -B1 --output=size,used,target /sysroot | tail -1
DEBU SSH cmd err, output: <nil>: 32719437824 9883750400 /sysroot
CRC VM:          Running
OpenShift:       Starting
Disk Usage:      9.884GB of 32.72GB (Inside the CRC VM)
Cache Usage:     12.05GB
Cache Directory: /Users/daithi/.crc/cache

CRC config

No custom config properties set, blank result.

Host Operating System

ProductName:    Mac OS X
ProductVersion: 10.15.3
BuildVersion:   19D76

Steps to reproduce

1. Extract crc tar.
2. Run ./crc setup.
3. Run ./crc start.
4. Provide pull secret.

Expected

CRC cluster to start.

Actual

CRC fails to start with the following error:

INFO Checking if oc binary is cached INFO Checking if podman remote binary is cached INFO Checking if running as non-root INFO Checking if HyperKit is installed INFO Checking if crc-driver-hyperkit is installed INFO Checking file permissions for /etc/resolver/testing INFO Checking file permissions for /etc/hosts ? Image pull secret [? for help] *********************************************************************************************************************************************************************************************************************************************************************************************************************************INFO Extracting bundle: crc_hyperkit_4.4.3.crcbundle ... *************************************************************INFO Checking size of the disk image /Users/daithi/.crc/cache/crc_hyperkit_4.4.3/crc.qcow2 ... ***********************INFO Creating CodeReady Containers VM for OpenShift 4.4.3... *********************************************************INFO CodeReady Containers VM is running ********************************************************************INFO Verifying validity of the cluster certificates ... **************************************************************INFO Network restart not needed ********************************************************************INFO Check internal and public DNS query ... ********************************************************************INFO Check DNS query from host ... ********************************************************************INFO Generating new SSH key ********************************************************************INFO Copying kubeconfig file to instance dir ... ********************************************************************INFO Cluster TLS certificates have expired, renewing them... [will take up to 5 minutes] *****************************ERRO Failed to renew TLS certificates: ssh command error:*************************************************************command : sudo podman run -it --rm --network=host -v /etc/kubernetes/:/etc/kubernetes/:Z --entrypoint=/usr/bin/cluster-kube-apiserver-operator 'openshift/cert-recovery' regenerate-certificates********************************************err : exit status 1***********************************************************************************************output : I0602 12:05:00.422802 1 certrotationcontroller.go:505] Waiting for CertRotation***********************I0602 12:05:00.723176 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "AggregatorProxyClientCert"******************************************************************************************************I0602 12:05:00.723208 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "AggregatorProxyClientCert"*********************************************************************************************I0602 12:05:00.723214 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "KubeAPIServerToKubeletClientCert" I0602 12:05:00.723218 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "KubeAPIServerToKubeletClientCert" I0602 12:05:00.723222 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "LocalhostServing" I0602 12:05:00.723225 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "LocalhostServing" I0602 12:05:00.723228 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "ServiceNetworkServing" I0602 12:05:00.723237 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "ServiceNetworkServing" I0602 12:05:00.723242 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "ExternalLoadBalancerServing" I0602 12:05:00.723245 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "ExternalLoadBalancerServing" I0602 12:05:00.723250 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "InternalLoadBalancerServing" I0602 12:05:00.723254 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "InternalLoadBalancerServing" I0602 12:05:00.723257 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "LocalhostRecoveryServing" I0602 12:05:00.723297 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "LocalhostRecoveryServing" I0602 12:05:00.723302 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "KubeControllerManagerClient" I0602 12:05:00.723324 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "KubeControllerManagerClient" I0602 12:05:00.723329 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "KubeSchedulerClient" I0602 12:05:00.723333 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "KubeSchedulerClient" I0602 12:05:00.723340 1 certrotationcontroller.go:527] Finished waiting for CertRotation I0602 12:05:00.723360 1 kubecontrollermanagercertrotation.go:84] Waiting for CertRotation I0602 12:05:00.723364 1 client_cert_rotation_controller.go:121] Waiting for CertRotationController - "CSRSigningCert" I0602 12:05:00.723369 1 client_cert_rotation_controller.go:128] Finished waiting for CertRotationController - "CSRSigningCert" I0602 12:05:00.723410 1 kubecontrollermanagercertrotation.go:90] Finished waiting for CertRotation I0602 12:05:00.723417 1 regenerate_certificates.go:197] Refreshing certificates. I0602 12:05:00.723609 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SignerUpdateRequired' "aggregator-client-signer" in "openshift-kube-apiserver-operator" requires a new signing cert/key pair: already expired I0602 12:05:00.982444 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretUpdated' Updated Secret/aggregator-client-signer -n openshift-kube-apiserver-operator because it changed I0602 12:05:00.983032 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'CABundleUpdateRequired' "kube-apiserver-aggregator-client-ca" in "openshift-config-managed" requires a new cert I0602 12:05:01.228632 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/kube-apiserver-aggregator-client-ca -n openshift-config-managed: cause by changes in data.ca-bundle.crt I0602 12:05:01.228815 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TargetUpdateRequired' "aggregator-client" in "openshift-kube-apiserver" requires a new target cert/key pair: already expired I0602 12:05:02.030705 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretUpdated' Updated Secret/aggregator-client -n openshift-kube-apiserver because it changed I0602 12:05:02.030814 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TargetUpdateRequired' "kubelet-client" in "openshift-kube-apiserver" requires a new target cert/key pair: already expired I0602 12:05:02.840718 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretUpdated' Updated Secret/kubelet-client -n openshift-kube-apiserver because it changed I0602 12:05:02.840857 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TargetUpdateRequired' "localhost-serving-cert-certkey" in "openshift-kube-apiserver" requires a new target cert/key pair: already expired I0602 12:05:03.629522 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretUpdated' Updated Secret/localhost-serving-cert-certkey -n openshift-kube-apiserver because it changed I0602 12:05:03.630924 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TargetUpdateRequired' "service-network-serving-certkey" in "openshift-kube-apiserver" requires a new target cert/key pair: already expired I0602 12:05:04.428975 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretUpdated' Updated Secret/service-network-serving-certkey -n openshift-kube-apiserver because it changed I0602 12:05:04.429486 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TargetUpdateRequired' "external-loadbalancer-serving-certkey" in "openshift-kube-apiserver" requires a new target cert/key pair: already expired I0602 12:05:05.229376 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'SecretUpdateFailed' Failed to update Secret/external-loadbalancer-serving-certkey -n openshift-kube-apiserver: Operation cannot be fulfilled on secrets "external-loadbalancer-serving-certkey": the object has been modified; please apply your changes to the latest version and try again I0602 12:05:05.229713 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TargetUpdateRequired' "internal-loadbalancer-serving-certkey" in "openshift-kube-apiserver" requires a new target cert/key pair: already expired I0602 12:05:06.027584 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretUpdated' Updated Secret/internal-loadbalancer-serving-certkey -n openshift-kube-apiserver because it changed I0602 12:05:06.029891 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SignerUpdateRequired' "kube-control-plane-signer" in "openshift-kube-apiserver-operator" requires a new signing cert/key pair: past its refresh time 2020-06-01 04:37:54 +0000 UTC I0602 12:05:06.827579 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'SecretUpdateFailed' Failed to update Secret/kube-control-plane-signer -n openshift-kube-apiserver-operator: Secret "kube-control-plane-signer" is invalid: type: Invalid value: "kubernetes.io/tls": field is immutable I0602 12:05:07.028686 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretDeleted' Deleted Secret/kube-control-plane-signer -n openshift-kube-apiserver-operator I0602 12:05:07.428604 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'SecretCreateFailed' Failed to create Secret/kube-control-plane-signer -n openshift-kube-apiserver-operator: secrets "kube-control-plane-signer" already exists I0602 12:05:07.428874 1 event.go:281] Event(v1.ObjectReference{Kind:"namespace", Namespace:"openshift-kube-apiserver-operator", Name:"openshift-kube-apiserver-operator", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'CABundleUpdateRequired' "kube-control-plane-signer-ca" in "openshift-kube-apiserver-operator" requires a new cert [Operation cannot be fulfilled on secrets "external-loadbalancer-serving-certkey": the object has been modified; please apply your changes to the latest version and try again, secrets "kube-control-plane-signer" already exists, Operation cannot be fulfilled on configmaps "kube-control-plane-signer-ca": the object has been modified; please apply your ch nges to the latest version and try again]

Logs

Before gather the logs try following if that fix your issue

$ crc delete -f
$ crc cleanup
$ crc setup
$ crc start --log-level debug

The output of crc start --log-level debug, after running through the above commands, (delete -f, cleanup, setup etc.) can be found at: https://gist.github.com/dobharweim/1c4929188b6128f10f85fe0586cee365

[cfergeau]

This is a known issue, see https://github.com/code-ready/crc/issues/1242

[gbraad]

Duplicate of #1242. We are aware of the issue and looking into this. A new release will be available soon to provide you with a working cluster with new certificates while we investigate why the renew process fails.