Kubevirt: failed calling webhook "virtualmachine-validator.kubevirt.io" - x509: certificate has expired when calling

[jaimehrubiks]

What happened:

I deployed sample vm.yaml file and get the following error:

_Error from server (InternalError): error when creating "vm.yaml": Internal error occurred: failed calling webhook "virtualmachine-validator.kubevirt.io": Post https://virt-api.kubevirt.svc:443/virtualmachines-validate?timeout=30s: x509: certificate has expired or is not yet valid_

I don't understand what is this certificate doing and why is it using https instead of http. Am I suppose to setup certificates somewhere? Maybe I am missing something.

How to reproduce it (as minimally and precisely as possible):

I just installed kubevirt and configured the daemonset to use a nodeSelector to the server I have KVM installed on

Anything else we need to know?:

I have already tried removing kubevirt and installing it again from zero.

$ k -n kubevirt get pods
_NAME READY STATUS RESTARTS AGE
virt-api-7c6b9f4569-77rwn 1/1 Running 0 9m27s
virt-api-7c6b9f4569-mkcvb 1/1 Running 0 9m27s
virt-controller-868975d87f-722qb 1/1 Running 0 9m6s
virt-controller-868975d87f-ndl52 1/1 Running 0 9m6s
virt-handler-8vzkn 1/1 Running 0 7m11s
virt-operator-59f6fccd-mlp5w 1/1 Running 0 9m35s
virt-operator-59f6fccd-tl7kp 1/1 Running 0 9m35s_

$ k -n kubevirt logs virt-api-7c6b9f4569-77rwn
...
{"component":"virt-api","level":"info","msg":"http: TLS handshake error from 10.233.69.0:38475: remote error: tls: bad certificate\n","pos":"server.go:3012","timestamp":"2020-04-30T19:07:30.657100Z"}

(10.223 is my pod cidr)

Of course if I do port forwarding and test the api I get the similar error
curl https://localhost:9000/apis/ -v
_* Trying ::1...

• TCP_NODELAY set
• Connected to localhost (::1) port 9000 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• successfully set certificate verify locations:
• CAfile: /etc/ssl/cert.pem
  CApath: none
• TLSv1.2 (OUT), TLS handshake, Client hello (1):
• TLSv1.2 (IN), TLS handshake, Server hello (2):
• TLSv1.2 (IN), TLS handshake, Certificate (11):
• TLSv1.2 (OUT), TLS alert, unknown CA (560):
• SSL certificate problem: unable to get local issuer certificate
• Closing connection 0
  curl: (60) SSL certificate problem: unable to get local issuer certificate
  More details here: https://curl.haxx.se/docs/sslcerts.html_

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Environment:

• KubeVirt version (use virtctl version): v0.26.5
• Kubernetes version (use kubectl version): 1.15.3
• VM or VMI specifications:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[jaimehrubiks]

Installing kemu/kvm I did yum update, which probably messed up my docker or kubelet or something that broke my certificates. Reinstalling the cluster with kubespray fixed my issues.

[hsychla]

We are facing the same problem but have not touched anything on that cluster other than deploying VMs.
The cert expired ~2.5 weeks ago.
Is there a solution to this issue that does not require us to reinstall a production cluster?

[rmohr]

@hsychla which version are you using?

[hsychla]

v0.19.0.
quite old, I know. but since this is a production cluster, we did not have a chace to update.

I am currently in the process of upgrading to 0.20.5.
But the only mention of cert was in release notes of 0.29.0?
So would we need to go there?

[rmohr]

@hsychla all certificates should be renewed if you delete all our control-plane pods in these old versions. Just do

kubectl delete secrets --namespace kubevirt -l kubevirt.io
kubectl delete pods --namespace kubevirt -l kubevirt.io

Once the pods were recreated everything should work again.

[rmohr]

I am currently in the process of upgrading to 0.20.5.
But the only mention of cert was in release notes of 0.29.0?
So would we need to go there?

@hsychla I can recommend 0.30.5 (except if you are on gce, there is a qemu issue in there which makes it not work there).

[hsychla]

@hsychla all certificates should be renewed if you delete all our control-plane pods in these old versions. Just do

kubectl delete secrets --namespace kubevirt -l kubevirt.io
kubectl delete pods --namespace kubevirt -l kubevirt.io

Once the pods were recreated everything should work again.

This worked nicely, thanks!