Installer: OpenShift 4.1.4 installation on vSphere failed: certificate has expired
Version
$ openshift-install version
openshift-install v4.1.4-201906271212-dirty
built from commit bf47826c077d16798c556b1bd143a5bbfac14271
release image quay.io/openshift-release-dev/ocp-release@sha256:a6c177eb007d20bb00bfd8f829e99bd40137167480112bd5ae1c25e40a4a163a
Platform:
vsphere
What happened?
Two weeks ago I created an OpenShift 4.1 cluster and it worked fine. Today I recreated it again using the same environment and I run into a certificate issue.
openshift-install wait-for bootstrap-complete --dir ./vsphere
INFO Waiting up to 30m0s for the Kubernetes API at https://api.openshift.corp.local:6443...
INFO Use the following commands to gather logs from the cluster
INFO openshift-install gather bootstrap --help
FATAL waiting for Kubernetes API: context deadline exceeded
Using the debug option I get
DEBUG Still waiting for the Kubernetes API: Get https://api.openshift.corp.local:6443/version?timeout=32s: x509: certificate has expired or is not yet valid
To be sure that I did not use an outdated version I reinstalled the latest RHCOS OVA template again and updated openshift-install to the version 4.1.4, but got the same result. Taking a look at https://api.openshift.corp.local:6443 also showed that the certificate is not valid anymore!
The .openshift_install.log shows the same message as above in the debugging output (i.e. x509: certificate has expired or is not yet valid).
The following command shows the same result:
# echo | openssl s_client -connect api.openshift.corp.local:6443 | openssl x509 -noout -text
depth=1 OU = openshift, CN = kube-apiserver-lb-signer
verify error:num=19:self signed certificate in certificate chain
DONE
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 4476027557613095924 (0x3e1e0b5940634bf4)
Signature Algorithm: sha256WithRSAEncryption
Issuer: OU = openshift, CN = kube-apiserver-lb-signer
Validity
Not Before: Jun 27 14:42:33 2019 GMT
Not After : Jun 28 14:42:37 2019 GMT
Subject: O = kube-master, CN = system:kube-apiserver
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:c6:1a:5e:cc:00:89:4d:4f:d6:ba:b3:ad:69:9c:
e4:58:13:2a:c6:28:56:8a:e2:af:01:da:c9:fb:90:
0b:5a:8e:a3:63:2d:07:03:10:a8:3b:94:e9:5f:8a:
1d:25:a0:1a:b4:82:50:43:6d:ec:9a:94:c5:03:9b:
3c:b9:15:16:41:41:8d:8a:d1:bb:4a:a1:07:af:6c:
ef:66:f6:be:10:74:b6:5f:51:be:77:f7:f3:17:e4:
d0:65:1b:f8:d5:4d:db:aa:18:6c:58:77:0a:47:44:
fb:aa:21:91:ad:72:95:05:5d:2f:c9:29:d2:78:a4:
27:d7:95:69:81:a2:43:40:ff:f8:c8:e9:4c:ef:75:
90:b5:11:d9:eb:3f:b2:66:7e:d5:aa:8f:a2:74:0e:
e6:bd:7d:1d:8c:97:fc:c6:77:74:e1:1a:08:c1:64:
c3:57:51:97:8b:a0:32:62:72:3e:b3:d0:2b:5b:f6:
bf:06:47:15:89:96:7a:d5:c5:1f:b6:eb:ec:6e:a8:
3c:6b:bb:90:b5:43:5b:50:af:3f:35:ad:35:e9:75:
7d:0f:77:2b:c4:1a:53:1e:b6:c7:58:3a:09:9f:b3:
5b:21:67:8a:0a:4d:bf:5a:b5:8f:2c:af:58:f9:20:
32:64:9a:30:73:80:31:8e:d6:da:79:2b:94:7f:cd:
99:cd
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
EE:3F:AC:6B:AE:A4:BF:DB:E7:02:10:93:4C:B2:40:09:D5:FB:24:E5
X509v3 Authority Key Identifier:
keyid:EE:3F:AC:6B:AE:A4:BF:DB:E7:02:10:93:4C:B2:40:09:D5:FB:24:E5
X509v3 Subject Alternative Name:
DNS:api.openshift.corp.local
Signature Algorithm: sha256WithRSAEncryption
be:48:22:b9:ab:ab:b5:fa:75:b4:ca:a3:22:0c:b4:69:b2:13:
6e:cf:9f:57:f7:26:c4:28:a7:52:8f:d1:bd:64:e1:a3:11:df:
43:76:56:dc:5f:b2:1f:bf:4f:53:ea:8f:eb:2a:4c:a0:08:46:
db:25:13:67:d7:b6:7d:f9:0a:44:a6:83:ed:d0:2b:8e:a3:6d:
84:70:ce:0a:cf:a6:99:6d:f6:b5:27:e8:20:fb:42:1c:27:2f:
50:1d:9a:e2:e3:63:38:11:a4:89:23:33:0a:00:f1:29:9a:5c:
f3:b8:7d:77:e8:8b:68:1c:26:34:a2:d6:96:b2:72:e1:91:51:
83:fd:9a:a7:1c:64:6c:bd:35:9e:a5:41:9f:02:52:4b:cd:27:
fb:8a:73:2c:6b:07:e1:92:03:50:a3:e4:8f:7b:fc:76:ed:f7:
d5:ff:31:7b:4d:ab:30:70:05:15:56:6b:22:8b:b3:e4:9e:a0:
47:cf:fc:59:4c:26:84:cb:a6:0f:f4:f8:b8:53:e0:5c:dd:6f:
20:1c:f3:e4:e5:77:63:a8:23:8a:19:e1:7e:cd:04:1d:63:41:
3d:c3:52:36:ec:5f:c8:86:17:4a:92:33:91:da:78:de:53:fa:
41:1a:3f:68:15:92:11:8f:8d:bc:ae:7b:2d:aa:dc:ec:29:a9:
a1:f6:5d:fe
What you expected to happen?
Successful installation of OpenShift 4.1.4 on vSphere (using a standard VLAN based port group).
How to reproduce it (as minimally and precisely as possible)?
It was a standard Terraform 0.11.14 based installation on vSphere according to the docs using a VLAN based port group and can probably be recreated very easy since it a certificate issue (not an installer issue).
References
1792 has a similar issue, but in my case it happens during the installation. Is there a quick fix for that?
tschwaller
All 11 comments
If you don't reuse the install for for new cluster the installer will generate new assets for your new cluster.
abhinavdahiya
on 9 Jul 2019
I got the issue with just the old config, but I deleted everything before that (even using a new git clone of the installer github repo) and recreated everything from scratch (ignition files, terraform apply, etc.). I do not understand where I got this cert with validity until June 28. Is the installer using preinstalled certs somewhere (e.g. on the OVA) or what can I do do to avoid that. I do not fully understand your comment, but thanks for any help.
tschwaller
on 9 Jul 2019
I ran into the same issue, some comments from my side:
- Do not reuse old config dir's. I strongly recommand
openshift-install create .. --dir=config/, if you delete the cluster and want to reinstall, delete the config/ dir and runopenshift-install create .. --dir=config/. The background is the installer creates certificates they are only valid for round about 24 hours, at some point the installer reuse some resources in the config/ dir. - Do not shutdown (stopping the VM's) your cluster in the first 24 hours! Wait for 24h hours, check the csr [1] if anything is approved you can shutdown your cluster if you like.
- If something is strange in your cluster, for example, you get a TLS error if want to watch pod logs via oc log then please check the certificates requests [1]
[1] https://docs.openshift.com/container-platform/4.1/installing/installing_vsphere/installing-vsphere.html#installation-approve-csrs_installing-vsphere
rbo
on 11 Jul 2019
@tschwaller @rbo If you want to reuse the directory you need to delete the hidden file .openshift_install_state.json that file contains the certificate information for the cluster.
Rikbruggink
on 12 Jul 2019
I will check the whole setup again and see if by mistake I used an old bootstrap.ign file, because I started several times from scratch (as described above, hanks for the hints!). Will report if it works....
tschwaller
on 12 Jul 2019
I believe I'm seeing the same or a similar problem. I'm attempting to deploy a cluster from scratch on vSphere (clean directory, no existing files) and the cluster nodes are unable to boot due to Ignition complaining about the certificate for the API.
Bootstrap server log:
$ journalctl -b -f -u bootkube.service
-- Logs begin at Fri 2019-07-19 11:18:51 UTC. --
Jul 19 11:19:13 openshift-bootstrap systemd[1]: Started Bootstrap a Kubernetes cluster.
Jul 19 11:19:13 openshift-bootstrap bootkube.sh[1488]: Pulling release image...
Jul 19 16:20:34 openshift-bootstrap bootkube.sh[1488]: 76eaac3ae928319b40d185bcbaa52edc2c660f4df6591dabd95e236c2ca979bd
Jul 19 16:20:48 openshift-bootstrap bootkube.sh[1488]: Rendering Cluster Version Operator Manifests...
Jul 19 16:20:48 openshift-bootstrap bootkube.sh[1488]: Rendering cluster config manifests...
Jul 19 16:20:51 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_scheduler.crd.yaml
Jul 19 16:20:51 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/config-bootstrap/manifests/0000_03_authorization-openshift_01_rolebindingrestriction.crd.yaml
Jul 19 16:20:51 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/config-bootstrap/manifests/0000_03_quota-openshift_01_clusterresourcequota.crd.yaml
Jul 19 16:20:51 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_build.crd.yaml
Jul 19 16:20:51 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_network.crd.yaml
Jul 19 16:20:51 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_project.crd.yaml
Jul 19 16:20:51 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/config-bootstrap/manifests/0000_10_quota-openshift_01_clusterresourcequota.crd.yaml
Jul 19 16:20:51 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_dns.crd.yaml
Jul 19 16:20:51 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_image.crd.yaml
Jul 19 16:20:51 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_infrastructure.crd.yaml
Jul 19 16:20:51 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_oauth.crd.yaml
Jul 19 16:20:51 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/config-bootstrap/manifests/0000_03_security-openshift_01_scc.crd.yaml
Jul 19 16:20:51 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_apiserver.crd.yaml
Jul 19 16:20:51 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_authentication.crd.yaml
Jul 19 16:20:51 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_featuregate.crd.yaml
Jul 19 16:20:51 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_02_config.clusterrole.yaml
Jul 19 16:20:51 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_console.crd.yaml
Jul 19 16:20:51 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_ingress.crd.yaml
Jul 19 16:20:51 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_openshift-config-managed-ns.yaml
Jul 19 16:20:51 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_openshift-config-ns.yaml
Jul 19 16:20:51 openshift-bootstrap bootkube.sh[1488]: Rendering Kubernetes API server core manifests...
Jul 19 16:20:55 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/kube-apiserver-bootstrap/bootstrap-manifests/kube-apiserver-pod.yaml
Jul 19 16:20:55 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/configmap-csr-controller-ca.yaml
Jul 19 16:20:55 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/configmap-sa-token-signing-certs.yaml
Jul 19 16:20:55 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-kube-apiserver-to-kubelet-signer.yaml
Jul 19 16:20:55 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-loadbalancer-serving-signer.yaml
Jul 19 16:20:55 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-localhost-serving-signer.yaml
Jul 19 16:20:55 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-service-network-serving-signer.yaml
Jul 19 16:20:55 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/cluster-role-binding-kube-apiserver.yaml
Jul 19 16:20:55 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/configmap-admin-kubeconfig-client-ca.yaml
Jul 19 16:20:55 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/cluster-role-kube-apiserver.yaml
Jul 19 16:20:55 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-aggregator-client-signer.yaml
Jul 19 16:20:55 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/secret-control-plane-client-signer.yaml
Jul 19 16:20:55 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/00_openshift-kube-apiserver-ns.yaml
Jul 19 16:20:55 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/kube-apiserver-bootstrap/manifests/00_openshift-kube-apiserver-operator-ns.yaml
Jul 19 16:20:55 openshift-bootstrap bootkube.sh[1488]: Rendering Kubernetes Controller Manager core manifests...
Jul 19 16:20:59 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/kube-controller-manager-bootstrap/bootstrap-manifests/kube-controller-manager-pod.yaml
Jul 19 16:20:59 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/kube-controller-manager-bootstrap/manifests/00_openshift-kube-controller-manager-ns.yaml
Jul 19 16:20:59 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/kube-controller-manager-bootstrap/manifests/00_openshift-kube-controller-manager-operator-ns.yaml
Jul 19 16:20:59 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/kube-controller-manager-bootstrap/manifests/secret-csr-signer-signer.yaml
Jul 19 16:20:59 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/kube-controller-manager-bootstrap/manifests/secret-initial-kube-controller-manager-service-account-private-key.yaml
Jul 19 16:20:59 openshift-bootstrap bootkube.sh[1488]: Rendering Kubernetes Scheduler core manifests...
Jul 19 16:21:03 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/kube-scheduler-bootstrap/bootstrap-manifests/kube-scheduler-pod.yaml
Jul 19 16:21:03 openshift-bootstrap bootkube.sh[1488]: Writing asset: /assets/kube-scheduler-bootstrap/manifests/00_openshift-kube-scheduler-ns.yaml
Jul 19 16:21:03 openshift-bootstrap bootkube.sh[1488]: Rendering MCO manifests...
Jul 19 16:21:06 openshift-bootstrap bootkube.sh[1488]: I0719 16:21:06.344439 1 bootstrap.go:86] Version: 4.1.4-201906271212-dirty (02c07496ba0417b3e12b78fb32baf6293d314f79)
Jul 19 16:21:06 openshift-bootstrap bootkube.sh[1488]: I0719 16:21:06.347754 1 bootstrap.go:141] manifests/machineconfigcontroller/controllerconfig.yaml
Jul 19 16:21:06 openshift-bootstrap bootkube.sh[1488]: I0719 16:21:06.349444 1 bootstrap.go:141] manifests/master.machineconfigpool.yaml
Jul 19 16:21:06 openshift-bootstrap bootkube.sh[1488]: I0719 16:21:06.349642 1 bootstrap.go:141] manifests/worker.machineconfigpool.yaml
Jul 19 16:21:06 openshift-bootstrap bootkube.sh[1488]: I0719 16:21:06.349828 1 bootstrap.go:141] manifests/bootstrap-pod-v2.yaml
Jul 19 16:21:06 openshift-bootstrap bootkube.sh[1488]: I0719 16:21:06.350112 1 bootstrap.go:141] manifests/machineconfigserver/csr-bootstrap-role-binding.yaml
Jul 19 16:21:06 openshift-bootstrap bootkube.sh[1488]: I0719 16:21:06.350343 1 bootstrap.go:141] manifests/machineconfigserver/kube-apiserver-serving-ca-configmap.yaml
Jul 19 16:21:06 openshift-bootstrap bootkube.sh[1488]: Starting etcd certificate signer...
Jul 19 16:21:09 openshift-bootstrap bootkube.sh[1488]: 711d0ba4e5233eda1577d381be0c3c5b3bca6fded1a2fa737e38ebb7fdc89c8e
Jul 19 16:21:09 openshift-bootstrap bootkube.sh[1488]: Waiting for etcd cluster...
OpenShift installer log:
$ openshift-install wait-for bootstrap-complete --dir staging --log-level debug
DEBUG OpenShift Installer v4.1.4-201906271212-dirty
DEBUG Built from commit bf47826c077d16798c556b1bd143a5bbfac14271
INFO Waiting up to 30m0s for the Kubernetes API at https://api.openshift.corp.local:6443...
DEBUG Still waiting for the Kubernetes API: the server could not find the requested resource
DEBUG Still waiting for the Kubernetes API: the server could not find the requested resource
DEBUG Still waiting for the Kubernetes API: the server could not find the requested resource
Log from the console of one of the master servers:
ignition[674]: GET https://api-int.openshift.corp.local:22623/config/master: attempt #100
ignition[674]: GET error: Get https://api-int.openshift.corp.local:22623/config/master: x509: certificate has expired or is not yet valid
And finally, inspecting the certificate shows the correct validity:
$ echo | openssl s_client -connect api-int.openshift.corp.local:6443 | openssl x509 -noout -text
...
Validity
Not Before: Jul 19 16:17:45 2019 GMT
Not After : Jul 20 16:17:49 2019 GMT
...
Just like the OP, I created a cluster the day before and it was working fine, but every attempt since then with clean directories (no existing files, including hidden files) has resulted in the servers being unable to boot due to the certificate problem.
jaredk-beta
on 19 Jul 2019
Right after I posted the above I noticed that the bootstrap server log showed a significant time change. Turns out the vSphere hosts I was attempting to run the cluster on were behind in their clocks by several hours :man_facepalming:. So since the cluster nodes hadn't had a chance to run NTP and update their clocks they weren't able to see the certificates as valid. Running the process over again with updated clocks and it's working normally so far.
jaredk-beta
on 19 Jul 2019
I did the installation again in at least 3 different setups (including the one which failed), but this time I made sure that my bootstrap.ign file was updated (I think this was the error I did the first time) and all three worked.
You just have to know that the created SSL certs for the installation are only valid until the next day and make sure they are not used somewhere if you use the same setup later on (which I did). Maybe this can be made a little bit more visible in the docs.
The ticket can be closed.
Thanks for the help and explanations!
tschwaller
on 26 Jul 2019
Same issue happened for install OCP 4.2 on bare metal, and recreate the
wanghh2000
on 5 Nov 2019
/close
abhinavdahiya
on 5 Nov 2019
@abhinavdahiya: Closing this issue.
In response to this:
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
openshift-ci-robot
on 5 Nov 2019
Related issues
alexouzounis
路
12Comments
deanpeterson
路
3Comments
davivcgarcia
路
5Comments
fridim
路
11Comments
tomassedovic
路
12Comments
Most helpful comment
@tschwaller @rbo If you want to reuse the directory you need to delete the hidden file .openshift_install_state.json that file contains the certificate information for the cluster.