Crc: [BUG] Certificates do not renew successfully using crc v1.10

[btannous]

General information

• OS: Linux (fedora 31)
• Hypervisor: KVM
• Did you run crc setup before starting it (Yes/No)? Yes (always)
• Running CRC on: Desktop, 8core, 32g memory, nothing else running (accessing remotely)

CRC version

❯ ./crc version
crc version: 1.10.0+9025021
OpenShift version: 4.4.3 (embedded in binary)

Host Operating System

❯ cat /etc/os-release
NAME=Fedora
VERSION="31 (Server Edition)"
ID=fedora
VERSION_ID=31
VERSION_CODENAME=""
PLATFORM_ID="platform:f31"
PRETTY_NAME="Fedora 31 (Server Edition)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:31"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f31/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=31
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=31
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Server Edition"
VARIANT_ID=server

Steps to reproduce

• blocked UDP port 123 out of my local gateway
• sudo date -s "9 JULY 2020 11:11:11"
• downloaded/extracted crc-1.10
• crc delete
• crc setup
• crc start -c 8 -m 25000 ...

Expected

Certificates get renewed and cluster comes up.

Actual

Certificates do not get renewed.

Logs

https://gist.github.com/dea1a60dddaf7becd05200d0a2eed3e5

https://gist.github.com/btannous/1ebb796dbb747a2f18294403aea6c15d

[gbraad]

We are currently investigating this ...

[cfergeau]

• blocked UDP port 123 out of my local gateway

Turns out this can be done directly for the crc VM with libvirt on linux

$ cat drop-ntp.xml
<filter name='drop-ntp' chain='ipv4'>
  <rule action='drop' direction='out' >
    <ip protocol='udp' dstportstart='123'/>
  </rule>
  <rule action='drop' direction='in' priority='100'>
    <ip protocol='udp' srcportstart='123'/>
  </rule>
</filter>

$ virsh -c qemu:///system net nwfilter-define drop-ntp.xml
$ virsh -c qemu:///system edit crc

Then, in crc XML definitions, a <filterref> element needs to be added:

<domain>
  ...
  <devices>
  ...
    <interface type='network'>
      ...
      <filterref filter='drop-ntp'/>
    </interface>

With that in place, we can run the cluster "in the future" by changing this in the domain xml:

<clock offset='variable' adjustment='30000' basis='utc'>

'adjustment' being a value in seconds. I need to experiment a bit more with this, as to exercise the cert recovery code, we probably need to change the time on the host too, and with NTP blocked, the cluster will probably sync with the host time without needing any changes to that <clock> element.

[BenHall]

The certificates expired this morning and it looks like this has had an impact. Any suggestions?

[btannous]

https://docs.openshift.com/container-platform/4.4/backup_and_restore/disaster_recovery/scenario-3-expired-certs.html
~I saw this, not sure if the process changed from pre 4.4 / 4.3 times?~ (edit: https://github.com/openshift/openshift-docs/blame/enterprise-4.4/modules/dr-recover-expired-control-plane-certs.adoc not too many updates there.)

Unless we are missing the arch tag https://github.com/openshift/openshift-docs/commit/2428fd8d5ca59fc4714a273b0f75cc825fea24e2

Working on a debug log for this.

[btannous]

https://gist.github.com/dea1a60dddaf7becd05200d0a2eed3e5 : see the updated debug logs.

[btannous]

~Is this the issue (or part of it)?~
https://github.com/code-ready/snc/blob/35cb5b98bd6575d7e553e23ba337f9e59812fac6/createdisk.sh#L277-L278

❯ pwd
/home/b/.crc/machines/crc
❯ ls
config.json  crc  id_rsa  id_rsa.pub  kubeconfig
❯ export KUBECONFIG=kubeconfig
❯ oc adm release info --config=config.json --image-for=cluster-kube-apiserver-operator
Flag --config has been deprecated, use --kubeconfig instead
error: info expects one argument, or a connection to an OpenShift 4.x server: invalid configuration: Missing or incomplete configuration info.  Please login or point to an existing, complete config file:

  1. Via the command-line flag --kubeconfig
  2. Via the KUBECONFIG environment variable
  3. In your home directory as ~/.kube/config

To view or setup config directly use the 'config' command.
❯ oc adm release info --image-for=cluster-kube-apiserver-operator
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1716497ba22b107f22c4d80d7c1e15424aa171f0592dc764d39c0e54c7f5225e

~seems possible:~
https://docs.openshift.com/container-platform/4.4/release_notes/ocp-4-4-release-notes.html#ocp-4-4-oc-config-flag

---

edit: it looks like that image is correct, so that is being handled properly, i think

[core@crc-dv9sm-master-0 static-pod-resources]$ sudo podman tag quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1716497ba22b107f22c4d80d7c1e15424aa171f0592dc764d39c0e54c7f5225e test

[core@crc-dv9sm-master-0 static-pod-resources]$ sudo podman images
REPOSITORY                                       TAG      IMAGE ID       CREATED        SIZE
quay.io/openshift-release-dev/ocp-release        <none>   811ff79e21a7   4 weeks ago    306 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   48d1e0743da6   5 weeks ago    773 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   e0a9eca3cd52   5 weeks ago    454 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   11b9012c799c   5 weeks ago    299 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   430b567dd9f8   5 weeks ago    651 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   5619baf8a8f9   5 weeks ago    297 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   ade76e3ee5cc   5 weeks ago    304 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   942f6ce61a45   5 weeks ago    327 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   d8f66027d90f   5 weeks ago    319 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   f88b5dcd9448   5 weeks ago    338 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   bb0d154349fb   5 weeks ago    328 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   84e10fc72c79   5 weeks ago    354 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   c9e8f4ea8a08   5 weeks ago    456 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   5fa68c0a4e10   5 weeks ago    306 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   2949ac0cd3d4   5 weeks ago    302 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   6840e8b5ffee   5 weeks ago    301 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   d0ee246754ce   5 weeks ago    300 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   0f676a7a42d3   5 weeks ago    293 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   6016739201d8   5 weeks ago    287 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   ee2b94ec164f   5 weeks ago    319 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   add8db87608d   5 weeks ago    322 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   15bf2baf4f05   5 weeks ago    302 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   0e40f49bfd1c   5 weeks ago    277 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   7a2767c54f13   5 weeks ago    318 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   1da13fa25328   5 weeks ago    364 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   4d25486db835   5 weeks ago    283 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   33209706ef64   5 weeks ago    429 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   84015ae3241d   5 weeks ago    324 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   8205b6710f0c   5 weeks ago    300 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   b8bd50d4c2a2   5 weeks ago    290 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   ac63e1fdbd6e   5 weeks ago    322 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   ebcf902e9314   5 weeks ago    467 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   85310c0efe95   5 weeks ago    300 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   8d03b22167bb   5 weeks ago    292 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   5fa7f3523cbd   5 weeks ago    326 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   60976debb9fa   5 weeks ago    319 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   d192ebdd67d0   5 weeks ago    302 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   505e413f9df1   5 weeks ago    303 MB
localhost/test                                   latest   505e413f9df1   5 weeks ago    303 MB # SEE THIS ONE!! IT MATCHES THE cert-recovery
localhost/openshift/cert-recovery                latest   505e413f9df1   5 weeks ago    303 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   97786da87789   5 weeks ago    300 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   dba244c934d3   5 weeks ago    251 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   c8821846d41d   5 weeks ago    343 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   57c7d5055a16   5 weeks ago    287 MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev   <none>   34c32d0059eb   5 weeks ago    237 MB
quay.io/crcont/dnsmasq                           latest   72e07c0aff7b   3 months ago   254 MB

---

edit: maybe this is why? it doesnt look like the /etc/kubernetes/static-pod-resources/recovery-kube-apiserver-pod/ folder exists on the crc 1.10 image (maybe that is normal and gets generated when trying to renew or something else?)
see: https://gist.github.com/btannous/dea1a60dddaf7becd05200d0a2eed3e5#file-crc1-10-log-L190

[core@crc-dv9sm-master-0 static-pod-resources]$ pwd
/etc/kubernetes/static-pod-resources
[core@crc-dv9sm-master-0 static-pod-resources]$ ls
configmaps   kube-apiserver-certs   kube-apiserver-pod-5  kube-controller-manager-certs   kube-controller-manager-pod-5  kube-scheduler-certs  kube-scheduler-pod-7
etcd-certs   kube-apiserver-pod-10  kube-apiserver-pod-6  kube-controller-manager-pod-10  kube-controller-manager-pod-6  kube-scheduler-pod-2  kube-scheduler-pod-8
etcd-member  kube-apiserver-pod-11  kube-apiserver-pod-7  kube-controller-manager-pod-11  kube-controller-manager-pod-7  kube-scheduler-pod-3  kube-scheduler-pod-9
etcd-pod-2   kube-apiserver-pod-12  kube-apiserver-pod-8  kube-controller-manager-pod-2   kube-controller-manager-pod-8  kube-scheduler-pod-5
etcd-pod-3   kube-apiserver-pod-4   kube-apiserver-pod-9  kube-controller-manager-pod-3   kube-controller-manager-pod-9  kube-scheduler-pod-6
[core@crc-dv9sm-master-0 static-pod-resources]$ sudo podman run -it --rm --network=host -v /etc/kubernetes/:/etc/kubernetes/:Z --entrypoint=/usr/bin/cluster-kube-apiserver-operator 'openshift/cert-recovery' regenerate-certificates
failed to get kubernetes rest config: open /etc/kubernetes/static-pod-resources/recovery-kube-apiserver-pod/admin.kubeconfig: no such file or directory
[core@crc-dv9sm-master-0 static-pod-resources]$ 

[phantomjinx]

@btannous
Have the same issue and been fighting it all today ...
Bit of debugging shows ...

1. The directory /etc/kubernetes/static-pod-resources/recovery-kube-apiserver-pod is created during the start of the recovery pod here
2. The pod then generates the new certificates here
3. The pod is deferred for removal (so on exit), including its config here

Therefore, once the recovery pod is destroyed the config directory is removed as well.

It is possible to run the command, manually, that created the pod again but the new certificates generated in the admin.kubeconfig are not signed by the correct CA so running any oc command after just errors out with unknown certificate-authority errors.

[gbraad]

It seems version 1.8.0 and 1.9.0 do not have this issue, so it is related to the latest renew implementation. See: https://github.com/code-ready/crc/issues/1270#issuecomment-637259954 https://github.com/code-ready/crc/issues/1271#issuecomment-637041784

[preinking]

I can confirm that this issue does not exist in 1.9.0 because this deploys fine from scratch on my environment. When performing the same using version 1.10.0 crc start fails consistently.

Logs
https://github.com/code-ready/crc/files/4717841/logs.txt

[rscher]

same here, cant start after applying crc 1.10.0 on Mac-

ERRO Failed to renew TLS certificates: ssh command error:
command : sudo podman run -it --rm --network=host -v /etc/kubernetes/:/etc/kubernetes/:Z --entrypoint=/usr/bin/cluster-kube-apiserver-operator 'openshift/cert-recovery' regenerate-certificates

===
Is there a patch ? I'm dead in the water

[btannous]

Just for documentation on this issue, I tested 1.8-1.11.
Looks like certs renew properly on all CRC version <1.10 and they do not renew on >=1.10.

[rscher]

Just for documentation on this issue, I tested 1.8-1.11.
Looks like certs renew properly on all CRC version <1.10 and they do not renew on >=1.10.

Where can I download the older versions crc 1.9 or 1.8 ?

[gbraad]

https://mirror.openshift.com/pub/openshift-v4/clients/crc/

[tadayosi]

For the time being, upgrading crc to latest 1.11.0 solves temporarily (i.e. postpones for another 30 days) the issue.

[btannous]

As commented in the PR https://github.com/code-ready/crc/pull/1292 , I validated that the fix works on my end, so once merged in and a new build comes out this should be likely closed.

[btannous]

@praveenkumar @cfergeau wooo! thank you all.

@BenHall certs should be good now (when a new build comes i’d assume) so unblocked :)!

[BenHall]

@btannous Thanks! I applied the changes this morning and will test properly when the certs expire at the end of the month.