Okd: Nightly & Beta images are not signed, upgrade won't succeed without `--force`

[cgruver]

Installed version: 4.4.0-0.okd-2020-04-09-104654

Update to version: 4.4.0-0.okd-2020-04-09-124138

Log snippet from cluster-version-operator-5657d5967-2w5c9

I0410 15:12:41.419403       1 sync_worker.go:471] Running sync 4.4.0-0.okd-2020-04-09-124138 (force=false) on generation 2 in state Updating at attempt 3
I0410 15:12:41.419925       1 sync_worker.go:477] Loading payload
I0410 15:12:41.421099       1 store.go:74] use cached most recent signature config maps
I0410 15:12:41.447831       1 store.go:65] remember most recent signature config maps: signatures-managed
I0410 15:12:41.447893       1 store.go:116] searching for sha256-ce16ac845c0a0d178149553a51214367f63860aea71c0337f25556f25e5b8bb3 in signature config map signatures-managed
I0410 15:12:41.614273       1 verify.go:298] Unable to verify sha256:ce16ac845c0a0d178149553a51214367f63860aea71c0337f25556f25e5b8bb3 against keyring verifier-public-key-openshift-ci
E0410 15:12:41.614738       1 sync_worker.go:329] unable to synchronize image (waiting 2m52.525702462s): The update cannot be verified: unable to locate a valid signature for one or more sources
I0410 15:12:42.754202       1 cvo.go:439] Started syncing cluster version "openshift-cluster-version/version" (2020-04-10 15:12:42.75414743 +0000 UTC m=+1022.963748872)
I0410 15:12:42.754369       1 cvo.go:468] Desired version from spec is v1.Update{Version:"4.4.0-0.okd-2020-04-09-124138", Image:"registry.svc.ci.openshift.org/origin/release@sha256:ce16ac845c0a0d178149553a51214367f63860aea71c0337f25556f25e5b8bb3", Force:false}
I0410 15:12:42.754658       1 cvo.go:441] Finished syncing cluster version "openshift-cluster-version/version" (500.97µs)

[cgruver]

Update:

Appears to be an issue with using a mirrored image registry.

I removed the ImageContentSourcePolicy that I created after mirroring the upgrade images.

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: okd4-4.4-2020-04-09-124138 
spec:
  repositoryDigestMirrors:
  - mirrors:
    - nexus.my.domain.org:5002/origin
    source: registry.svc.ci.openshift.org/origin/4.4-2020-04-09-124138
  - mirrors:
    - nexus.my.domain.org:5002/origin
    source: registry.svc.ci.openshift.org/origin/release

The cluster-version-operator restarted as soon as I removed the above ImageContentSourcePolicy

I0410 15:49:39.008810       1 reflector.go:383] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: Watch close - *v1.ClusterVersion total 7 items received
I0410 15:49:55.313057       1 start.go:140] Shutting down due to terminated
I0410 15:49:55.313223       1 start.go:188] Stepping down as leader
I0410 15:49:55.313474       1 cvo.go:439] Started syncing cluster version "openshift-cluster-version/version" (2020-04-10 15:49:55.313467667 +0000 UTC m=+1075.871631801)
I0410 15:49:55.313638       1 cvo.go:468] Desired version from spec is v1.Update{Version:"4.4.0-0.okd-2020-04-09-124138", Image:"registry.svc.ci.openshift.org/origin/release@sha256:ce16ac845c0a0d178149553a51214367f63860aea71c0337f25556f25e5b8bb3", Force:false}
I0410 15:49:55.313811       1 cvo.go:441] Finished syncing cluster version "openshift-cluster-version/version" (339.147µs)
I0410 15:49:55.313833       1 cvo.go:366] Shutting down ClusterVersionOperator
I0410 15:49:55.485669       1 start.go:199] Finished shutdown

An update is now in process.

However, the following still appears in the logs:

I0410 15:56:21.119050       1 verify.go:298] Unable to verify sha256:ce16ac845c0a0d178149553a51214367f63860aea71c0337f25556f25e5b8bb3 against keyring verifier-public-key-openshift-ci
E0410 15:56:21.119152       1 sync_worker.go:329] unable to synchronize image (waiting 1m26.262851224s): The update cannot be verified: unable to locate a valid signature for one or more sources
I0410 15:56:21.119167       1 cvo.go:439] Started syncing cluster version "openshift-cluster-version/version" (2020-04-10 15:56:21.119164685 +0000 UTC m=+238.200408329)
I0410 15:56:21.119208       1 cvo.go:468] Desired version from spec is v1.Update{Version:"4.4.0-0.okd-2020-04-09-124138", Image:"registry.svc.ci.openshift.org/origin/release@sha256:ce16ac845c0a0d178149553a51214367f63860aea71c0337f25556f25e5b8bb3", Force:false}
I0410 15:56:21.132567       1 cvo.go:441] Finished syncing cluster version "openshift-cluster-version/version" (13.396691ms)

[cgruver]

Possible relationship to: https://bugzilla.redhat.com/show_bug.cgi?id=1789765

[vrutkovs]

Its a bit different - nightly images are not signed, so by default CVO refuses to use those.

Workaround: use oc adm upgrade --force --to=<release name>

[chriskuipers]

For anyone who is running into this issue by starting the upgrade procedure from the web ui, running oc adm upgrade --force --to=<version> won't work and will result in the notice info: Cluster is already at version 4.4.0-0.okd-2020-05-23-055148-beta5.

I was able to re-trigger the upgrade with the --force parameter by first "downgrading" the cluster back to the version it still is at by running oc adm upgrade --force --to=<version> --allow-explicit-upgrade.

This will leverage the same info: Cluster is already at version <version> notice, but now lets you upgrade using the --force parameter as described here.

[jasonbrooks]

I was able to re-trigger the upgrade with the --force parameter by first "downgrading" the cluster back to the version it still is at by running oc adm upgrade --force --to=<version> --allow-explicit-upgrade.

This failed for me with error: The update 4.4.0-0.okd-2020-04-21-163702-beta4 is not one of the available updates: 4.4.0-0.okd-2020-05-23-055148-beta5, but oc adm upgrade --clear worked to cancel the upgrade and allow me to proceed with the forced upgrade.

[pflaeging]

OK, I can confirm, the following sequence is working for me (OKD4.4b4 on bare metal):

oc adm upgrade --clear
oc adm upgrade --force --to=4.4.0-0.okd-2020-04-21-163702-beta4 --allow-explicit-upgrade
oc adm upgrade --force --to=4.4.0-0.okd-2020-05-23-055148-beta5

[vrutkovs]

https://github.com/openshift/cluster-update-keys/pull/19 has updated expected location of new CI keys.

That means clusters since https://origin-release.apps.ci.l2s4.p1.openshiftapps.com/releasestream/4.5.0-0.okd/release/4.5.0-0.okd-2020-07-08-164508 should be able to update to new releases without --force

I'll keep this open for some time to try it out

[cgruver]

Thanks! I will test in my lab.

[vrutkovs]

Started an update via console, no --force or additional hacks are necessary. Unfortunately the change is not retroactive, so release earlier than Jul 9 still require --force to update (once).

Please reopen if it still occurs

[gerr1t]

@vrutkovs , I cant upgrade:

oc adm upgrade --to=4.5.0-0.okd-2020-06-24-173007 --force

NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.0-0.okd-2020-05-23-055148-beta5 True True 21m Unable to apply 4.5.0-0.okd-2020-06-24-173007: could not download the update

[vrutkovs]

You're updating from May version to June version. These still require --force, as the fix can't rewrite signatures for previous payloads and signature bucket location in your cluster

[gerr1t]

@vrutkovs , I am using --force :)

[cgruver]

Do the images for that June version even exist anymore? That's a nightly, correct? I think they still disappear after 48 hours.

[vrutkovs]

Right, 4.5.0-0.okd-2020-06-24-173007 is a nightly, so it was removed after 48hrs.

See https://origin-release.apps.ci.l2s4.p1.openshiftapps.com/#4.5.0-0.okd for available nightlies. You may have to use --allow-explicit-upgrade if there is no edge between beta5 and selected nightly.

[gerr1t]

I see. For whatever reason, my upgrade list has not been updated since the 24th of June. Any clue why that is? And which controller can I check to see what is causing the list not to be updated?

Updates:

VERSION                       IMAGE
4.5.0-0.okd-2020-06-22-071006 registry.svc.ci.openshift.org/origin/release@sha256:3b7f81ba18678820573c54741efb362c6a6ef64dfb54ce44676aed4129f35d1a
4.5.0-0.okd-2020-06-22-081006 registry.svc.ci.openshift.org/origin/release@sha256:acccd343440cfddb6849b475a8fa9600e55dbba303cf2deb61402d623fe8d2fa
4.5.0-0.okd-2020-06-22-151136 registry.svc.ci.openshift.org/origin/release@sha256:592b9ff608b7f427de435b84d1febd582dfb968c7518afe5a1757838f35b9661
4.5.0-0.okd-2020-06-22-154136 registry.svc.ci.openshift.org/origin/release@sha256:98518192975530721f6a39a699e9f6a57b8dd4c96b696d755cd5c0eb87e0f7a3
4.5.0-0.okd-2020-06-22-190343 registry.svc.ci.openshift.org/origin/release@sha256:21d8a61c72d3fd571c6a06d65f071629c2723ec95f497ace5107585e97ac419c
4.5.0-0.okd-2020-06-22-193343 registry.svc.ci.openshift.org/origin/release@sha256:2c50a9a4d6dd5966b158aa367a46c7053637aa8cfe37d2fbefadd3067df6e45b
4.5.0-0.okd-2020-06-22-200343 registry.svc.ci.openshift.org/origin/release@sha256:9eff85127a97a9e18378b460a4f611654a3cace1e8ea438da00a09d26cdc8180
4.5.0-0.okd-2020-06-23-034044 registry.svc.ci.openshift.org/origin/release@sha256:496814b8b5221ba38ec4e30bd499afd38213cd276a801a1fb9b460d847c830a1
4.5.0-0.okd-2020-06-23-134719 registry.svc.ci.openshift.org/origin/release@sha256:9a294906bd58d2144f9eaabb906c8b44bceae2c8db515f60163d7b2f5f9b892e
4.5.0-0.okd-2020-06-23-152633 registry.svc.ci.openshift.org/origin/release@sha256:c834613be7193b7d1e45e92b15c492fff29db74a501290c6b4ebbc63845e1252
4.5.0-0.okd-2020-06-23-170700 registry.svc.ci.openshift.org/origin/release@sha256:e4739b3fdc37754256364b5cda25827b34bbb6629f0f8dcca46bb3f3593d9222
4.5.0-0.okd-2020-06-23-184550 registry.svc.ci.openshift.org/origin/release@sha256:3cc7b9e0a795502ad25f956fc57fdc46bdfbd91946da5f29de44f91d38615190
4.5.0-0.okd-2020-06-23-215133 registry.svc.ci.openshift.org/origin/release@sha256:159a8080d842c41655edbe3cb63556644dfe9d2c2428e5833112c07470d6c887
4.5.0-0.okd-2020-06-23-231337 registry.svc.ci.openshift.org/origin/release@sha256:58becc5c3b2c104a65d586dfa27995d18fac5d3f01060ca1ed473bc5718d1b84
4.5.0-0.okd-2020-06-24-001138 registry.svc.ci.openshift.org/origin/release@sha256:7e73b580b4b174778ce03d32f0bb2f9ad1840d2eb909054833313d41952ff6a4
4.5.0-0.okd-2020-06-24-022925 registry.svc.ci.openshift.org/origin/release@sha256:66dc49d36465c604d60a7af47353a34f41a02b9cf4fda46cf1771c3c9a374abc
4.5.0-0.okd-2020-06-24-045852 registry.svc.ci.openshift.org/origin/release@sha256:bfdc71716744024ab542b846b28c40f3b92337939c1f4ba1fc4837f13668981c
4.5.0-0.okd-2020-06-24-052852 registry.svc.ci.openshift.org/origin/release@sha256:0f642e0275f1178ad6e7f5df55f0839e3b8bd4e25f9177ab2584ee4c87ef27b6
4.5.0-0.okd-2020-06-24-070126 registry.svc.ci.openshift.org/origin/release@sha256:efb34c461d8f98f22a3cfbe36a92d9b51b0af679f78d21bb95c3f93f9ca40344
4.5.0-0.okd-2020-06-24-073126 registry.svc.ci.openshift.org/origin/release@sha256:c3ce301429ce2dafb60e7a230d928137bf6a2c3eadaf2364baeb3bec559c4db5
4.5.0-0.okd-2020-06-24-092928 registry.svc.ci.openshift.org/origin/release@sha256:4fedb05f7c1cc5d8849bf232869f07165898ae14e2fedccf05fd0bd943501895
4.5.0-0.okd-2020-06-24-143822 registry.svc.ci.openshift.org/origin/release@sha256:363b12e5b05a8afc3f72d2cf13da8fc769637a3ad1405f18f1fc9d3ca9542a33
4.5.0-0.okd-2020-06-24-150822 registry.svc.ci.openshift.org/origin/release@sha256:a06abfdf65165dd9f178dec0543f8fc960a90f53b1bd1fa7b39c9adf7ed2b7f5
4.5.0-0.okd-2020-06-24-170007 registry.svc.ci.openshift.org/origin/release@sha256:fc602da983b8185c3e2c94f96ca13a511126e7dc078d057636787df3df839cf7
4.5.0-0.okd-2020-06-24-173007 registry.svc.ci.openshift.org/origin/release@sha256:cf0c9c404282a5529a8b8e392fd058ca6301d9adebc153bd3871d28205727fa5 

[vrutkovs]

Lets track in a separate bug, please provide a must-gather

[cgruver]

@vrutkovs This can probably be unpinned soon.