Okd: 4.7.0-0.okd-2021-04-11-124433: The update cannot be verified: unable to locate a valid signature for one or more sources

Created on 25 Apr 2021  路  13Comments  路  Source: openshift/okd

Describe the bug
Hi,

Today I used the update button to update from 4.7.0-0.okd-2021-04-11-124433 to 4.7.0-0.okd-2021-04-24-103438.
But it seems unable to verify the image.

apiVersion: config.openshift.io/v1
kind: ClusterVersion
metadata:
  creationTimestamp: '2021-04-21T18:21:29Z'
  generation: 2
  managedFields:
    - apiVersion: config.openshift.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:spec':
          .: {}
          'f:channel': {}
          'f:clusterID': {}
          'f:upstream': {}
      manager: cluster-bootstrap
      operation: Update
      time: '2021-04-21T18:21:29Z'
    - apiVersion: config.openshift.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:spec':
          'f:desiredUpdate':
            .: {}
            'f:image': {}
            'f:version': {}
      manager: Mozilla
      operation: Update
      time: '2021-04-25T09:36:15Z'
    - apiVersion: config.openshift.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:spec':
          'f:desiredUpdate':
            'f:force': {}
        'f:status':
          .: {}
          'f:availableUpdates': {}
          'f:conditions': {}
          'f:desired':
            .: {}
            'f:image': {}
            'f:version': {}
          'f:history': {}
          'f:observedGeneration': {}
          'f:versionHash': {}
      manager: cluster-version-operator
      operation: Update
      time: '2021-04-25T09:36:19Z'
  name: version
  resourceVersion: '725979'
  selfLink: /apis/config.openshift.io/v1/clusterversions/version
  uid: dbfdb75c-0e09-41ec-9c37-736b3eba9440
spec:
  channel: stable-4
  clusterID: 6d5876e1-6fea-4f40-be9c-335882390845
  desiredUpdate:
    image: >-
      registry.ci.openshift.org/origin/release@sha256:1c71a740375b34df53c6e2bfe389018f3160cea46c4022ed6080f583c54ecbc0
    version: 4.7.0-0.okd-2021-04-24-103438
  upstream: 'https://origin-release.svc.ci.openshift.org/graph'
status:
  availableUpdates:
    - image: >-
        registry.ci.openshift.org/origin/release@sha256:1c71a740375b34df53c6e2bfe389018f3160cea46c4022ed6080f583c54ecbc0
      version: 4.7.0-0.okd-2021-04-24-103438
  conditions:
    - lastTransitionTime: '2021-04-21T19:15:17Z'
      message: Done applying 4.7.0-0.okd-2021-04-11-124433
      status: 'True'
      type: Available
    - lastTransitionTime: '2021-04-25T12:38:14Z'
      message: >-
        The update cannot be verified: unable to locate a valid signature for
        one or more sources
      reason: ImageVerificationFailed
      status: 'True'
      type: Failing
    - lastTransitionTime: '2021-04-25T09:36:19Z'
      message: >-
        Unable to apply 4.7.0-0.okd-2021-04-24-103438: the image may not be safe
        to use
      reason: ImageVerificationFailed
      status: 'True'
      type: Progressing
    - lastTransitionTime: '2021-04-25T12:35:03Z'
      status: 'True'
      type: RetrievedUpdates
  desired:
    image: >-
      registry.ci.openshift.org/origin/release@sha256:1c71a740375b34df53c6e2bfe389018f3160cea46c4022ed6080f583c54ecbc0
    version: 4.7.0-0.okd-2021-04-24-103438
  history:
    - completionTime: null
      image: >-
        registry.ci.openshift.org/origin/release@sha256:1c71a740375b34df53c6e2bfe389018f3160cea46c4022ed6080f583c54ecbc0
      startedTime: '2021-04-25T09:36:19Z'
      state: Partial
      verified: false
      version: 4.7.0-0.okd-2021-04-24-103438
    - completionTime: '2021-04-21T19:15:17Z'
      image: >-
        quay.io/openshift/okd@sha256:4414db9e8945504d1d5423ec11d64df9fa8b4aa60d978f8b4bdb3d0720800741
      startedTime: '2021-04-21T18:21:52Z'
      state: Completed
      verified: false
      version: 4.7.0-0.okd-2021-04-11-124433
  observedGeneration: 2
  versionHash: 4boqI_3hclA=

Version

How reproducible

use the update function in okd
Log bundle

Is a log needed ? the update hasn't done anything yet.

Rob

picture RobVerduijn

Most helpful comment

Got same problem, mitigated:

$ oc get clusterversion
NAME      VERSION                         AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.okd-2021-04-11-124433   True        True          70s     Unable to apply 4.7.0-0.okd-2021-04-24-103438: the image may not be safe to use
$ oc adm upgrade --clear=true
Cleared the update field, still at 4.7.0-0.okd-2021-04-24-103438
$ oc get clusterversion
NAME      VERSION                         AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.okd-2021-04-11-124433   True        True          8m4s    Working towards 4.7.0-0.okd-2021-04-11-124433: 601 of 669 done (89% complete)
...
version   4.7.0-0.okd-2021-04-11-124433   True        False         107s    Cluster version is 4.7.0-0.okd-2021-04-11-124433
$ oc adm upgrade --force --to=4.7.0-0.okd-2021-04-24-103438
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Updating to 4.7.0-0.okd-2021-04-24-103438
$ oc get clusterversion
NAME      VERSION                         AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.okd-2021-04-11-124433   True        True          5s      Working towards 4.7.0-0.okd-2021-04-24-103438: downloading update
...
version   4.7.0-0.okd-2021-04-11-124433   True        True          57m     Working towards 4.7.0-0.okd-2021-04-24-103438: 301 of 669 done (44% complete)
...
version   4.7.0-0.okd-2021-04-24-103438   True        False         20s     Cluster version is 4.7.0-0.okd-2021-04-24-103438
picture OleksandrShtepa on 28 Apr 2021
👍3

All 13 comments

Hey Rob,

Long time no see/hear. This has happened before. Some image signing process hasn't run yet. Maybe one of the admins need to to restart that process.

GerbenWelter picture GerbenWelter on 25 Apr 2021

Right, seems the signer has stuck. We'll restart it, until then please use 'oc adm upgrade --force' or set .spec.desired.force to true in ClusterVersion

vrutkovs picture vrutkovs on 25 Apr 2021

I'll wait for the signer to do it's job.

RobVerduijn picture RobVerduijn on 25 Apr 2021

We see this almost every month. Is there no sustainable solution for this?

myg0v picture myg0v on 26 Apr 2021

I have the same issue

bobby0724 picture bobby0724 on 26 Apr 2021

Right, seems the signer has stuck. We'll restart it, until then please use 'oc adm upgrade --force' or set .spec.desired.force to true in ClusterVersion

just for general info 'oc adm upgrade --force' didn't work

bobby0724 picture bobby0724 on 27 Apr 2021
😕1

Right, seems the signer has stuck. We'll restart it, until then please use 'oc adm upgrade --force' or set .spec.desired.force to true in ClusterVersion

just for general info 'oc adm upgrade --force' didn't work

The second option did work for me to force the update for an unverified image, specifically set .spec.desiredUpdate.force to true

twling picture twling on 27 Apr 2021
👍2 
$ export RELEASE="4.7.0-0.okd-2021-04-24-103438"                                  
$ oc adm release info quay.io/openshift/okd:${RELEASE} | head -n 2 
Name:           4.7.0-0.okd-2021-04-24-103438
Digest:         sha256:1c71a740375b34df53c6e2bfe389018f3160cea46c4022ed6080f583c54ecbc0
$ export DIGEST="1c71a740375b34df53c6e2bfe389018f3160cea46c4022ed6080f583c54ecbc0"                                                         
$ curl -Ls https://storage.googleapis.com/openshift-ci-release/releases/signatures/openshift/release/sha256\=${DIGEST}/signature-1 | gpg -d
{
  "critical": {
    "type": "atomic container signature",
    "image": {
      "docker-manifest-digest": "sha256:1c71a740375b34df53c6e2bfe389018f3160cea46c4022ed6080f583c54ecbc0"
    },
    "identity": {
      "docker-reference": "quay.io/openshift/okd:4.7.0-0.okd-2021-04-24-103438"
    }
  },
  "optional": {
    "creator": "openshift release-controller",
    "timestamp": 1619554498
  }
}gpg: Signature made Tue 27 Apr 2021 22:14:58 CEST
gpg:                using RSA key 65C28371F55D29A9

However the previous signer key has expired, so you'd have to force upgrade this time (and, most likely, the next time too).

Sorry for the inconvenience

vrutkovs picture vrutkovs on 28 Apr 2021

Got same problem, mitigated:

$ oc get clusterversion
NAME      VERSION                         AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.okd-2021-04-11-124433   True        True          70s     Unable to apply 4.7.0-0.okd-2021-04-24-103438: the image may not be safe to use
$ oc adm upgrade --clear=true
Cleared the update field, still at 4.7.0-0.okd-2021-04-24-103438
$ oc get clusterversion
NAME      VERSION                         AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.okd-2021-04-11-124433   True        True          8m4s    Working towards 4.7.0-0.okd-2021-04-11-124433: 601 of 669 done (89% complete)
...
version   4.7.0-0.okd-2021-04-11-124433   True        False         107s    Cluster version is 4.7.0-0.okd-2021-04-11-124433
$ oc adm upgrade --force --to=4.7.0-0.okd-2021-04-24-103438
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Updating to 4.7.0-0.okd-2021-04-24-103438
$ oc get clusterversion
NAME      VERSION                         AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.okd-2021-04-11-124433   True        True          5s      Working towards 4.7.0-0.okd-2021-04-24-103438: downloading update
...
version   4.7.0-0.okd-2021-04-11-124433   True        True          57m     Working towards 4.7.0-0.okd-2021-04-24-103438: 301 of 669 done (44% complete)
...
version   4.7.0-0.okd-2021-04-24-103438   True        False         20s     Cluster version is 4.7.0-0.okd-2021-04-24-103438
OleksandrShtepa picture OleksandrShtepa on 28 Apr 2021
👍3

thanks

bobby0724 picture bobby0724 on 28 Apr 2021

my upgrade --force take a little while but finally completes thanks

bobby0724 picture bobby0724 on 29 Apr 2021

Got same problem, mitigated:

$ oc get clusterversion
NAME      VERSION                         AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.okd-2021-04-11-124433   True        True          70s     Unable to apply 4.7.0-0.okd-2021-04-24-103438: the image may not be safe to use
$ oc adm upgrade --clear=true
Cleared the update field, still at 4.7.0-0.okd-2021-04-24-103438
$ oc get clusterversion
NAME      VERSION                         AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.okd-2021-04-11-124433   True        True          8m4s    Working towards 4.7.0-0.okd-2021-04-11-124433: 601 of 669 done (89% complete)
...
version   4.7.0-0.okd-2021-04-11-124433   True        False         107s    Cluster version is 4.7.0-0.okd-2021-04-11-124433
$ oc adm upgrade --force --to=4.7.0-0.okd-2021-04-24-103438
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Updating to 4.7.0-0.okd-2021-04-24-103438
$ oc get clusterversion
NAME      VERSION                         AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.okd-2021-04-11-124433   True        True          5s      Working towards 4.7.0-0.okd-2021-04-24-103438: downloading update
...
version   4.7.0-0.okd-2021-04-11-124433   True        True          57m     Working towards 4.7.0-0.okd-2021-04-24-103438: 301 of 669 done (44% complete)
...
version   4.7.0-0.okd-2021-04-24-103438   True        False         20s     Cluster version is 4.7.0-0.okd-2021-04-24-103438

I got:

oc get clusterversion

NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.6.0-0.okd-2021-02-14-205305 True True 64m Unable to apply 4.7.0-0.okd-2021-04-24-103438: the control plane is reporting an internal error

Can anyone help?

apmlima picture apmlima on 11 May 2021

https://github.com/openshift/cluster-update-keys/pull/36 would resolve that. 4.8 nightlies signature verification works.

vrutkovs picture vrutkovs on 15 May 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

kkk25641463 picture kkk25641463  路  5Comments
vrutkovs picture vrutkovs  路  12Comments
cgruver picture cgruver  路  10Comments
vrutkovs picture vrutkovs  路  3Comments
slauger picture slauger  路  5Comments