Okd: machine-config operator degrades after pull secret is changed

[jeffdyoung]

Description:

machine-config becomes degraded 15mins after cluster start. Didn't notice an issue until I updated the default ingress cert (5 days later) and lost the ability to login to the web console.

Cluster Version:
OKD 4.4.0-0.okd-2020-01-28-022517 on AWS

must-gather: https://drive.google.com/open?id=1Ex9fyflP7FVC049X5JvISCudwxN2QQIr
cc: @vrutkovs

[vrutkovs]

'Node ip-10-0-135-100.us-east-2.compute.internal is reporting: "unexpected
on-disk state validating against rendered-master-573b7311eb1884238b3971e8453058c2"'

2020-01-30T01:34:00.15541173Z I0130 01:34:00.155323 129603 daemon.go:959] Validating against current config rendered-master-573b7311eb1884238b3971e8453058c2
2020-01-30T01:34:00.163999315Z E0130 01:34:00.163951 129603 daemon.go:1351] content mismatch for file /var/lib/kubelet/config.json: {"auths":{"
2020-01-30T01:34:00.163999315Z
2020-01-30T01:34:00.163999315Z A: registry.redhat.io":{"auth":"","email":""}}}
2020-01-30T01:34:00.163999315Z
2020-01-30T01:34:00.163999315Z
2020-01-30T01:34:00.163999315Z B: fake":{"auth":"bar"}}}
2020-01-30T01:34:00.163999315Z
2020-01-30T01:34:00.163999315Z

Did you switch the pull secret in flight?

[jeffdyoung]

Switched pull secret after the install. The image streams weren't being pulled from registry.redhat.io

[vrutkovs]

Hmm, I don't think that is supported. I assume you replaced the pull secret in MC, right?

/cc @LorbusChris

[jeffdyoung]

I only updated the pull-secret in the openshift-config namespace.

[vrutkovs]

That seems to be an MCO bug then (replacing the config map is a correct way, its documented so it should work)

[jdhirst]

I can confirm the issue occurs with any change to the pull secret including removing the secret for insights/support.

Followed this guide: https://docs.openshift.com/container-platform/4.3/support/remote_health_monitoring/opting-out-of-remote-health-reporting.html

Result:

Node master-0.okd.hirstgroup.net is reporting: "unexpected on-disk state validating against rendered-master-117eb4416146e766dfa49fba73e06f4f"

[jeffdyoung]

Should changing my secret back to {"auths":{"fake":{"auth": "bar"}}} fix this?

[jdhirst]

It doesn't work to revert it from my experience. I had to rebuild the whole cluster

[sgreene570]

Should changing my secret back to {"auths":{"fake":{"auth": "bar"}}} fix this?

I'm confused, was your original pull secret this pull secret or an actual pull secret?

[jdhirst]

@sgreene570 the pull secret I had was an actual pull secret. Everything worked fine with it. I then changed it by removing the cloud.redhat.com credentials from the pull secret. This is when it broke MCO and rendered it degraded.

It seems any change to the pull secret at all causes this conflict.

[jeffdyoung]

This is an OKD cluster, so my original pull secret was {"auths":{"fake":{"auth": "bar"}}}
OKD currently tries to use OCP imagestreams by default. (this being addressed: https://github.com/openshift/okd/issues/34 ).

Not knowing the ramifications... I changed the cluster pull secret to my registry.redhat.io secret so image streams could be downloaded by the samples operator (which worked).

I'll probably rebuild once the community samples are working.

[jeffdyoung]

@vrutkovs Do you need more information from this cluster? I'd like to delete and rebuild and have a working cluster again.

[vrutkovs]

It seems to be easy to reproduce (changing the pull secret triggers it).

Could you check if https://github.com/openshift/installer/pull/3020 improves the situation?

[vrutkovs]

Looked into this today:

• changing secret immediately(?) writes down the file on the first master
• machine-config-daemon tries to validate /var/lib/kubelet/config.json against existing configuration
• that causes

content mismatch for file /var/lib/kubelet/config.json: {"auths":{"

A: fake":{"auth":"bar"}}}


B: cloud.openshift.com":{"auth":

Generated rendered- configs looks sane, not sure why would the file be written before MCD validates the existing configuration.

@LorbusChris @sgreene570 any ideas?

[sgreene570]

content mismatch for file /var/lib/kubelet/config.json: {"auths":{"

A: fake":{"auth":"bar"}}}


B: cloud.openshift.com":{"auth":

@LorbusChris @sgreene570 any ideas?

This error is from when the MCD goes to validate the files on disk (including /var/lib/kubelet/config.json) after a node reboots after the change is applied, right?
Are the nodes somehow comparing the wrong configs to the current state?

[vrutkovs]

iirc this happens before the node is rebooted, so MCD is comparing it to a wrong commit (I'll confirm that).

I'll check if rebased MCD fixes this

[cragr]

Possible Workaround, need verification.

Originally, I installed my cluster using {"auths":{"fake":{"auth": "bar"}}} as the pull-secret in my install-config.yaml. After the cluster was up I realized I should have used my real pull-secret. So I created another install-config.yaml with the correct pull secret in a temp directory and created a manifest using "openshift-install create manifests --dir=install_temp". This generated a openshift-config-secret-pull-secret.yaml under the install_temp/manifests directory. I changed to the openshift-config project and deleted the pull secret and recreated using the newly generated openshift-config-secret-pull-secret.yaml (oc create -f openshift-config-secret-pull-secret.yaml) . Then I updated the /var/lib/kubelet/config.json on each node with my pull-secret. Next, I cordon'd, drained, rebooted, and uncordon'd each node (with only one node being down at any given time). After this the images began downloading and the list of failed images in the openshift-sample operator dwindled as the imagestreams (oc describe clusteroperators openshift-samples) in the openshift project populated (oc get is -n openshift).

There are about 5 more to go for me and then the openshift-samples operator should no longer be degraded. It has been a slow process, over an hour and still going. I'm unsure if all my steps were necessary, specifically rebooting the nodes, but it worked for me.

[sgreene570]

There are about 5 more to go for me and then the openshift-samples operator should no longer be degraded. It has been a slow process, over an hour and still going. I'm unsure if all my steps were necessary, specifically rebooting the nodes, but it worked for me.

Whew that sounds like a lot of effort! Hopefully rebased MCD solves this problem. Rebooting the nodes might not be necessary in this case, but it is a good way to verify that the pull secret won't be reverted on the next node reboot.

[vrutkovs]

MCO rebase in https://origin-release.svc.ci.openshift.org/releasestream/4.4.0-0.okd/release/4.4.0-0.okd-2020-03-25-115303 seems to fix that - it takes a while for cluster to update (about 10 mins here), but it doesn't degrade and I can no longer pull images from registry.redhat.io.

Please file a new bug if this occurs again and link to this one