Okd: oVirt: credential-operator not placing credentials

[abaxo]

As referenced in #61 there is an issue when deploying a new cluster on oVirt (4.3.8) where the ovirt-credentials secret is created in the kube-system namespace rather than the expected openshift-machine-api namespace. This causes the scale-up of the workers to fail (machines remain in provisioned state, and don't reach 'up' as the VMs are not spawned.). Manually placing the secret into the openshift-machine-api namespace causes workers to successfully scale.

Additionally, this behavior of being unable to request the ovirt-credentials is shown when deploying the ovirt-csi-driver operator - the request is also not fulfilled in that case.

Here is the credential request:

apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  creationTimestamp: "2020-03-13T21:18:56Z"
  finalizers:
  - cloudcredential.openshift.io/deprovision
  generation: 1
  labels:
    controller-tools.k8s.io: "1.0"
  name: openshift-machine-api-ovirt
  namespace: openshift-cloud-credential-operator
  resourceVersion: "1927"
  selfLink: /apis/cloudcredential.openshift.io/v1/namespaces/openshift-cloud-credential-operator/credentialsrequests/openshift-machine-api-ovirt
  uid: 6c3fe339-d6d0-49fe-bfc6-d883712a7476
spec:
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: OvirtProviderSpec
  secretRef:
    name: ovirt-credentials
    namespace: openshift-machine-api
status:
  conditions:
  - lastProbeTime: "2020-03-13T21:20:42Z"
    lastTransitionTime: "2020-03-13T21:20:42Z"
    message: cloud creds are insufficient to satisfy CredentialsRequest
    reason: CloudCredsInsufficient
    status: "True"
    type: InsufficientCloudCreds
  lastSyncGeneration: 0
  provisioned: false

The relevant part of the cloud-credential-operator log:

time="2020-03-15T17:23:28Z" level=info msg="syncing credentials request" controller=credreq cr=openshift-cloud-credential-operator/openshift-machine-api-ovirt
time="2020-03-15T17:23:28Z" level=debug msg="found secret namespace" controller=credreq cr=openshift-cloud-credential-operator/openshift-machine-api-ovirt secret=openshift-machine-api/ovirt-credentials
time="2020-03-15T17:23:28Z" level=debug msg="running Exists" actuator=Ovirt cr=openshift-cloud-credential-operator/openshift-machine-api-ovirt
time="2020-03-15T17:23:28Z" level=debug msg="target secret exists" actuator=Ovirt cr=openshift-cloud-credential-operator/openshift-machine-api-ovirt
time="2020-03-15T17:23:28Z" level=debug msg="running update" actuator=Ovirt cr=openshift-cloud-credential-operator/openshift-machine-api-ovirt
time="2020-03-15T17:23:28Z" level=debug msg="running sync" actuator=Ovirt cr=openshift-cloud-credential-operator/openshift-machine-api-ovirt
time="2020-03-15T17:23:28Z" level=warning msg="secret did not have expected key: ovirt-credentials" actuator=Ovirt cr=openshift-cloud-credential-operator/openshift-machine-api-ovirt
time="2020-03-15T17:23:28Z" level=error msg="issue with cloud credentials secret" actuator=Ovirt cr=openshift-cloud-credential-operator/openshift-machine-api-ovirt error="secret did not have expected key: ovirt-credentials"
time="2020-03-15T17:23:28Z" level=error msg="error syncing credentials: secret did not have expected key: ovirt-credentials" controller=credreq cr=openshift-cloud-credential-operator/openshift-machine-api-ovirt secret=openshift-machine-api/ovirt-credentials
time="2020-03-15T17:23:28Z" level=error msg="errored with condition: InsufficientCloudCreds" controller=credreq cr=openshift-cloud-credential-operator/openshift-machine-api-ovirt secret=openshift-machine-api/ovirt-credentials
time="2020-03-15T17:23:28Z" level=debug msg="updating credentials request status" controller=credreq cr=openshift-cloud-credential-operator/openshift-machine-api-ovirt secret=openshift-machine-api/ovirt-credentials
time="2020-03-15T17:23:28Z" level=debug msg="status unchanged" controller=credreq cr=openshift-cloud-credential-operator/openshift-machine-api-ovirt secret=openshift-machine-api/ovirt-credentials
time="2020-03-15T17:23:28Z" level=debug msg="syncing cluster operator status" controller=credreq_status
time="2020-03-15T17:23:28Z" level=debug msg="1 cred requests" controller=credreq_status
time="2020-03-15T17:23:28Z" level=debug msg="set ClusterOperator condition" controller=credreq_status message="1 of 1 credentials requests are failing to sync." reason=CredentialsFailing status=True type=Degraded
time="2020-03-15T17:23:28Z" level=debug msg="set ClusterOperator condition" controller=credreq_status message="0 of 1 credentials requests provisioned, 1 reporting errors." reason=Reconciling status=True type=Progressing
time="2020-03-15T17:23:28Z" level=debug msg="set ClusterOperator condition" controller=credreq_status message= reason= status=True type=Available
time="2020-03-15T17:23:28Z" level=debug msg="set ClusterOperator condition" controller=credreq_status message= reason= status=True type=Upgradeable

OKD is currently configured to use the default oVirt admin@internal user, which has no fancy restrictions to it - it is as it was out of the box.

[vrutkovs]

Could you attach oc adm must-gather archive to look into the details of this?

[abaxo]

At the risk of sounding like an idiot (I dont know golang) Is it just that the provider type is missing for ovirt? The spec exists in the release-4.4 branch but isn't implemented here: https://github.com/openshift/cloud-credential-operator/blob/release-4.4/pkg/apis/cloudcredential/v1/credentialsrequest_types.go#L151

[abaxo]

Ah no, thats just for passthrough secrets - nevermind.

[bsperduto]

Here is a must-gather from what appears to be the same issue

https://1drv.ms/u/s!AvqdSB6VIPS0mbcAZgQOgihDyoZ7sA?e=eIMHUa

[vrutkovs]

Some findings from must-gather:

• Authentication operator is not happy:

  - lastTransitionTime: "2020-03-18T12:54:19Z"
    message: 'route is not available at canonical host oauth-openshift.apps.test.okdtest.avsim.net:
      []'
    reason: FailedHost
    status: "True"
    type: RouteStatusDegraded

Probably not related though

• ovirt-credentials secret exists in kube-system - but it appears to be empty? Perhaps its an installer issue where the secret is created, but its in fact empty?
  @rgolangh could it be some fun terraform bug?

[bsperduto]

I just checked it actually does have the data in kube-system namespace.

[vrutkovs]

Ah, okay, so its must-gather removing those then

[rgolangh]

ok I think I know - in the process of copying the secret to a credential request there was
some missing field. The error is not printed, this is a logging bug:
time="2020-03-15T17:23:28Z" level=warning msg="secret did not have expected key: ovirt-credentials" actuator=Ovirt cr=openshift-cloud-credential-operator/openshift-machine-api-ovirt

Instead of 'ovirt-credentials' it should say which key was missing.

Since I added another 'ovirt_ca_bundle' key to the secret, and the it is not serialized by default, it is missing in the secret in kube-system.

1. For future installs, in you installation host, edit ~/.ovirt/ovirt-config.yaml and add
   yaml ovirt_ca_bundle: |- <content of /etc/pki/ovirt-engine/ca.pem>
2. For a workaround:
   
   
   
   • stop the cluster version operator
     
     $ oc scale --replicas 0 -n openshift-cluster-version deployments/cluster-version-operator
   
   
   • add the same content from 1. to the secret/ovirt-credentials under kube-system namespace
   
   
   • follow the credentials operator logs, it should sync correctly
   
   

Optionally you can remove ~/.ovirt/ovirt-config.yaml and let the installer wizard to recreate it -
answer Yes for 'Certificate is trusted' question, and then paste the ca.pem in the next question.

Thanks a lot for the patience and the report!

[vrutkovs]

I suppose the same issue is present in OCP 4.4 installer? If yes, we can backport the fix to catch this earlier

[rgolangh]

The only problem is a mix of old ovirt-config.yaml with new instllation.
I suspect that this happens because we are dealing with non GA artifacts

[bsperduto]

I don't believe the ovirt survey is in the okd installer branch yet is it @vrutkovs?

[bsperduto]

So it looks like there's still an issue somewhere at least with the OKD installer, it did not generate a secret with that line added. Below is the secret that was created. This is after editing the ovirt_config file

apiVersion: v1
data:
  ovirt_cafile: ""
  ovirt_insecure: dHJ1ZQ==
  ovirt_password: *****
  ovirt_url: *****
  ovirt_username: ****
kind: Secret
metadata:
  creationTimestamp: "2020-03-19T16:11:11Z"
  name: ovirt-credentials
  namespace: kube-system
  resourceVersion: "51"
  selfLink: /api/v1/namespaces/kube-system/secrets/ovirt-credentials
  uid: b3f98db9-b081-45b0-bd5e-eaae2251a7af
type: Opaque

[abaxo]

Nice!!! Thanks @rgolangh - I am annoyed I didnt try that. I even spotted the PR that was merged for adding the extra param.

I've just applied the value to the secret in kube-system namespace, bounced hte credentials operator and we have lift-off!

time="2020-03-19T16:40:33Z" level=info msg="syncing credentials request" controller=credreq cr=openshift-cloud-credential-operator/ovirt-csi-driver
time="2020-03-19T16:40:33Z" level=debug msg="found secret namespace" controller=credreq cr=openshift-cloud-credential-operator/ovirt-csi-driver secret=ovirt-csi-driver/ovirt-credentials
time="2020-03-19T16:40:33Z" level=debug msg="lastsyncgeneration is current and lastsynctimestamp was less than an hour ago, so no need to sync" controller=credreq cr=openshift-cloud-credential-operator/ovirt-csi-driver secret=ovirt-csi-driver/ovirt-credentials
time="2020-03-19T16:40:33Z" level=debug msg="syncing cluster operator status" controller=credreq_status
time="2020-03-19T16:40:33Z" level=debug msg="2 cred requests" controller=credreq_status
time="2020-03-19T16:40:33Z" level=debug msg="set ClusterOperator condition" controller=credreq_status message="No credentials requests reporting errors." reason=NoCredentialsFailing status=False type=Degraded
time="2020-03-19T16:40:33Z" level=debug msg="set ClusterOperator condition" controller=credreq_status message="2 of 2 credentials requests provisioned and reconciled." reason=ReconcilingComplete status=False type=Progressing

[vrutkovs]

I don't believe the ovirt survey is in the okd installer branch yet is it @vrutkovs?

No, not yet. OKD uses a forked installer and we're going to sync with release-4.4 soon.

Okay, so we'll try this:

• cherrypick https://github.com/openshift/installer/pull/3267 on fcos branch
  This would add ovirt_ca_bundle option in platform.ovirt of install-config.yaml
• around Beta we'll rebase installer on latest 4.4, aiui this would bring oVirt option in the survey

@rgolangh any other PRs required to enable CA bundle required? IIRC fcos has been rebased last time around end of january

[vrutkovs]

This should be fixed in https://origin-release.svc.ci.openshift.org/releasestream/4.4.0-0.okd/release/4.4.0-0.okd-2020-03-19-225539

@bsperduto @abaxo could you give it a try with ovirt_ca_bundle option set in install-config.yaml?

[abaxo]

Thanks @vrutkovs I've got a build running, I will check in the morning how it gets on.

[abaxo]

Aside from me doing something stupid and having to repeat the test, it seems to be working nicely. The workers have spun up as expected which looks like it is sorted as previously that was a behavior that we expected, but didn't happen. Once the cluster is fully up I'll check the secret and the logs, then we should be able to close this.

[abaxo]

Snippet of the cloud credential operator logs:

time="2020-03-20T10:54:02Z" level=info msg="started metrics calculator goroutine"
time="2020-03-20T10:54:02Z" level=info msg="calculating metrics for all CredentialsRequests" controller=metrics
time="2020-03-20T10:54:02Z" level=info msg="reconcile complete" controller=metrics elapsed=13.280693ms
time="2020-03-20T10:54:02Z" level=info msg="syncing credentials request" controller=credreq cr=openshift-cloud-credential-operator/openshift-network
time="2020-03-20T10:54:02Z" level=debug msg="ignoring cr as it is for a different cloud" controller=credreq cr=openshift-cloud-credential-operator/openshift-network secret=openshift-network-operator/installer-cloud-credentials
time="2020-03-20T10:54:02Z" level=debug msg="updating credentials request status" controller=credreq cr=openshift-cloud-credential-operator/openshift-network secret=openshift-network-operator/installer-cloud-credentials
time="2020-03-20T10:54:02Z" level=debug msg="status unchanged" controller=credreq cr=openshift-cloud-credential-operator/openshift-network secret=openshift-network-operator/installer-cloud-credentials
time="2020-03-20T10:54:02Z" level=debug msg="syncing cluster operator status" controller=credreq_status
time="2020-03-20T10:54:02Z" level=info msg="Saving hash of proxy CA configmap" configmap=openshift-cloud-credential-operator/cco-trusted-ca controller=configmap
time="2020-03-20T10:54:02Z" level=debug msg="1 cred requests" controller=credreq_status
time="2020-03-20T10:54:02Z" level=debug msg="set ClusterOperator condition" controller=credreq_status message="No credentials requests reporting errors." reason=NoCredentialsFailing status=False type=Degraded
time="2020-03-20T10:54:02Z" level=debug msg="set ClusterOperator condition" controller=credreq_status message="1 of 1 credentials requests provisioned and reconciled." reason=ReconcilingComplete status=False type=Progressing
time="2020-03-20T10:54:02Z" level=debug msg="set ClusterOperator condition" controller=credreq_status message= reason= status=True type=Available
time="2020-03-20T10:54:02Z" level=debug msg="set ClusterOperator condition" controller=credreq_status message= reason= status=True type=Upgradeable

This works, though I didn't expect that the ca_bundle field would exist but be empty in the secret in the kube-system namespace. Is this a product of setting insecure: true?

install-config snippet:

platform:
  ovirt:
    api_vip: 10.50.102.20
    dns_vip: 10.50.102.21
    ingress_vip: 10.50.102.22
    ovirt_cluster_id: 33fe2fa8-eb60-11e9-bb60-00163e186f47
    ovirt_network_name: ocp-network
    ovirt_storage_domain_id: e4323f39-f50b-4462-8df1-fff0dd587a9f
    ovirt_ca_bundle: |
      -----BEGIN CERTIFICATE-----
      MIIDzDCCArSgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwSzELMAkGA1UEBhMCVVMx
      FzAVBgNVBAoMDmNvcmUudWsuZXRpLmlvMSMwIQYDVQQDDBpvdmlydC5jb3JlLnVr
...

I wonder if it expect this in the ~/.ovirt/ovirt-config.yaml file?

Additionally the cluster build completed successfully in ~25mins, which is a good deal faster than before.

[abaxo]

Yeah so judging by the code merged, it is expecting it in the same place as the ovirt_url, ovirt_username and ovirt_password fields... https://github.com/openshift/installer/pull/3322/commits/4d60f8f7bf25de8eb0f4293e0374f1a6dde71bfd

I think thats just a documentation piece. Maybe there is an existing documentation ticket this should be included as part of?

I will see if I can get another build done to see if it picks the secret up from the ~/.ovirt/ovirt-config.yaml file instead. (unless someone else can confirm)

[abaxo]

I just did a fresh build with the ovirt_ca_bundle value in the ~/.ovirt/ovirt-config.yaml and it worked nicely.

[rgolangh]

I get it that this parameter is bit confusing, as it looks like it could belong to the install-config.
I try to keep all the connection related knobs of the platform in ovirt-config.yaml. As far as I see it
this config file is agnostic to openshift - it should be a part of what the ovirt-engine sdk use and load, and I intend to open an issue for that in the project sdk repo.

[abaxo]

Keeping it there makes sense to me too while the other ovirt params exist there. It might be worth considering what happens if you have multiple ovirt hosted engines? For example I have 1 per region in 2 regions (one in the US and one in the UK and don't want to manage them with a single hosted engine. Is it possible to pass in the path to the ovirt config file to use?

[rgolangh]

export OVIRT_CONFIG

https://github.com/openshift/installer/blob/d3acf80d3fff65163d005c2b551d6b024baccec6/pkg/asset/installconfig/ovirt/config.go#L11

[abaxo]

Awesome, thanks :)

I think this can probably be closed now as we now have the credential operator injecting credentials reliably now?

[rgolangh]

Yes, this is verified both OKD and OCP. (I don't have permission to close it)

BTW, what version of ovirt are you using?

[abaxo]

I've been testing over the course of 4.3.7, 4.3.8 and as of yesterday 4.3.9