Okd: openshift-install not creating the worker vm using IPI

[bobby0724]

Describe the bug
I am trying to deploy a new cluster into my vsphere infrastructure using IPI
Bootstrap process completes but the workers are never created

Version

openshift-install 4.5.0-0.okd-2020-10-15-235428
IPI installation
How reproducible

i have try 10 times bootstrap completes but the workers are never created

this is my install-config.yml
apiVersion: v1
baseDomain: okd.local
metadata:
name: lab

compute:

• hyperthreading: Enabled
  name: worker
  replicas: 3
  platform:
  vsphere:
  cpus: 4
  coresPerSocket: 4
  memoryMB: 16384
  osDisk:
  diskSizeGB: 120

controlPlane:
hyperthreading: Enabled
name: master
replicas: 3
platform:
vsphere:
cpus: 4
coresPerSocket: 4
memoryMB: 16384
osDisk:
diskSizeGB: 120

networking:
clusterNetworks:

• cidr: 10.254.0.0/16
  hostPrefix: 23
  networkType: OVNKubernetes
  serviceNetwork:
• 172.30.0.0/16

Log bundle

DEBUG Still waiting for the cluster to initialize: Working towards 4.5.0-0.okd-2020-10-15-235428: 86% complet

[fortinj66]

No workers are created in VMWare or in the cluster?

[bobby0724]

No workers are created in VMWare or in the cluster?

no vm workers are created, only the masters, i also notice that the ingress operator never startup

[fortinj66]

what does the VMWare terminal indicate logwise for the masters?

can you run oc get nodes and oc get events

[bobby0724]

what does the VMWare terminal indicate logwise for the masters?

can you run oc get nodes and oc get events

give me 10 min i will ran the installation and i can get you all the info

[bobby0724]

Here is the log of the cluster creation

You can see the vms created on this image
https://flic.kr/p/2ka6e65

cluster creation log

DEBUG vsphereprivate_import_ova.import: Creating...
DEBUG vsphereprivate_import_ova.import: Still creating... [10s elapsed]
DEBUG vsphereprivate_import_ova.import: Still creating... [20s elapsed]
DEBUG vsphereprivate_import_ova.import: Creation complete after 23s [id=vm-362769]
DEBUG data.vsphere_virtual_machine.template: Refreshing state...
DEBUG module.master.vsphere_virtual_machine.vm[2]: Creating...
DEBUG module.bootstrap.vsphere_virtual_machine.vm: Creating...
DEBUG module.master.vsphere_virtual_machine.vm[1]: Creating...
DEBUG module.master.vsphere_virtual_machine.vm[0]: Creating...
DEBUG module.master.vsphere_virtual_machine.vm[0]: Creation complete after 2s [id=42093637-8ddf-997c-9649-294b67179db6]
DEBUG module.master.vsphere_virtual_machine.vm[2]: Creation complete after 7s [id=420953d6-89f4-38e9-dd92-0447803772e5]
DEBUG module.master.vsphere_virtual_machine.vm[1]: Creation complete after 8s [id=42091a7d-cd34-c6be-342a-15ffb79ccdfc]
DEBUG module.bootstrap.vsphere_virtual_machine.vm: Creation complete after 9s [id=420925b7-151f-c43c-2221-2c851e35855b]
DEBUG
DEBUG Apply complete! Resources: 8 added, 0 changed, 0 destroyed.
DEBUG OpenShift Installer 4.5.0-0.okd-2020-10-15-235428
DEBUG Built from commit 63200c80c431b8dbaa06c0cc13282d819bd7e5f8
INFO Waiting up to 20m0s for the Kubernetes API at https://api.lab.okd.local:6443...
DEBUG Still waiting for the cluster to initialize: Working towards 4.5.0-0.okd-2020-10-15-235428: 86% complete, waiting on authentication, console, ingress, kube-storage-version-migrator, monitoring

$ oc get nodes
NAME STATUS ROLES AGE VERSION
lab-4dgs7-master-0 Ready master 9m12s v1.18.3
lab-4dgs7-master-1 Ready master 9m30s v1.18.3
lab-4dgs7-master-2 Ready master 9m8s v1.18.3

$ oc get events
LAST SEEN TYPE REASON OBJECT MESSAGE
5m52s Normal LeaderElection configmap localhost_98641d9a-e659-4680-9baa-1a536a4c93a9 stopped leading
5m51s Normal LeaderElection endpoints localhost_976bcdba-2098-4571-b664-0b3ec6760a35 stopped leading
4m23s Normal TerminationStart namespace/kube-system Received signal to terminate, becoming unready, but keeping serving
4m23s Normal TerminationPreShutdownHooksFinished namespace/kube-system All pre-shutdown hooks have been finished
9m14s Normal RegisteredNode node/lab-4dgs7-master-0 Node lab-4dgs7-master-0 event: Registered Node lab-4dgs7-master-0 in Controller
5m21s Normal RegisteredNode node/lab-4dgs7-master-0 Node lab-4dgs7-master-0 event: Registered Node lab-4dgs7-master-0 in Controller
2m49s Normal RegisteredNode node/lab-4dgs7-master-0 Node lab-4dgs7-master-0 event: Registered Node lab-4dgs7-master-0 in Controller
2m29s Normal NodeDone node/lab-4dgs7-master-0 Setting node lab-4dgs7-master-0, currentConfig rendered-master-6055e9537d025060e9bcacc2b4afe24e to Done
39s Normal RegisteredNode node/lab-4dgs7-master-0 Node lab-4dgs7-master-0 event: Registered Node lab-4dgs7-master-0 in Controller
9m29s Normal RegisteredNode node/lab-4dgs7-master-1 Node lab-4dgs7-master-1 event: Registered Node lab-4dgs7-master-1 in Controller
5m21s Normal RegisteredNode node/lab-4dgs7-master-1 Node lab-4dgs7-master-1 event: Registered Node lab-4dgs7-master-1 in Controller
2m49s Normal RegisteredNode node/lab-4dgs7-master-1 Node lab-4dgs7-master-1 event: Registered Node lab-4dgs7-master-1 in Controller
2m29s Normal NodeDone node/lab-4dgs7-master-1 Setting node lab-4dgs7-master-1, currentConfig rendered-master-6055e9537d025060e9bcacc2b4afe24e to Done
39s Normal RegisteredNode node/lab-4dgs7-master-1 Node lab-4dgs7-master-1 event: Registered Node lab-4dgs7-master-1 in Controller
9m9s Normal RegisteredNode node/lab-4dgs7-master-2 Node lab-4dgs7-master-2 event: Registered Node lab-4dgs7-master-2 in Controller
5m21s Normal RegisteredNode node/lab-4dgs7-master-2 Node lab-4dgs7-master-2 event: Registered Node lab-4dgs7-master-2 in Controller
2m49s Normal RegisteredNode node/lab-4dgs7-master-2 Node lab-4dgs7-master-2 event: Registered Node lab-4dgs7-master-2 in Controller
105s Normal NodeDone node/lab-4dgs7-master-2 Setting node lab-4dgs7-master-2, currentConfig rendered-master-6055e9537d025060e9bcacc2b4afe24e to Done
39s Normal RegisteredNode node/lab-4dgs7-master-2 Node lab-4dgs7-master-2 event: Registered Node lab-4dgs7-master-2 in Controller
4m21s Normal Status upgrade clusteroperator/machine-api Progressing towards operator: 4.5.0-0.okd-2020-10-15-235428
7m43s Normal OperatorVersionChanged /machine-config clusteroperator/machine-config-operator started a version change from [] to [{operator 4.5.0-0.okd-2020-10-15-235428}]
100s Normal OperatorVersionChanged /machine-config clusteroperator/machine-config-operator version changed from [] to [{operator 4.5.0-0.okd-2020-10-15-235428}]

$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication Unknown Unknown True 7m52s
cloud-credential 4.5.0-0.okd-2020-10-15-235428 True False False 10m
cluster-autoscaler 4.5.0-0.okd-2020-10-15-235428 True False False 3m39s
config-operator 4.5.0-0.okd-2020-10-15-235428 True False False 3m54s
console 4.5.0-0.okd-2020-10-15-235428 Unknown True False 3m49s
csi-snapshot-controller 4.5.0-0.okd-2020-10-15-235428 True False False 4m17s
dns 4.5.0-0.okd-2020-10-15-235428 True False False 7m9s
etcd 4.5.0-0.okd-2020-10-15-235428 True False False 6m14s
image-registry 4.5.0-0.okd-2020-10-15-235428 True False False 4m24s
ingress False True True 3m38s
insights 4.5.0-0.okd-2020-10-15-235428 True False False 4m28s
kube-apiserver 4.5.0-0.okd-2020-10-15-235428 True True True 5m28s
kube-controller-manager 4.5.0-0.okd-2020-10-15-235428 True True False 6m14s
kube-scheduler 4.5.0-0.okd-2020-10-15-235428 True True False 4m31s
kube-storage-version-migrator 4.5.0-0.okd-2020-10-15-235428 False False False 7m53s
machine-api 4.5.0-0.okd-2020-10-15-235428 True False False 96s
machine-approver 4.5.0-0.okd-2020-10-15-235428 True False False 4m18s
machine-config 4.5.0-0.okd-2020-10-15-235428 True False False 2m8s
marketplace 4.5.0-0.okd-2020-10-15-235428 True False False 3m57s
monitoring Unknown True Unknown 4m56s
network 4.5.0-0.okd-2020-10-15-235428 True False False 8m12s
node-tuning 4.5.0-0.okd-2020-10-15-235428 True False False 7m59s
openshift-apiserver 4.5.0-0.okd-2020-10-15-235428 True False False 4m35s
openshift-controller-manager 4.5.0-0.okd-2020-10-15-235428 True False False 4m5s
openshift-samples 4.5.0-0.okd-2020-10-15-235428 True False False 2m55s
operator-lifecycle-manager 4.5.0-0.okd-2020-10-15-235428 True False False 7m29s
operator-lifecycle-manager-catalog 4.5.0-0.okd-2020-10-15-235428 True False False 7m31s
operator-lifecycle-manager-packageserver 4.5.0-0.okd-2020-10-15-235428 True False False 4m38s
service-ca 4.5.0-0.okd-2020-10-15-235428 True False False 7m54s
storage 4.5.0-0.okd-2020-10-15-235428 True False False 4m27s

[fortinj66]

It doesn't looks like the bootstrap process has finished... Nodes get created after the bootstrap process is complete...

try running ./openshift-install gather bootstrap --dir=<installation_directory> and post the results...

(https://docs.okd.io/latest/installing/installing-troubleshooting.html#installation-bootstrap-gather_installing-troubleshooting)

[fortinj66]

also, if possible, you may want to make sure all the nodes are running on different VMWare hosts, or as much as possible... I've had issues were OKD hosts saturated the VMWare host CPU and it caused issues...

[bobby0724]

It doesn't looks like the bootstrap process has finished... Nodes get created after the bootstrap process is complete...

try running ./openshift-install gather bootstrap --dir=<installation_directory> and post the results...

(https://docs.okd.io/latest/installing/installing-troubleshooting.html#installation-bootstrap-gather_installing-troubleshooting)

it does finish look

DEBUG OpenShift Installer 4.5.0-0.okd-2020-10-15-235428
DEBUG Built from commit 63200c80c431b8dbaa06c0cc13282d819bd7e5f8
INFO Waiting up to 20m0s for the Kubernetes API at https://api.lab.okd.local:6443...
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.okd.local:6443/version?timeout=32s: dial tcp 192.168.1.201:6443: connect: connection refused
INFO API v1.18.3 up
INFO Waiting up to 40m0s for bootstrapping to complete... DEBUG Bootstrap status: complete
INFO Destroying the bootstrap resources...
DEBUG Symlinking plugin terraform-provider-aws src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-aws"
DEBUG Symlinking plugin terraform-provider-google src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-google"
DEBUG Symlinking plugin terraform-provider-openstack src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-openstack"
DEBUG Symlinking plugin terraform-provider-vsphere src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-vsphere"
DEBUG Symlinking plugin terraform-provider-random src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-random"
DEBUG Symlinking plugin terraform-provider-vsphereprivate src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-vsphereprivate"
DEBUG Symlinking plugin terraform-provider-azurerm src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-azurerm"
DEBUG Symlinking plugin terraform-provider-azureprivatedns src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-azureprivatedns"
DEBUG Symlinking plugin terraform-provider-ignition src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-ignition"
DEBUG Symlinking plugin terraform-provider-local src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-local"
DEBUG Symlinking plugin terraform-provider-ovirt src: "/usr/local/bin/openshift-install" dst: "/tmp/openshift-install-612627491/plugins/terraform-provider-ovirt"
DEBUG Initializing modules...
DEBUG - bootstrap in ../../../tmp/openshift-install-612627491/bootstrap
DEBUG - master in ../../../tmp/openshift-install-612627491/master

[fortinj66]

The image you sent and the logs didn't show that completing... Bootstrap node was still there...

what vmware hosts are created now?

[bobby0724]

The image you sent and the logs didn't show that completing... Bootstrap node was still there...

what vmware hosts are created now?

No, that is the issue I am having bootstrap completes but workers are never created, the cluster provisioning never completes

[fortinj66]

can you run oc adm must-gather

and another shot of your currently created VMWare nodes?

and check oc get csr

[bobby0724]

can you run oc adm must-gather

and another shot of your currently created VMWare nodes?

and check oc get csr

$ oc get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
csr-2mvx2 8m58s kubernetes.io/kubelet-serving system:node:lab-5jjb9-master-1 Approved,Issued
csr-8qvkh 9m17s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-x6b4s 8m38s kubernetes.io/kubelet-serving system:node:lab-5jjb9-master-2 Approved,Issued
csr-xxx7v 9m3s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-zbt4g 9m1s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-zp5l2 8m40s kubernetes.io/kubelet-serving system:node:lab-5jjb9-master-0 Approved,Issued

the output of the gather command is to big to paste it here what are you looking for
bootstrap vm is deleted every time

[fortinj66]

Do you have dropbox or similar you can put that logfile in?

[bobby0724]

Do you have dropbox or similar you can put that logfile in?

sure

https://www.dropbox.com/s/w09xr5js43btt29/gather.txt?dl=0

[fortinj66]

There should have been a directory created that you need to tar/gzip up... That's what you need to put there:

(https://docs.okd.io/latest/support/gathering-cluster-data.html#support_gathering_data_gathering-cluster-data)

[bobby0724]

There should have been a directory created that you need to tar/gzip up... That's what you need to put there:

(https://docs.okd.io/latest/support/gathering-cluster-data.html#support_gathering_data_gathering-cluster-data)

got it i just upload it
https://www.dropbox.com/sh/5vme5hm379excb9/AAAtfJnhJuIlEvsF7G8qpNcBa?dl=0

[bobby0724]

Bootstrap log

$ openshift-install --dir=/home/okdadmin/okd4lab/vsphere wait-for bootstrap-complete --log-level=debug
DEBUG OpenShift Installer 4.5.0-0.okd-2020-10-15-235428
DEBUG Built from commit 63200c80c431b8dbaa06c0cc13282d819bd7e5f8
INFO Waiting up to 20m0s for the Kubernetes API at https://api.lab.okd.local:6443...
INFO API v1.18.3 up
INFO Waiting up to 40m0s for bootstrapping to complete...
DEBUG Bootstrap status: complete
INFO It is now safe to remove the bootstrap resources
INFO Time elapsed: 0s

Install log
$ openshift-install --dir=/home/okdadmin/okd4lab/vsphere wait-for install-complete --log-level=debug
DEBUG OpenShift Installer 4.5.0-0.okd-2020-10-15-235428
DEBUG Built from commit 63200c80c431b8dbaa06c0cc13282d819bd7e5f8
DEBUG Fetching Install Config...
DEBUG Loading Install Config...
DEBUG Loading SSH Key...
DEBUG Loading Base Domain...
DEBUG Loading Platform...
DEBUG Loading Cluster Name...
DEBUG Loading Base Domain...
DEBUG Loading Platform...
DEBUG Loading Pull Secret...
DEBUG Loading Platform...
DEBUG Using Install Config loaded from state file
DEBUG Reusing previously-fetched Install Config
INFO Waiting up to 30m0s for the cluster at https://api.lab.okd.local:6443 to initialize...
DEBUG Still waiting for the cluster to initialize: Some cluster operators are still updating: authentication, console, ingress, kube-storage-version-migrator, monitoring

[fortinj66]

can you post the installer log? should be <install_dir>/.openshift_install.log

[fortinj66]

I'm looking at my config:

compute:
- hyperthreading: Enabled
  name: worker
  replicas: 3
  platform:
    vsphere:
      cpus: 4
      coresPerSocket: 1
      memoryMB: 24576
controlPlane:
  hyperthreading: Enabled
  name: master
  platform: {}
  replicas: 3

Other than memory, the only difference is coresPerSocket: 1

Can you try that rather than coresPerSocket: 4

[bobby0724]

Yes that not the issue
for some reason the installer is not creating the worker nodes, but also the cluster is not getting ready the ingress project never comes up :( i will stop here at least the UPI is working

[fortinj66]

Ingress won't come up without worker nodes...

I just rebuilt my cluster with IPI without issues... and with coresPerSocker at 4...

what happens if you run oc get events --all-namespaces | grep openshift-machine-api

I get

[root@os-utils-d02 logs]# oc get events --all-namespaces | grep  "Created Machine"
openshift-machine-api                              32m         Normal    Create                                       machine/dev-c1v4-mfpr9-worker-jmk8q                               Created Machine dev-c1v4-mfpr9-worker-jmk8q
openshift-machine-api                              32m         Normal    Create                                       machine/dev-c1v4-mfpr9-worker-rw6pt                               Created Machine dev-c1v4-mfpr9-worker-rw6pt
openshift-machine-api                              32m         Normal    Create                                       machine/dev-c1v4-mfpr9-worker-xbmgg                               Created Machine dev-c1v4-mfpr9-worker-xbmgg
[root@os-utils-d02 logs]#

[fortinj66]

The only other thing I would try would be to take out the worker changes you made and try vanilla:

compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform: {}
  replicas: 3
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform: {}
  replicas: 3

[fortinj66]

Here's your issue.

There are a bunch of these in must-gather.local.331163555932549138/quay-io-openshift-okd-content-sha256-5a9b4b4cdd15e8587d02490cf753c385213e6a97e5c2bf7babf416585e42fd70/namespaces/openshift-machine-api/pods/machine-api-controllers-6c4f6c6dc8-vd8x5/machine-controller/machine-controller/logs/current.log:

namespaces/openshift-machine-api/pods/machine-api-controllers-6c4f6c6dc8-vd8x5/machine-controller/machine-controller/logs/current.log:2020-11-24T21:13:43.289573655Z E1124 21:13:43.289519 1 controller.go:272] lab-5jjb9-worker-v9sp8: failed to check if machine exists: lab-5jjb9-worker-v9sp8: failed to create scope for machine: failed to create vSphere session: error setting up new vSphere SOAP client: Post https://sacla300.sac.flex.com/sdk: dial tcp: lookup sacla300.sac.flex.com on 172.30.0.10:53: no such host

It can't find your vSphere host in DNS

In mine I get:

sh-4.2$ nslookup vcenter.maeagle.corp
Server:         172.30.0.10
Address:        172.30.0.10#53

Name:   vcenter.maeagle.corp
Address: 10.199.202.180

Once you figure out what is going on with DNS I'll bet this will work. First thing I would check is can you resolve it from one of the master hosts... what does the /etc/resolve.conf look like.

[bobby0724]

Here's your issue.

There are a bunch of these in must-gather.local.331163555932549138/quay-io-openshift-okd-content-sha256-5a9b4b4cdd15e8587d02490cf753c385213e6a97e5c2bf7babf416585e42fd70/namespaces/openshift-machine-api/pods/machine-api-controllers-6c4f6c6dc8-vd8x5/machine-controller/machine-controller/logs/current.log:

namespaces/openshift-machine-api/pods/machine-api-controllers-6c4f6c6dc8-vd8x5/machine-controller/machine-controller/logs/current.log:2020-11-24T21:13:43.289573655Z E1124 21:13:43.289519 1 controller.go:272] lab-5jjb9-worker-v9sp8: failed to check if machine exists: lab-5jjb9-worker-v9sp8: failed to create scope for machine: failed to create vSphere session: error setting up new vSphere SOAP client: Post https://sacla300.sac.flex.com/sdk: dial tcp: lookup sacla300.sac.flex.com on 172.30.0.10:53: no such host

It can't find your vSphere host in DNS

Thank you i really appreciate that you take the time to review the logs i was also doing that, i think i know why is that happening i will apply a workaround a retry rebuild

[fortinj66]

You're welcome!

I'm learning too and it give me the opportunity to deep dive into issues to better understand the systems...

[fortinj66]

Were you able to resolve the DNS issue and complete the install?

[bobby0724]

Were you able to resolve the DNS issue and complete the install?

yes adding internal forwarder fix the issue

[fortinj66]

awesome! good luck!

[mikelo]

I think this issue should be reopened. I had misconfigured DNS (vcenter DN was not getting resolved) but the installer worked fine until it was time to create the worker nodes, which AFAIK is done by the master nodes.

This means the installer uses a different DNS VS the master nodes, the installer IMO should run a dig vCenter.url at the beginning and stop in case the resolution fails.

(tested on 4.6.12)

cc @mariosomma

[openshift-ci-robot]

@mikelo: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[swefredde]

Hi!

Thank you guys for this informative troubleshooting. I have the exact same symptoms as @bobby0724 (no worker nodes are starting) but running in ovirt. By following the steps taken by @fortinj66 i found the likely error in the logs from

namespace: openshift-machine-api
pod: machine-api-controller-XYXYXY
container: machine-controller

{"level":"error","ts":1613987970.6140544,"logger":"controller","msg":"Reconciler error","controller":"provdierID-controller","name":"shift-4prmr-master-1","namespace":"","error":"failed getting VM from oVirt: Failed to parse CA Certificate in file '/tmp/ovirt-ca-bundle933032459'","stacktrace":"github.com/go-logr/zapr.(zapLogger).Errorn\t/go/cluster-api-provider-ovirt/vendor/github.com/go-logr/zapr/zapr.go:132nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandlern\t/go/cluster-api-provider-ovirt/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:237nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItemn\t/go/cluster-api-provider-ovirt/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).workern\t/go/cluster-api-provider-ovirt/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:188nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1n\t/go/cluster-api-provider-ovirt/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155nk8s.io/apimachinery/pkg/util/wait.BackoffUntiln\t/go/cluster-api-provider-ovirt/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156nk8s.io/apimachinery/pkg/util/wait.JitterUntiln\t/go/cluster-api-provider-ovirt/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133nk8s.io/apimachinery/pkg/util/wait.Untiln\t/go/cluster-api-provider-ovirt/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90"}

I made a custom install andI pasted the oVirt CA in the file ovirt-config.yaml and my private CA in the file install-config.yaml.

I have a "base" like sub.domain.tld where i install all openshift nodes but my ovirt API is at domain.tld (both in private IP-range (192.168.0.0/16)). I have all DNS issues resolved (finally) so I can resolve hosts and IP's from both domain.tld and sub.domain.tld.

Any suggestions why there is a failure to parse my ovirt certificate?

My CA's are self signed and before when I ran code ready containers I had to give a secret with an annotation to be able to download source from my private git server but I thought that was what I was doing by adding my CA's to the custom config files before running the installation. Did I miss anything?

Regards and happy hacking,
/F

[swefredde]

Hmm, didn't realize this was closed...

If someone stumble on this I thought I could tell what was the problem.

When giving the info in the beginning of the installation I gave the FQDN[:PORT] to my ovirt-engine. Since I gave port 443 a Certificate was presented and I was asked if I wanted to trust it or not. Since I just had given the URL to my ovirt-engine where I have my custom certificate and the cn=fqdn was correct I accepted (a little to fast it showed). I assumed the presented certificate was my installed custom certificate but it must have been either the original ovirt certificate or an installer generated cert.

However, when I answered No I was asked to paste the real cert and then it worked.

/F