Installer: Still waiting for the Kubernetes API: the server could not find the requested resource

Created on 15 Jan 2020  Â·  35Comments  Â·  Source: openshift/installer

Version

$ openshift-install version
./openshift-install unreleased-master-2361-g6a0eb9136b57a8e2609554b0eaeed3a3534e8cab
built from commit 6a0eb9136b57a8e2609554b0eaeed3a3534e8cab
release image registry.svc.ci.openshift.org/origin/release:4.3

Platform:

What happened?

I run "./installer/bin/openshift-install --dir install-dir create cluster"
and can see the message:
INFO Obtaining RHCOS image file from 'https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.4/44.81.202001030903.0/x86_64/rhcos-44.81.202001030903.0-qemu.x86_64.qcow2.gz?sha256=a7931dc062f4dcd1f614e487bac8c53699d54c78c84caa7b528a52f6788b5504'
INFO The file was found in cache: /root/.cache/openshift-installer/image_cache/b9c268865d8b4d59ae24cb5ebdaee1b7. Reusing...
INFO Creating infrastructure resources...
INFO Waiting up to 30m0s for the Kubernetes API at https://api.mycluster.5u04.info:6443...

When I check with

racut:/# [ 1687.486884] ignition[1600]: GET https://api-int.mycluster.5u04.info:22623/config/master: attempt #269
exit[ 1692.487719] ignition[1600]: GET https://api-int.mycluster.5u04.info:22623/config/master: attempt #270

ping api.mycluster.5u04.info

PING api.mycluster.5u04.info (192.168.126.10) 56(84) bytes of data.
64 bytes from api.mycluster.5u04.info (192.168.126.10): icmp_seq=1 ttl=64 time=0.147 ms
64 bytes from api.mycluster.5u04.info (192.168.126.10): icmp_seq=2 ttl=64 time=0.328 ms

ping is fine in my baremetal machine but I see "unreachable" error from VM continuously.

What you expected to happen?

Unrechearble error is removed.

How to reproduce it (as minimally and precisely as possible)?

All the time.

$ your-commands-here

Anything else we need to know?

As a result:
INFO Obtaining RHCOS image file from 'https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.4/44.81.202001030903.0/x86_64/rhcos-44.81.202001030903.0-qemu.x86_64.qcow2.gz?sha256=a7931dc062f4dcd1f614e487bac8c53699d54c78c84caa7b528a52f6788b5504'
INFO The file was found in cache: /root/.cache/openshift-installer/image_cache/b9c268865d8b4d59ae24cb5ebdaee1b7. Reusing...
INFO Creating infrastructure resources...
INFO Waiting up to 30m0s for the Kubernetes API at https://api.mycluster.5u04.info:6443...
ERROR Attempted to gather ClusterOperator status after installation failure: listing ClusterOperator objects: the server could not find the requested resource (get clusteroperators.config.openshift.io)
INFO Pulling debug logs from the bootstrap machine
ERROR Attempted to gather debug logs after installation failure: failed to create SSH client, ensure the proper ssh key is in your keyring or specify with --key: dial tcp 185.53.177.30:22: connect: connection timed out
FATAL Bootstrap failed to complete: waiting for Kubernetes API: context deadline exceeded

References

  • enter text here.
lifecyclrotten platforlibvirt triagneeds-information
Source
picture doyoungim999

Most helpful comment

I finally got a running mini-cluster deployed on libvirt.

The Workaround with using OCP images was the key:

  1. Put every suggestion from https://github.com/openshift/installer/blob/master/docs/dev/libvirt/README.md in place
  2. Patch Installer source for RAM on bootstrap to 16GB (installer/pkg/asset/machines/libvirt/machines.go)
  3. Build the installer for libvirt and 4.3
  4. Set export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=quay.io/openshift-release-dev/ocp-release:4.3.1-x86_64 before running the installer.

A proper pull secret from RedHat has to be entered for usage of OCP images for install-config.

As a result i have a master and a worker running.

picture thdevelthreesixtiy on 12 Mar 2020
👍3

All 35 comments

Please specify the platform type: aws, libvirt, openstack, baremetal, or none (UPI)

pierreprinetti picture pierreprinetti on 23 Jan 2020

Platform is libvirt

doyoungim999 picture doyoungim999 on 24 Jan 2020

can you include the .openshift_install.log ?

abhinavdahiya picture abhinavdahiya on 28 Jan 2020

Hi I attached the last part which is issues:
What is the ip for api and etcd ?
I added in /etc/hosts
192.168.126.10 api.ocp.5u04.info api-int.ocp.5u04.info

192.168.126.11 etcd.ocp.5u04.info

time="2020-01-28T22:05:25-06:00" level=debug msg="libvirt_domain.master[0]: Creation complete after 1s [id=4fe0f2ea-60f8-42bf-b5a0-10fba8b4c8de]"
time="2020-01-28T22:05:25-06:00" level=debug
time="2020-01-28T22:05:25-06:00" level=debug msg="Apply complete! Resources: 9 added, 0 changed, 0 destroyed."
time="2020-01-28T22:05:25-06:00" level=debug msg="OpenShift Installer unreleased-master-2447-g1cc6935753842458bb84504f8e247ddc86cb93c7-dirty"
time="2020-01-28T22:05:25-06:00" level=debug msg="Built from commit 1cc6935753842458bb84504f8e247ddc86cb93c7"
time="2020-01-28T22:05:25-06:00" level=info msg="Waiting up to 20m0s for the Kubernetes API at https://api.ocp.5u04.info:6443..."
time="2020-01-28T22:05:28-06:00" level=debug msg="Still waiting for the Kubernetes API: Get https://api.ocp.5u04.info:6443/version?timeout=32s: dial tcp 192.168.122.10:6443: connect: no route to host"
time="2020-01-28T22:06:14-06:00" level=debug msg="Still waiting for the Kubernetes API: Get https://api.ocp.5u04.info:6443/version?timeout=32s: dial tcp 192.168.122.10:6443: connect: no route to host"
time="2020-01-28T22:06:44-06:00" level=debug msg="Still waiting for the Kubernetes API: Get https://api.ocp.5u04.info:6443/version?timeout=32s: dial tcp 192.168.126.10:6443: connect: connection refused"
time="2020-01-28T22:07:14-06:00" level=debug msg="Still waiting for the Kubernetes API: Get https://api.ocp.5u04.info:6443/version?timeout=32s: dial tcp 192.168.126.10:6443: connect: connection refused"
time="2020-01-28T22:07:36-06:00" level=debug msg="Still waiting for the Kubernetes API: Get https://api.ocp.5u04.info:6443/version?timeout=32s: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kube-apiserver-lb-signer\")"
time="2020-01-28T22:08:06-06:00" level=debug msg="Still waiting for the Kubernetes API: Get https://api.ocp.5u04.info:6443/version?timeout=32s: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kube-apiserver-lb-signer\")"

doyoungim999 picture doyoungim999 on 29 Jan 2020

I found It does not matter with my /etc/hosts file. domain url is looking for their ip.

api server is not started ?

time="2020-01-28T22:42:04-06:00" level=info msg="Waiting up to 20m0s for the Kubernetes API at https://api.mycluster.5u04.info:6443..."
time="2020-01-28T22:42:07-06:00" level=debug msg="Still waiting for the Kubernetes API: Get https://api.mycluster.5u04.info:6443/version?timeout=32s: dial tcp 192.168.126.10:6443: connect: no route to host"
time="2020-01-28T22:42:15-06:00" level=debug msg="Still waiting for the Kubernetes API: Get https://api.mycluster.5u04.info:6443/version?timeout=32s: dial tcp 192.168.126.10:6443: connect: connection refused"
time="2020-01-28T22:42:45-06:00" level=debug msg="Still waiting for the Kubernetes API: Get https://api.mycluster.5u04.info:6443/version?timeout=32s: dial tcp 192.168.126.10:6443: connect: connection refused"
time="2020-01-28T22:43:15-06:00" level=debug msg="Still waiting for the Kubernetes API: Get https://api.mycluster.5u04.info:6443/version?timeout=32s: dial tcp 192.168.126.10:6443: connect: connection refused"
time="2020-01-28T22:43:48-06:00" level=debug msg="Still waiting for the Kubernetes API: Get https://api.mycluster.5u04.info:6443/version?timeout=32s: dial tcp 192.168.126.10:6443: connect: connection refused"
time="2020-01-28T22:44:10-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T22:44:40-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"

doyoungim999 picture doyoungim999 on 29 Jan 2020

Hi
I am using libvirt.
from the issue#1909 ( https://github.com/openshift/installer/issues/1909)
I can find out the same issue is resolved after he changed to
registry.svc.ci.openshift.org/origin/release:4.2.
i cannot find out in virtlib image. Where can I fix it ?

doyoungim999 picture doyoungim999 on 29 Jan 2020

This is whole log:
time="2020-01-28T23:28:44-06:00" level=debug msg="OpenShift Installer unreleased-master-2447-g1cc6935753842458bb84504f8e247ddc86cb93c7-dirty"
time="2020-01-28T23:28:44-06:00" level=debug msg="Built from commit 1cc6935753842458bb84504f8e247ddc86cb93c7"
time="2020-01-28T23:28:44-06:00" level=debug msg="Fetching Master Machines..."
time="2020-01-28T23:28:44-06:00" level=debug msg="Loading Master Machines..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Loading Cluster ID..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Loading SSH Key..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Loading Base Domain..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Loading Platform..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Loading Cluster Name..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Loading Base Domain..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Loading Platform..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Loading Pull Secret..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Loading Platform..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Loading Platform Credentials Check..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Loading Image..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Loading Master Ignition Config..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Loading Root CA..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Fetching Cluster ID..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Fetching SSH Key..."
time="2020-01-28T23:28:44-06:00" level=debug msg=" Generating SSH Key..."
time="2020-01-28T23:28:47-06:00" level=debug msg=" Fetching Base Domain..."
time="2020-01-28T23:28:47-06:00" level=debug msg=" Fetching Platform..."
time="2020-01-28T23:28:47-06:00" level=debug msg=" Generating Platform..."
time="2020-01-28T23:28:59-06:00" level=debug msg=" Generating Base Domain..."
time="2020-01-28T23:29:05-06:00" level=debug msg=" Fetching Cluster Name..."
time="2020-01-28T23:29:05-06:00" level=debug msg=" Fetching Base Domain..."
time="2020-01-28T23:29:05-06:00" level=debug msg=" Reusing previously-fetched Base Domain"
time="2020-01-28T23:29:05-06:00" level=debug msg=" Fetching Platform..."
time="2020-01-28T23:29:05-06:00" level=debug msg=" Reusing previously-fetched Platform"
time="2020-01-28T23:29:05-06:00" level=debug msg=" Generating Cluster Name..."
time="2020-01-28T23:29:08-06:00" level=debug msg=" Fetching Pull Secret..."
time="2020-01-28T23:29:08-06:00" level=debug msg=" Generating Pull Secret..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Platform..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Platform"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Generating Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Generating Cluster ID..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Platform Credentials Check..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Generating Platform Credentials Check..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Image..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Generating Image..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Master Ignition Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Root CA..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Generating Root CA..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Generating Master Ignition Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg="Generating Master Machines..."
time="2020-01-28T23:29:16-06:00" level=debug msg="Fetching Worker Machines..."
time="2020-01-28T23:29:16-06:00" level=debug msg="Loading Worker Machines..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Cluster ID..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Platform Credentials Check..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Image..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Worker Ignition Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Root CA..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Cluster ID..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Cluster ID"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Platform Credentials Check..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Platform Credentials Check"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Image..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Image"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Worker Ignition Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Root CA..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Root CA"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Generating Worker Ignition Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg="Generating Worker Machines..."
time="2020-01-28T23:29:16-06:00" level=debug msg="Fetching Common Manifests..."
time="2020-01-28T23:29:16-06:00" level=debug msg="Loading Common Manifests..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Cluster ID..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Ingress Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading DNS Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Cluster ID..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Platform Credentials Check..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Infrastructure Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Cluster ID..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Cloud Provider Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Cluster ID..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Platform Credentials Check..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Additional Trust Bundle Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Network Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Network CRDs..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Proxy Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Network Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Scheduler Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Image Content Source Policy..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Root CA..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Certificate (etcd-signer)..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Certificate (etcd-ca-bundle)..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Certificate (etcd-signer)..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Certificate (etcd-client)..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Certificate (etcd-signer)..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Certificate (etcd-metric-ca-bundle)..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Certificate (etcd-metric-signer)..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Certificate (etcd-metric-signer)..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Certificate (etcd-metric-signer-client)..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Certificate (etcd-metric-signer)..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Certificate (mcs)..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Root CA..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading CVOOverrides..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading EtcdCAConfigMap..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading EtcdClientSecret..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading EtcdHostServiceEndpoints..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading EtcdHostService..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading EtcdMetricClientSecret..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading EtcdMetricServingCAConfigMap..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading EtcdMetricSignerSecret..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading EtcdNamespace..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading EtcdService..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading EtcdSignerSecret..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading KubeCloudConfig..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading EtcdServingCAConfigMap..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading KubeSystemConfigmapRootCA..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading MachineConfigServerTLSSecret..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading OpenshiftConfigSecretPullSecret..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Loading OpenshiftMachineConfigOperator..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Cluster ID..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Cluster ID"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Ingress Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Generating Ingress Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching DNS Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Cluster ID..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Cluster ID"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Platform Credentials Check..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Platform Credentials Check"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Generating DNS Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Infrastructure Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Cluster ID..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Cluster ID"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Cloud Provider Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Cluster ID..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Cluster ID"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Platform Credentials Check..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Platform Credentials Check"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Generating Cloud Provider Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Additional Trust Bundle Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Generating Additional Trust Bundle Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Generating Infrastructure Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Network Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Network CRDs..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Generating Network CRDs..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Generating Network Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Proxy Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Network Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Network Config"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Generating Proxy Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Scheduler Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Generating Scheduler Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Image Content Source Policy..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Generating Image Content Source Policy..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Root CA..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Root CA"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Certificate (etcd-signer)..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Generating Certificate (etcd-signer)..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Certificate (etcd-ca-bundle)..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Certificate (etcd-signer)..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Certificate (etcd-signer)"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Generating Certificate (etcd-ca-bundle)..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Certificate (etcd-client)..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Fetching Certificate (etcd-signer)..."
time="2020-01-28T23:29:16-06:00" level=debug msg=" Reusing previously-fetched Certificate (etcd-signer)"
time="2020-01-28T23:29:16-06:00" level=debug msg=" Generating Certificate (etcd-client)..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching Certificate (etcd-metric-ca-bundle)..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching Certificate (etcd-metric-signer)..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating Certificate (etcd-metric-signer)..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating Certificate (etcd-metric-ca-bundle)..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching Certificate (etcd-metric-signer)..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Reusing previously-fetched Certificate (etcd-metric-signer)"
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching Certificate (etcd-metric-signer-client)..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching Certificate (etcd-metric-signer)..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Reusing previously-fetched Certificate (etcd-metric-signer)"
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating Certificate (etcd-metric-signer-client)..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching Certificate (mcs)..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching Root CA..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Reusing previously-fetched Root CA"
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating Certificate (mcs)..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching CVOOverrides..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating CVOOverrides..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching EtcdCAConfigMap..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating EtcdCAConfigMap..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching EtcdClientSecret..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating EtcdClientSecret..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching EtcdHostServiceEndpoints..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating EtcdHostServiceEndpoints..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching EtcdHostService..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating EtcdHostService..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching EtcdMetricClientSecret..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating EtcdMetricClientSecret..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching EtcdMetricServingCAConfigMap..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating EtcdMetricServingCAConfigMap..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching EtcdMetricSignerSecret..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating EtcdMetricSignerSecret..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching EtcdNamespace..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating EtcdNamespace..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching EtcdService..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating EtcdService..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching EtcdSignerSecret..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating EtcdSignerSecret..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching KubeCloudConfig..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating KubeCloudConfig..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching EtcdServingCAConfigMap..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating EtcdServingCAConfigMap..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching KubeSystemConfigmapRootCA..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating KubeSystemConfigmapRootCA..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching MachineConfigServerTLSSecret..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating MachineConfigServerTLSSecret..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching OpenshiftConfigSecretPullSecret..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating OpenshiftConfigSecretPullSecret..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching OpenshiftMachineConfigOperator..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating OpenshiftMachineConfigOperator..."
time="2020-01-28T23:29:17-06:00" level=debug msg="Generating Common Manifests..."
time="2020-01-28T23:29:17-06:00" level=debug msg="Fetching Openshift Manifests..."
time="2020-01-28T23:29:17-06:00" level=debug msg="Loading Openshift Manifests..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Loading Cluster ID..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Loading Kubeadmin Password..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Loading CloudCredsSecret..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Loading KubeadminPasswordSecret..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Loading RoleCloudCredsSecretReader..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Loading Private Cluster Outbound Service..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Loading Baremetal Config CR..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching Cluster ID..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Reusing previously-fetched Cluster ID"
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching Kubeadmin Password..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating Kubeadmin Password..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching CloudCredsSecret..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating CloudCredsSecret..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching KubeadminPasswordSecret..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating KubeadminPasswordSecret..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching RoleCloudCredsSecretReader..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating RoleCloudCredsSecretReader..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching Private Cluster Outbound Service..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating Private Cluster Outbound Service..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Fetching Baremetal Config CR..."
time="2020-01-28T23:29:17-06:00" level=debug msg=" Generating Baremetal Config CR..."
time="2020-01-28T23:29:17-06:00" level=debug msg="Generating Openshift Manifests..."
time="2020-01-28T23:29:53-06:00" level=debug msg="OpenShift Installer unreleased-master-2447-g1cc6935753842458bb84504f8e247ddc86cb93c7-dirty"
time="2020-01-28T23:29:53-06:00" level=debug msg="Built from commit 1cc6935753842458bb84504f8e247ddc86cb93c7"
time="2020-01-28T23:29:53-06:00" level=debug msg="Fetching Terraform Variables..."
time="2020-01-28T23:29:53-06:00" level=debug msg="Loading Terraform Variables..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Cluster ID..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading SSH Key..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using SSH Key loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Base Domain..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Platform..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Platform loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Base Domain loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Cluster Name..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Base Domain..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Platform..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Cluster Name loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Pull Secret..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Pull Secret loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Platform..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Install Config loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Cluster ID loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Image..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Image loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading BootstrapImage..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Bootstrap Ignition Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Kubeconfig Admin Internal Client..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (admin-kubeconfig-client)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (admin-kubeconfig-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-complete-server-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-localhost-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-localhost-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-service-network-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-service-network-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-lb-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-lb-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Kubeconfig Kubelet..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-complete-server-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kubelet-client)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kubelet-bootstrap-kubeconfig-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Kubeconfig Admin Client (Loopback)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (admin-kubeconfig-client)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-localhost-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Master Machines..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Cluster ID..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Platform Credentials Check..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Platform Credentials Check loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Image..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Master Ignition Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Root CA..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Root CA loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Master Ignition Config loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Master Machines from both state file and target directory"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Master Machines loaded from target directory"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Worker Machines..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Cluster ID..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Platform Credentials Check..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Image..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Worker Ignition Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Root CA..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Worker Ignition Config loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Worker Machines from both state file and target directory"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Worker Machines loaded from target directory"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Common Manifests..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Cluster ID..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Ingress Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Ingress Config loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading DNS Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Cluster ID..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Platform Credentials Check..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using DNS Config loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Infrastructure Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Cluster ID..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Cloud Provider Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Cluster ID..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Platform Credentials Check..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Cloud Provider Config loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Additional Trust Bundle Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Additional Trust Bundle Config loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Infrastructure Config loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Network Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Network CRDs..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Network CRDs loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Network Config loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Proxy Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Network Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Proxy Config loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Scheduler Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Scheduler Config loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Image Content Source Policy..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Image Content Source Policy loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Root CA..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (etcd-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Certificate (etcd-signer) loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (etcd-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (etcd-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Certificate (etcd-ca-bundle) loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (etcd-client)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (etcd-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Certificate (etcd-client) loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (etcd-metric-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (etcd-metric-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Certificate (etcd-metric-signer) loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Certificate (etcd-metric-ca-bundle) loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (etcd-metric-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (etcd-metric-signer-client)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (etcd-metric-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Certificate (etcd-metric-signer-client) loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (mcs)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Root CA..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Certificate (mcs) loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading CVOOverrides..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using CVOOverrides loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading EtcdCAConfigMap..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using EtcdCAConfigMap loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading EtcdClientSecret..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using EtcdClientSecret loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading EtcdHostServiceEndpoints..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using EtcdHostServiceEndpoints loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading EtcdHostService..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using EtcdHostService loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading EtcdMetricClientSecret..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using EtcdMetricClientSecret loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading EtcdMetricServingCAConfigMap..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using EtcdMetricServingCAConfigMap loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading EtcdMetricSignerSecret..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using EtcdMetricSignerSecret loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading EtcdNamespace..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using EtcdNamespace loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading EtcdService..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using EtcdService loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading EtcdSignerSecret..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using EtcdSignerSecret loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading KubeCloudConfig..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using KubeCloudConfig loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading EtcdServingCAConfigMap..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using EtcdServingCAConfigMap loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading KubeSystemConfigmapRootCA..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using KubeSystemConfigmapRootCA loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading MachineConfigServerTLSSecret..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using MachineConfigServerTLSSecret loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading OpenshiftConfigSecretPullSecret..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using OpenshiftConfigSecretPullSecret loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading OpenshiftMachineConfigOperator..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using OpenshiftMachineConfigOperator loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Common Manifests from both state file and target directory"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Common Manifests loaded from target directory"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Openshift Manifests..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Cluster ID..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Kubeadmin Password..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Kubeadmin Password loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading CloudCredsSecret..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using CloudCredsSecret loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading KubeadminPasswordSecret..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using KubeadminPasswordSecret loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading RoleCloudCredsSecretReader..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using RoleCloudCredsSecretReader loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Private Cluster Outbound Service..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Private Cluster Outbound Service loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Baremetal Config CR..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Baremetal Config CR loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Openshift Manifests from both state file and target directory"
time="2020-01-28T23:29:53-06:00" level=debug msg=" On-disk Openshift Manifests matches asset in state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Using Openshift Manifests loaded from state file"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Proxy Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (admin-kubeconfig-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (admin-kubeconfig-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (aggregator)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (aggregator-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (aggregator-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (system:kube-apiserver-proxy)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (aggregator-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (aggregator-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (system:kube-apiserver-proxy)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (aggregator)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (etcd-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (etcd-metric-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (etcd-metric-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (etcd-metric-signer-client)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (etcd-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (etcd-client)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (journal-gatewayd)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Root CA..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-lb-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-external-lb-server)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-lb-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-internal-lb-server)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-lb-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-lb-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-localhost-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-localhost-server)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-localhost-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-localhost-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-service-network-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-service-network-server)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-service-network-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-service-network-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-complete-server-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-complete-client-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (admin-kubeconfig-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kubelet-client-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kubelet-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-control-plane-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-control-plane-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-lb-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-localhost-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-service-network-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-to-kubelet-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-to-kubelet-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kubelet-bootstrap-kubeconfig-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-to-kubelet-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-to-kubelet-client)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-to-kubelet-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-apiserver-to-kubelet-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-control-plane-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-control-plane-kube-controller-manager-client)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-control-plane-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-control-plane-kube-scheduler-client)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-control-plane-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kube-control-plane-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kubelet-client-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kubelet-client)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kubelet-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kubelet-serving-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (kubelet-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Certificate (mcs)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Root CA..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Key Pair (service-account.pub)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Release Image Pull Spec..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Image..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Master Ignition Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Master Machines..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Loading Worker Machines..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Fetching Cluster ID..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Reusing previously-fetched Cluster ID"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Fetching Image..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Reusing previously-fetched Image"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Fetching BootstrapImage..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Generating BootstrapImage..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Fetching Bootstrap Ignition Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:53-06:00" level=debug msg=" Fetching Kubeconfig Admin Internal Client..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Fetching Certificate (admin-kubeconfig-client)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Fetching Certificate (admin-kubeconfig-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Generating Certificate (admin-kubeconfig-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Generating Certificate (admin-kubeconfig-client)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-complete-server-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-localhost-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-localhost-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Generating Certificate (kube-apiserver-localhost-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Generating Certificate (kube-apiserver-localhost-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-service-network-ca-bundle)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-service-network-signer)..."
time="2020-01-28T23:29:53-06:00" level=debug msg=" Generating Certificate (kube-apiserver-service-network-signer)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Generating Certificate (kube-apiserver-service-network-ca-bundle)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-lb-ca-bundle)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-lb-signer)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Generating Certificate (kube-apiserver-lb-signer)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Generating Certificate (kube-apiserver-lb-ca-bundle)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Generating Certificate (kube-apiserver-complete-server-ca-bundle)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:54-06:00" level=debug msg=" Generating Kubeconfig Admin Internal Client..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Kubeconfig Kubelet..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-complete-server-ca-bundle)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-apiserver-complete-server-ca-bundle)"
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Certificate (kubelet-client)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Certificate (kubelet-bootstrap-kubeconfig-signer)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Generating Certificate (kubelet-bootstrap-kubeconfig-signer)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Generating Certificate (kubelet-client)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:54-06:00" level=debug msg=" Generating Kubeconfig Kubelet..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Kubeconfig Admin Client (Loopback)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Certificate (admin-kubeconfig-client)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Reusing previously-fetched Certificate (admin-kubeconfig-client)"
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-localhost-ca-bundle)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-apiserver-localhost-ca-bundle)"
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:54-06:00" level=debug msg=" Generating Kubeconfig Admin Client (Loopback)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Master Machines..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Reusing previously-fetched Master Machines"
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Worker Machines..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Reusing previously-fetched Worker Machines"
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Common Manifests..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Reusing previously-fetched Common Manifests"
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Openshift Manifests..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Reusing previously-fetched Openshift Manifests"
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Proxy Config..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Reusing previously-fetched Proxy Config"
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Certificate (admin-kubeconfig-ca-bundle)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Certificate (admin-kubeconfig-signer)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Reusing previously-fetched Certificate (admin-kubeconfig-signer)"
time="2020-01-28T23:29:54-06:00" level=debug msg=" Generating Certificate (admin-kubeconfig-ca-bundle)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Certificate (aggregator)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Generating Certificate (aggregator)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Certificate (aggregator-ca-bundle)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Fetching Certificate (aggregator-signer)..."
time="2020-01-28T23:29:54-06:00" level=debug msg=" Generating Certificate (aggregator-signer)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Generating Certificate (aggregator-ca-bundle)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (system:kube-apiserver-proxy)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (aggregator-signer)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Certificate (aggregator-signer)"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Generating Certificate (system:kube-apiserver-proxy)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (aggregator-signer)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Certificate (aggregator-signer)"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (system:kube-apiserver-proxy)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (aggregator)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Certificate (aggregator)"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Generating Certificate (system:kube-apiserver-proxy)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (etcd-ca-bundle)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Certificate (etcd-ca-bundle)"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (etcd-metric-ca-bundle)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Certificate (etcd-metric-ca-bundle)"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (etcd-metric-signer)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Certificate (etcd-metric-signer)"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (etcd-metric-signer-client)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Certificate (etcd-metric-signer-client)"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (etcd-signer)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Certificate (etcd-signer)"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (etcd-client)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Certificate (etcd-client)"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (journal-gatewayd)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Root CA..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Root CA"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Generating Certificate (journal-gatewayd)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-lb-ca-bundle)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-apiserver-lb-ca-bundle)"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-external-lb-server)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-lb-signer)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-apiserver-lb-signer)"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Generating Certificate (kube-apiserver-external-lb-server)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-internal-lb-server)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-lb-signer)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-apiserver-lb-signer)"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Generating Certificate (kube-apiserver-internal-lb-server)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-lb-signer)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-apiserver-lb-signer)"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-localhost-ca-bundle)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-apiserver-localhost-ca-bundle)"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-localhost-server)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-localhost-signer)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-apiserver-localhost-signer)"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Generating Certificate (kube-apiserver-localhost-server)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-localhost-signer)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-apiserver-localhost-signer)"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-service-network-ca-bundle)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-apiserver-service-network-ca-bundle)"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-service-network-server)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-service-network-signer)..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-apiserver-service-network-signer)"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:55-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:55-06:00" level=debug msg=" Generating Certificate (kube-apiserver-service-network-server)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-service-network-signer)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-apiserver-service-network-signer)"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-complete-server-ca-bundle)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-apiserver-complete-server-ca-bundle)"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-complete-client-ca-bundle)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (admin-kubeconfig-ca-bundle)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Certificate (admin-kubeconfig-ca-bundle)"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kubelet-client-ca-bundle)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kubelet-signer)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Generating Certificate (kubelet-signer)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Generating Certificate (kubelet-client-ca-bundle)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kube-control-plane-ca-bundle)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kube-control-plane-signer)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Generating Certificate (kube-control-plane-signer)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-lb-signer)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-apiserver-lb-signer)"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-localhost-signer)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-apiserver-localhost-signer)"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-service-network-signer)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-apiserver-service-network-signer)"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Generating Certificate (kube-control-plane-ca-bundle)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-to-kubelet-ca-bundle)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-to-kubelet-signer)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Generating Certificate (kube-apiserver-to-kubelet-signer)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Generating Certificate (kube-apiserver-to-kubelet-ca-bundle)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kubelet-bootstrap-kubeconfig-signer)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Certificate (kubelet-bootstrap-kubeconfig-signer)"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Generating Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Generating Certificate (kube-apiserver-complete-client-ca-bundle)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-to-kubelet-ca-bundle)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-apiserver-to-kubelet-ca-bundle)"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-to-kubelet-client)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-to-kubelet-signer)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-apiserver-to-kubelet-signer)"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Generating Certificate (kube-apiserver-to-kubelet-client)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-to-kubelet-signer)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-apiserver-to-kubelet-signer)"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kube-control-plane-ca-bundle)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-control-plane-ca-bundle)"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kube-control-plane-kube-controller-manager-client)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kube-control-plane-signer)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-control-plane-signer)"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Generating Certificate (kube-control-plane-kube-controller-manager-client)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kube-control-plane-kube-scheduler-client)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kube-control-plane-signer)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-control-plane-signer)"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Generating Certificate (kube-control-plane-kube-scheduler-client)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kube-control-plane-signer)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-control-plane-signer)"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Certificate (kubelet-bootstrap-kubeconfig-ca-bundle)"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kubelet-client-ca-bundle)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Certificate (kubelet-client-ca-bundle)"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kubelet-client)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Certificate (kubelet-client)"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kubelet-signer)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Certificate (kubelet-signer)"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kubelet-serving-ca-bundle)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (kubelet-signer)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Certificate (kubelet-signer)"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Generating Certificate (kubelet-serving-ca-bundle)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Certificate (mcs)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Certificate (mcs)"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Root CA..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Root CA"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Key Pair (service-account.pub)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Generating Key Pair (service-account.pub)..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Release Image Pull Spec..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Generating Release Image Pull Spec..."
time="2020-01-28T23:29:56-06:00" level=debug msg="Using internal constant for release image registry.svc.ci.openshift.org/origin/release:4.3"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Image..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Image"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Generating Bootstrap Ignition Config..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Master Ignition Config..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Master Ignition Config"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Master Machines..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Master Machines"
time="2020-01-28T23:29:56-06:00" level=debug msg=" Fetching Worker Machines..."
time="2020-01-28T23:29:56-06:00" level=debug msg=" Reusing previously-fetched Worker Machines"
time="2020-01-28T23:29:56-06:00" level=debug msg="Generating Terraform Variables..."
time="2020-01-28T23:29:57-06:00" level=info msg="Obtaining RHCOS image file from 'https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.4/44.81.202001241431.0/x86_64/rhcos-44.81.202001241431.0-qemu.x86_64.qcow2.gz?sha256=56154a5c68e94879ff276e9ad6a7efc080cb0cb99b034f44e1bcd41a371f6878'"
time="2020-01-28T23:29:57-06:00" level=info msg="The file was found in cache: /root/.cache/openshift-installer/image_cache/893b690279d58ad4a014862439fd1b95. Reusing..."
time="2020-01-28T23:29:57-06:00" level=info msg="Consuming Worker Machines from target directory"
time="2020-01-28T23:29:57-06:00" level=debug msg="Purging asset \"Worker Machines\" from disk"
time="2020-01-28T23:29:57-06:00" level=info msg="Consuming Openshift Manifests from target directory"
time="2020-01-28T23:29:57-06:00" level=debug msg="Purging asset \"Openshift Manifests\" from disk"
time="2020-01-28T23:29:57-06:00" level=info msg="Consuming Master Machines from target directory"
time="2020-01-28T23:29:57-06:00" level=debug msg="Purging asset \"Master Machines\" from disk"
time="2020-01-28T23:29:57-06:00" level=info msg="Consuming Common Manifests from target directory"
time="2020-01-28T23:29:57-06:00" level=debug msg="Purging asset \"Common Manifests\" from disk"
time="2020-01-28T23:29:57-06:00" level=debug msg="Fetching Kubeconfig Admin Client..."
time="2020-01-28T23:29:57-06:00" level=debug msg="Loading Kubeconfig Admin Client..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Loading Certificate (admin-kubeconfig-client)..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Loading Certificate (kube-apiserver-complete-server-ca-bundle)..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Fetching Certificate (admin-kubeconfig-client)..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Reusing previously-fetched Certificate (admin-kubeconfig-client)"
time="2020-01-28T23:29:57-06:00" level=debug msg=" Fetching Certificate (kube-apiserver-complete-server-ca-bundle)..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Reusing previously-fetched Certificate (kube-apiserver-complete-server-ca-bundle)"
time="2020-01-28T23:29:57-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:57-06:00" level=debug msg="Generating Kubeconfig Admin Client..."
time="2020-01-28T23:29:57-06:00" level=debug msg="Fetching Kubeadmin Password..."
time="2020-01-28T23:29:57-06:00" level=debug msg="Reusing previously-fetched Kubeadmin Password"
time="2020-01-28T23:29:57-06:00" level=debug msg="Fetching Certificate (journal-gatewayd)..."
time="2020-01-28T23:29:57-06:00" level=debug msg="Reusing previously-fetched Certificate (journal-gatewayd)"
time="2020-01-28T23:29:57-06:00" level=debug msg="Fetching Metadata..."
time="2020-01-28T23:29:57-06:00" level=debug msg="Loading Metadata..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Loading Cluster ID..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Fetching Cluster ID..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Reusing previously-fetched Cluster ID"
time="2020-01-28T23:29:57-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:57-06:00" level=debug msg="Generating Metadata..."
time="2020-01-28T23:29:57-06:00" level=debug msg="Fetching Cluster..."
time="2020-01-28T23:29:57-06:00" level=debug msg="Loading Cluster..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Loading Cluster ID..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Loading Install Config..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Loading Platform Credentials Check..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Loading Terraform Variables..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Loading Kubeadmin Password..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Fetching Cluster ID..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Reusing previously-fetched Cluster ID"
time="2020-01-28T23:29:57-06:00" level=debug msg=" Fetching Install Config..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Reusing previously-fetched Install Config"
time="2020-01-28T23:29:57-06:00" level=debug msg=" Fetching Platform Credentials Check..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Reusing previously-fetched Platform Credentials Check"
time="2020-01-28T23:29:57-06:00" level=debug msg=" Fetching Terraform Variables..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Reusing previously-fetched Terraform Variables"
time="2020-01-28T23:29:57-06:00" level=debug msg=" Fetching Kubeadmin Password..."
time="2020-01-28T23:29:57-06:00" level=debug msg=" Reusing previously-fetched Kubeadmin Password"
time="2020-01-28T23:29:57-06:00" level=debug msg="Generating Cluster..."
time="2020-01-28T23:29:57-06:00" level=info msg="Creating infrastructure resources..."
time="2020-01-28T23:29:57-06:00" level=debug msg="Symlinking plugin terraform-provider-openstack src: \"/root/go/src/github.com/openshift/installer/bin/openshift-install\" dst: \"/tmp/openshift-install-647652407/plugins/terraform-provider-openstack\""
time="2020-01-28T23:29:57-06:00" level=debug msg="Symlinking plugin terraform-provider-ovirt src: \"/root/go/src/github.com/openshift/installer/bin/openshift-install\" dst: \"/tmp/openshift-install-647652407/plugins/terraform-provider-ovirt\""
time="2020-01-28T23:29:57-06:00" level=debug msg="Symlinking plugin terraform-provider-libvirt src: \"/root/go/src/github.com/openshift/installer/bin/openshift-install\" dst: \"/tmp/openshift-install-647652407/plugins/terraform-provider-libvirt\""
time="2020-01-28T23:29:57-06:00" level=debug msg="Symlinking plugin terraform-provider-local src: \"/root/go/src/github.com/openshift/installer/bin/openshift-install\" dst: \"/tmp/openshift-install-647652407/plugins/terraform-provider-local\""
time="2020-01-28T23:29:57-06:00" level=debug msg="Symlinking plugin terraform-provider-random src: \"/root/go/src/github.com/openshift/installer/bin/openshift-install\" dst: \"/tmp/openshift-install-647652407/plugins/terraform-provider-random\""
time="2020-01-28T23:29:57-06:00" level=debug msg="Symlinking plugin terraform-provider-aws src: \"/root/go/src/github.com/openshift/installer/bin/openshift-install\" dst: \"/tmp/openshift-install-647652407/plugins/terraform-provider-aws\""
time="2020-01-28T23:29:57-06:00" level=debug msg="Symlinking plugin terraform-provider-azurerm src: \"/root/go/src/github.com/openshift/installer/bin/openshift-install\" dst: \"/tmp/openshift-install-647652407/plugins/terraform-provider-azurerm\""
time="2020-01-28T23:29:57-06:00" level=debug msg="Symlinking plugin terraform-provider-azureprivatedns src: \"/root/go/src/github.com/openshift/installer/bin/openshift-install\" dst: \"/tmp/openshift-install-647652407/plugins/terraform-provider-azureprivatedns\""
time="2020-01-28T23:29:57-06:00" level=debug msg="Symlinking plugin terraform-provider-google src: \"/root/go/src/github.com/openshift/installer/bin/openshift-install\" dst: \"/tmp/openshift-install-647652407/plugins/terraform-provider-google\""
time="2020-01-28T23:29:57-06:00" level=debug msg="Symlinking plugin terraform-provider-ignition src: \"/root/go/src/github.com/openshift/installer/bin/openshift-install\" dst: \"/tmp/openshift-install-647652407/plugins/terraform-provider-ignition\""
time="2020-01-28T23:29:57-06:00" level=debug msg="Symlinking plugin terraform-provider-vsphere src: \"/root/go/src/github.com/openshift/installer/bin/openshift-install\" dst: \"/tmp/openshift-install-647652407/plugins/terraform-provider-vsphere\""
time="2020-01-28T23:29:57-06:00" level=debug msg="Initializing modules..."
time="2020-01-28T23:29:57-06:00" level=debug msg="- bootstrap in ../../../../../tmp/openshift-install-647652407/bootstrap"
time="2020-01-28T23:29:57-06:00" level=debug msg="- volume in ../../../../../tmp/openshift-install-647652407/volume"
time="2020-01-28T23:29:57-06:00" level=debug
time="2020-01-28T23:29:57-06:00" level=debug msg="Initializing the backend..."
time="2020-01-28T23:29:57-06:00" level=debug
time="2020-01-28T23:29:57-06:00" level=debug msg="Initializing provider plugins..."
time="2020-01-28T23:29:57-06:00" level=debug
time="2020-01-28T23:29:57-06:00" level=debug msg="Terraform has been successfully initialized!"
time="2020-01-28T23:29:57-06:00" level=debug
time="2020-01-28T23:29:57-06:00" level=debug msg="You may now begin working with Terraform. Try running \"terraform plan\" to see"
time="2020-01-28T23:29:57-06:00" level=debug msg="any changes that are required for your infrastructure. All Terraform commands"
time="2020-01-28T23:29:57-06:00" level=debug msg="should now work."
time="2020-01-28T23:29:57-06:00" level=debug
time="2020-01-28T23:29:57-06:00" level=debug msg="If you ever set or change modules or backend configuration for Terraform,"
time="2020-01-28T23:29:57-06:00" level=debug msg="rerun this command to reinitialize your working directory. If you forget, other"
time="2020-01-28T23:29:57-06:00" level=debug msg="commands will detect it and remind you to do so if necessary."
time="2020-01-28T23:29:59-06:00" level=debug msg="data.libvirt_network_dns_host_template.masters_int[0]: Refreshing state..."
time="2020-01-28T23:29:59-06:00" level=debug msg="data.libvirt_network_dns_host_template.etcds[0]: Refreshing state..."
time="2020-01-28T23:29:59-06:00" level=debug msg="data.libvirt_network_dns_host_template.masters[0]: Refreshing state..."
time="2020-01-28T23:29:59-06:00" level=debug msg="data.libvirt_network_dns_host_template.bootstrap[0]: Refreshing state..."
time="2020-01-28T23:29:59-06:00" level=debug msg="data.libvirt_network_dns_srv_template.etcd_cluster[0]: Refreshing state..."
time="2020-01-28T23:29:59-06:00" level=debug msg="data.libvirt_network_dns_host_template.bootstrap_int[0]: Refreshing state..."
time="2020-01-28T23:29:59-06:00" level=debug msg="libvirt_pool.storage_pool: Creating..."
time="2020-01-28T23:29:59-06:00" level=debug msg="libvirt_network.net: Creating..."
time="2020-01-28T23:30:04-06:00" level=debug msg="libvirt_pool.storage_pool: Creation complete after 5s [id=7d66f383-20cc-4ef3-b25a-30a0f6fff4df]"
time="2020-01-28T23:30:04-06:00" level=debug msg="libvirt_ignition.master: Creating..."
time="2020-01-28T23:30:04-06:00" level=debug msg="module.volume.libvirt_volume.coreos_base: Creating..."
time="2020-01-28T23:30:04-06:00" level=debug msg="module.bootstrap.libvirt_ignition.bootstrap: Creating..."
time="2020-01-28T23:30:09-06:00" level=debug msg="libvirt_network.net: Still creating... [10s elapsed]"
time="2020-01-28T23:30:14-06:00" level=debug msg="libvirt_ignition.master: Still creating... [10s elapsed]"
time="2020-01-28T23:30:14-06:00" level=debug msg="module.volume.libvirt_volume.coreos_base: Still creating... [10s elapsed]"
time="2020-01-28T23:30:14-06:00" level=debug msg="module.bootstrap.libvirt_ignition.bootstrap: Still creating... [10s elapsed]"
time="2020-01-28T23:30:19-06:00" level=debug msg="libvirt_network.net: Still creating... [20s elapsed]"
time="2020-01-28T23:30:24-06:00" level=debug msg="libvirt_ignition.master: Still creating... [20s elapsed]"
time="2020-01-28T23:30:24-06:00" level=debug msg="module.volume.libvirt_volume.coreos_base: Still creating... [20s elapsed]"
time="2020-01-28T23:30:24-06:00" level=debug msg="module.bootstrap.libvirt_ignition.bootstrap: Still creating... [20s elapsed]"
time="2020-01-28T23:30:29-06:00" level=debug msg="libvirt_network.net: Still creating... [30s elapsed]"
time="2020-01-28T23:30:34-06:00" level=debug msg="libvirt_ignition.master: Still creating... [30s elapsed]"
time="2020-01-28T23:30:34-06:00" level=debug msg="module.volume.libvirt_volume.coreos_base: Still creating... [30s elapsed]"
time="2020-01-28T23:30:34-06:00" level=debug msg="module.bootstrap.libvirt_ignition.bootstrap: Still creating... [30s elapsed]"
time="2020-01-28T23:30:39-06:00" level=debug msg="libvirt_network.net: Still creating... [40s elapsed]"
time="2020-01-28T23:30:44-06:00" level=debug msg="libvirt_ignition.master: Still creating... [40s elapsed]"
time="2020-01-28T23:30:44-06:00" level=debug msg="module.volume.libvirt_volume.coreos_base: Still creating... [40s elapsed]"
time="2020-01-28T23:30:44-06:00" level=debug msg="module.bootstrap.libvirt_ignition.bootstrap: Still creating... [40s elapsed]"
time="2020-01-28T23:30:49-06:00" level=debug msg="libvirt_network.net: Still creating... [50s elapsed]"
time="2020-01-28T23:30:54-06:00" level=debug msg="libvirt_ignition.master: Still creating... [50s elapsed]"
time="2020-01-28T23:30:54-06:00" level=debug msg="module.volume.libvirt_volume.coreos_base: Still creating... [50s elapsed]"
time="2020-01-28T23:30:54-06:00" level=debug msg="module.bootstrap.libvirt_ignition.bootstrap: Still creating... [50s elapsed]"
time="2020-01-28T23:30:59-06:00" level=debug msg="libvirt_network.net: Still creating... [1m0s elapsed]"
time="2020-01-28T23:31:04-06:00" level=debug msg="libvirt_ignition.master: Still creating... [1m0s elapsed]"
time="2020-01-28T23:31:04-06:00" level=debug msg="module.volume.libvirt_volume.coreos_base: Still creating... [1m0s elapsed]"
time="2020-01-28T23:31:04-06:00" level=debug msg="module.bootstrap.libvirt_ignition.bootstrap: Still creating... [1m0s elapsed]"
time="2020-01-28T23:31:09-06:00" level=debug msg="libvirt_network.net: Still creating... [1m10s elapsed]"
time="2020-01-28T23:31:14-06:00" level=debug msg="libvirt_ignition.master: Still creating... [1m10s elapsed]"
time="2020-01-28T23:31:14-06:00" level=debug msg="module.volume.libvirt_volume.coreos_base: Still creating... [1m10s elapsed]"
time="2020-01-28T23:31:14-06:00" level=debug msg="module.bootstrap.libvirt_ignition.bootstrap: Still creating... [1m10s elapsed]"
time="2020-01-28T23:31:19-06:00" level=debug msg="libvirt_network.net: Still creating... [1m20s elapsed]"
time="2020-01-28T23:31:24-06:00" level=debug msg="libvirt_ignition.master: Still creating... [1m20s elapsed]"
time="2020-01-28T23:31:24-06:00" level=debug msg="module.volume.libvirt_volume.coreos_base: Still creating... [1m20s elapsed]"
time="2020-01-28T23:31:24-06:00" level=debug msg="module.bootstrap.libvirt_ignition.bootstrap: Still creating... [1m20s elapsed]"
time="2020-01-28T23:31:29-06:00" level=debug msg="libvirt_network.net: Still creating... [1m30s elapsed]"
time="2020-01-28T23:31:34-06:00" level=debug msg="libvirt_ignition.master: Still creating... [1m30s elapsed]"
time="2020-01-28T23:31:34-06:00" level=debug msg="module.volume.libvirt_volume.coreos_base: Still creating... [1m30s elapsed]"
time="2020-01-28T23:31:34-06:00" level=debug msg="module.bootstrap.libvirt_ignition.bootstrap: Still creating... [1m30s elapsed]"
time="2020-01-28T23:31:39-06:00" level=debug msg="libvirt_network.net: Still creating... [1m40s elapsed]"
time="2020-01-28T23:31:44-06:00" level=debug msg="libvirt_ignition.master: Still creating... [1m40s elapsed]"
time="2020-01-28T23:31:44-06:00" level=debug msg="module.volume.libvirt_volume.coreos_base: Still creating... [1m40s elapsed]"
time="2020-01-28T23:31:44-06:00" level=debug msg="module.bootstrap.libvirt_ignition.bootstrap: Still creating... [1m40s elapsed]"
time="2020-01-28T23:31:49-06:00" level=debug msg="libvirt_network.net: Still creating... [1m50s elapsed]"
time="2020-01-28T23:31:54-06:00" level=debug msg="libvirt_ignition.master: Still creating... [1m50s elapsed]"
time="2020-01-28T23:31:54-06:00" level=debug msg="module.volume.libvirt_volume.coreos_base: Still creating... [1m50s elapsed]"
time="2020-01-28T23:31:54-06:00" level=debug msg="module.bootstrap.libvirt_ignition.bootstrap: Still creating... [1m50s elapsed]"
time="2020-01-28T23:31:59-06:00" level=debug msg="libvirt_network.net: Still creating... [2m0s elapsed]"
time="2020-01-28T23:31:59-06:00" level=debug msg="libvirt_ignition.master: Creation complete after 1m55s [id=/var/lib/libvirt/openshift-images/mycluster-rtt9n/mycluster-rtt9n-master.ign;5e3118cf-a7fc-af05-fd43-dc6da674055e]"
time="2020-01-28T23:31:59-06:00" level=debug msg="libvirt_network.net: Creation complete after 2m0s [id=a6dd99a4-fef5-415d-a88c-339e34f8dd44]"
time="2020-01-28T23:32:01-06:00" level=debug msg="module.volume.libvirt_volume.coreos_base: Creation complete after 1m57s [id=/var/lib/libvirt/openshift-images/mycluster-rtt9n/mycluster-rtt9n-base]"
time="2020-01-28T23:32:01-06:00" level=debug msg="libvirt_volume.master[0]: Creating..."
time="2020-01-28T23:32:01-06:00" level=debug msg="module.bootstrap.libvirt_volume.bootstrap: Creating..."
time="2020-01-28T23:32:01-06:00" level=debug msg="module.bootstrap.libvirt_ignition.bootstrap: Creation complete after 1m57s [id=/var/lib/libvirt/openshift-images/mycluster-rtt9n/mycluster-rtt9n-bootstrap.ign;5e3118d1-265f-6b37-6738-5461a74c32fb]"
time="2020-01-28T23:32:02-06:00" level=debug msg="module.bootstrap.libvirt_volume.bootstrap: Creation complete after 0s [id=/var/lib/libvirt/openshift-images/mycluster-rtt9n/mycluster-rtt9n-bootstrap]"
time="2020-01-28T23:32:02-06:00" level=debug msg="module.bootstrap.libvirt_domain.bootstrap: Creating..."
time="2020-01-28T23:32:02-06:00" level=debug msg="libvirt_volume.master[0]: Creation complete after 0s [id=/var/lib/libvirt/openshift-images/mycluster-rtt9n/mycluster-rtt9n-master-0]"
time="2020-01-28T23:32:02-06:00" level=debug msg="libvirt_domain.master[0]: Creating..."
time="2020-01-28T23:32:02-06:00" level=debug msg="module.bootstrap.libvirt_domain.bootstrap: Creation complete after 1s [id=9c7cac0b-b397-419f-8bc2-7800e2814b62]"
time="2020-01-28T23:32:02-06:00" level=debug msg="libvirt_domain.master[0]: Creation complete after 1s [id=58d0fe33-07b3-4405-9ecd-e6e78623ac7c]"
time="2020-01-28T23:32:03-06:00" level=debug
time="2020-01-28T23:32:03-06:00" level=debug msg="Apply complete! Resources: 9 added, 0 changed, 0 destroyed."
time="2020-01-28T23:32:03-06:00" level=debug msg="OpenShift Installer unreleased-master-2447-g1cc6935753842458bb84504f8e247ddc86cb93c7-dirty"
time="2020-01-28T23:32:03-06:00" level=debug msg="Built from commit 1cc6935753842458bb84504f8e247ddc86cb93c7"
time="2020-01-28T23:32:03-06:00" level=info msg="Waiting up to 20m0s for the Kubernetes API at https://api.mycluster.5u04.info:6443..."
time="2020-01-28T23:32:06-06:00" level=debug msg="Still waiting for the Kubernetes API: Get https://api.mycluster.5u04.info:6443/version?timeout=32s: dial tcp 192.168.126.10:6443: connect: no route to host"
time="2020-01-28T23:32:14-06:00" level=debug msg="Still waiting for the Kubernetes API: Get https://api.mycluster.5u04.info:6443/version?timeout=32s: dial tcp 192.168.126.10:6443: connect: connection refused"
time="2020-01-28T23:32:44-06:00" level=debug msg="Still waiting for the Kubernetes API: Get https://api.mycluster.5u04.info:6443/version?timeout=32s: dial tcp 192.168.126.10:6443: connect: connection refused"
time="2020-01-28T23:33:14-06:00" level=debug msg="Still waiting for the Kubernetes API: Get https://api.mycluster.5u04.info:6443/version?timeout=32s: dial tcp 192.168.126.10:6443: connect: connection refused"
time="2020-01-28T23:33:47-06:00" level=debug msg="Still waiting for the Kubernetes API: Get https://api.mycluster.5u04.info:6443/version?timeout=32s: dial tcp 192.168.126.10:6443: connect: connection refused"
time="2020-01-28T23:34:11-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:34:41-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:35:11-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:35:41-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:36:11-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:36:41-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:37:11-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:37:42-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:38:12-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:38:42-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:39:12-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:39:42-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:40:12-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:40:42-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:41:12-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:41:42-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:42:12-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:42:42-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:43:12-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:43:42-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:44:12-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:44:42-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:45:12-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:45:42-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:46:12-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:46:42-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:47:12-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:47:42-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:48:12-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:48:42-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:49:12-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:49:42-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:50:12-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:50:42-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:51:12-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:51:42-06:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
time="2020-01-28T23:52:03-06:00" level=error msg="Attempted to gather ClusterOperator status after installation failure: listing ClusterOperator objects: the server could not find the requested resource (get clusteroperators.config.openshift.io)"
time="2020-01-28T23:52:03-06:00" level=debug msg="Fetching Install Config..."
time="2020-01-28T23:52:03-06:00" level=debug msg="Loading Install Config..."
time="2020-01-28T23:52:03-06:00" level=debug msg=" Loading SSH Key..."
time="2020-01-28T23:52:03-06:00" level=debug msg=" Using SSH Key loaded from state file"
time="2020-01-28T23:52:03-06:00" level=debug msg=" Loading Base Domain..."
time="2020-01-28T23:52:03-06:00" level=debug msg=" Loading Platform..."
time="2020-01-28T23:52:03-06:00" level=debug msg=" Using Platform loaded from state file"
time="2020-01-28T23:52:03-06:00" level=debug msg=" Using Base Domain loaded from state file"
time="2020-01-28T23:52:03-06:00" level=debug msg=" Loading Cluster Name..."
time="2020-01-28T23:52:03-06:00" level=debug msg=" Loading Base Domain..."
time="2020-01-28T23:52:03-06:00" level=debug msg=" Loading Platform..."
time="2020-01-28T23:52:03-06:00" level=debug msg=" Using Cluster Name loaded from state file"
time="2020-01-28T23:52:03-06:00" level=debug msg=" Loading Pull Secret..."
time="2020-01-28T23:52:03-06:00" level=debug msg=" Using Pull Secret loaded from state file"
time="2020-01-28T23:52:03-06:00" level=debug msg=" Loading Platform..."
time="2020-01-28T23:52:03-06:00" level=debug msg="Using Install Config loaded from state file"
time="2020-01-28T23:52:03-06:00" level=debug msg="Reusing previously-fetched Install Config"
time="2020-01-28T23:52:03-06:00" level=info msg="Pulling debug logs from the bootstrap machine"
time="2020-01-28T23:52:03-06:00" level=debug msg="Using SSH_AUTH_SOCK /tmp/ssh-g1Rg45kgR62d/agent.677 to connect to an existing agent"
time="2020-01-28T23:56:18-06:00" level=error msg="Attempted to gather debug logs after installation failure: failed to create SSH client, ensure the proper ssh key is in your keyring or specify with --key: dial tcp 165.160.15.20:22: connect: connection timed out"
time="2020-01-28T23:56:18-06:00" level=fatal msg="Bootstrap failed to complete: waiting for Kubernetes API: context deadline exceeded"

doyoungim999 picture doyoungim999 on 29 Jan 2020

/assign @zeenix @praveenkumar

It would be nice if you guys can take a look at this. Thanks!

abhinavdahiya picture abhinavdahiya on 31 Jan 2020

@doyoungim999 how did you build the installer, using master? Also have you also used image override to have a different release payload?

praveenkumar picture praveenkumar on 11 Feb 2020

Same issue here.
Stuck with Installer for libvirt after machines for bootstrap and master have been startet.
openshift_install-fail-200210.log

Installer build from head yesterday. No overrides,

thdevelthreesixtiy picture thdevelthreesixtiy on 11 Feb 2020

It seems the problem is in data/data/libvirt/main.tf

The DNS Names for API and Master are the Same there:
`data "libvirt_network_dns_host_template" "bootstrap" {
count = var.bootstrap_dns ? 1 : 0
ip = var.libvirt_bootstrap_ip
hostname = "api.${var.cluster_domain}"
}

data "libvirt_network_dns_host_template" "masters" {
count = var.master_count
ip = var.libvirt_master_ips[count.index]
hostname = "api.${var.cluster_domain}"
}

data "libvirt_network_dns_host_template" "bootstrap_int" {
count = var.bootstrap_dns ? 1 : 0
ip = var.libvirt_bootstrap_ip
hostname = "api-int.${var.cluster_domain}"
}

data "libvirt_network_dns_host_template" "masters_int" {
count = var.master_count
ip = var.libvirt_master_ips[count.index]
hostname = "api-int.${var.cluster_domain}"
}`

So the master VM is not able to get the ignition config from the bootstrap vm.
When directly accessing https://api-int.test1.tt.testing:22623/config/master from host with correct DNS by browser the config is available.

I will try to poke the data/data/libvirt/main.tf...

thdevelthreesixtiy picture thdevelthreesixtiy on 11 Feb 2020

It seems the problem is in data/data/libvirt/main.tf

The DNS Names for API and Master are the Same there:

So the master VM is not able to get the ignition config from the bootstrap vm.
When directly accessing https://api-int.test1.tt.testing:22623/config/master from host with correct DNS by browser the config is available.

I am seeing the same bug with a fresh libvirt setup.

Any pointers how to edit the main.tf file to make the cluster accessible? What DNS settings are you using to make it accessible by browser?

avidwriter picture avidwriter on 12 Feb 2020

I conform the finding for head as well - but it behaves different. For me in the head build the bootstrap does not initilize at all.

For 4.3 branch I tried changing "api" to "mst" in lines 126 and 138 for hostname - thats the DNS names for the master node:

data "libvirt_network_dns_host_template" "masters" {
count = var.master_count
ip = var.libvirt_master_ips[count.index]
hostname = "mst.${var.cluster_domain}"
}

data "libvirt_network_dns_host_template" "masters_int" {
count = var.master_count
ip = var.libvirt_master_ips[count.index]
hostname = "mst-int.${var.cluster_domain}"
}

DNS for bootstrap from other hosts in virtual network is correct afterwards.
So pinging https://api-int.test1.tt.testing and getting the master.json from https://api-int.test1.tt.testing:22623/config/master from a vm inside the virtual network is fine. I checked using an additional vm for manual testing.

But the ignition process @ master.node still fails for some reason to access data from bootstrap.

At the initital boot it resolves the IP of the bottstap correctly to .10 but fails to get the content:
iginition[855]: GET error: Get https://api-int.test1.tt.testing:22623/config/master: dial tcp: 192.168.126.10:22623: connect: connection refused

At any later boot - triggered automaticaly oder manually - the message on master console looks different:
iginition[854]: GET error: Get https://api-int.test1.tt.testing:22623/config/master: dial tcp: lookup api-int.test1.tt.testing:22623 on [::1]:53: read udp [::1]:38345->[::1]:53: read: connection refused

For access from vm-host to vm-bootstrap node i made manual entries in /etc/hosts for bootstrap and master node:
192.168.126.10 api.test1.tt.testing api-int.test1.tt.testing
192.168.126.11 etcd-0.test1.tt.testing

openshift_install.log

So i am stuck here again.

Finally i wonder why there is no "master" subproject under data/libvirt as for "aws" or "baremetal".
I expected it is obligatory to have this for getting the master node configured by the installer ?

thdevelthreesixtiy picture thdevelthreesixtiy on 13 Feb 2020

@doyoungim999 @avidwriter @thdevelthreesixtiy Thanks for your reports. Many times I've come across the same issues/symptoms and after spending days, realize that it was caused by me missing or skipping a step or two while following the instructions to setup the env. Could you please ensure that is not the case for you?

zeenix picture zeenix on 14 Feb 2020

Thx for the link. I already did.
Originally on Ubuntu i even installed me a fresh Fedora 31 to get all the instructions from there in place.
There all the dnmasq and stuff works like described there.
But it makes no real difference in the end besides that resolving hostnames works without changing /etc/hosts.

The master still is not able to get its ignition config from bootstrap:22623.

The boostrap machine does not bring up a working api on :6443. The Installer is not able to get objects from kubernetes:
time="2020-02-15T10:23:45+01:00" level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"

Trying to get pod list or config from kubernetes@bootstrap fails with error similar to that of installer:
~/go/src/github.com/openshift/installer$ export KUBECONFIG="cluster-0/auth/kubeconfig"
~/go/src/github.com/openshift/installer$ kubectl get --all-namespaces pods
error: the server doesn't have a resource type "pods"
~/go/src/github.com/openshift/installer$ kubectl cluster-info dump
Error from server (NotFound): the server could not find the requested resource (get nodes)

Trying to access boostrap via ssh from host fails:
~/go/src/github.com/openshift/installer$ ssh "[email protected]"
[email protected]: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
Tried to use key from cluster-0/tls/journal-gatewayd.key for auth did not work either.

thdevelthreesixtiy picture thdevelthreesixtiy on 15 Feb 2020

I fetched the latest master yesterday and having the same issue. Are there any known workarounds?

pkliczewski picture pkliczewski on 25 Feb 2020

Retest with current head.
Established ssh connection to bootsrap successfully.

journalctl -b -f -u release-image.service -u bootkube.service
...
Feb 25 11:19:00 test1-pkx7z-bootstrap release-image-download.sh[1711]: Pull failed. Retrying registry.svc.ci.openshift.org/origin/release:4.5...
Feb 25 11:19:02 test1-pkx7z-bootstrap release-image-download.sh[1711]: Error: error pulling image "registry.svc.ci.openshift.org/origin/release:4.5": unable to pull registry.svc.ci.openshift.org/origin/release:4.5: unable to pull image: Error initializing source docker://registry.svc.ci.openshift.org/origin/release:4.5: Error reading manifest 4.5 in registry.svc.ci.openshift.org/origin/release: manifest unknown: manifest unknown
...

This looks like missing dependency on openshift repo.

log-bundle-20200225122811.tar.gz

thdevelthreesixtiy picture thdevelthreesixtiy on 25 Feb 2020

Retest with current 4.3 with different error:

With ssh on on bootstrap:

journalctl -b -f -u bootkube.service
...
Feb 25 15:29:21 test1-czx6d-bootstrap bootkube.sh[1974]: {"level":"warn","ts":"2020-02-25T15:29:21.376Z","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-36e171fd-5bd6-4e50-a096-d7d1b2310195/etcd-0.test1.tt.testing:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 192.168.126.11:2379: connect: connection refused\""}
Feb 25 15:29:21 test1-czx6d-bootstrap bootkube.sh[1974]: https://etcd-0.test1.tt.testing:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Feb 25 15:29:21 test1-czx6d-bootstrap bootkube.sh[1974]: Error: unhealthy cluster
...

This seems to be the issue where the master does not get his ign.config from bootstrap to bring up the etc-demon to be configured from bootstrap...

.openshift_install.log
log-bundle-20200225151135.tar.gz

thdevelthreesixtiy picture thdevelthreesixtiy on 25 Feb 2020

Recommend checking your etcd SRV record. I had the same experience in a bare-metal type of env, and thought it was an issue with 4.3, but turned out to be a small typo in our SRV record.

oybed picture oybed on 27 Feb 2020

Thx for the idea. I checked it. Name resolution seems to work ok.

I managed to get a Text Console at the master and reviewed the syslog with journalctl:

I found out that the master.ign for is actually fetched successfully from bootstrap:
Mar 02 08:07:50 test1-f79hn-master-0.test1.tt.testing ignition[868]: fetched referenced config at https://api-int.test1.tt.testing:22623/config/master with SHA512: ...

But it has several errors:
[...]
t1.tt.testing ignition[868]: error at line 1, column 966
no filesystem specified
t1.tt.testing ignition[868]: error at line 1, column 1366
no filesystem specified
t1.tt.testing ignition[868]: error at line 1, column 1764
no filesystem specified
[...]

This seems to be an old error which has been marked as fixed before here:
https://github.com/openshift/os/issues/394

There it is suggested to use "filesystem":"root" style for ignition as in bootstrap.ign.
I will try this out..

thdevelthreesixtiy picture thdevelthreesixtiy on 2 Mar 2020

@thdevelthreesixtiy any luck getting the cluster up and functional? hoping this issue can be resolved soon as we're dead in the water with no way to develop currently.

avidwriter picture avidwriter on 4 Mar 2020

@avidwriter: No - i am still stuck.

The issue is topic of several other tickets. Most of them marked closed with regard to using image override feature for OCP. For OKD @abhinavdahiya suggests using an "fcos branch" here: https://github.com/openshift/installer/issues/3030
I dont know of any "fcos branch".

In https://github.com/openshift/installer/issues/3043 it seems that for OCP deplyoment is working with image override.

Other tickets with same issue:
https://github.com/openshift/installer/issues/2856
https://github.com/openshift/os/issues/394

All of them indicate an problem with machine-config-operator during generation of ignition fpr master. Tweaking the template manually during installation is no solution. The rendering process itself seems to be the problem.

thdevelthreesixtiy picture thdevelthreesixtiy on 9 Mar 2020
👍1

@thdevelthreesixtiy

thanks for the info on the error. I tried the 'fcos' suggestion-- saw there's a branch in Git for that:

git clone https://github.com/openshift/installer.git -b 'fcos'

it seemed to get further and there is a bootstrap and master machine running in virsh and I can curl the API endpoint at port 6443 at the hostname (after adding it to /etc/hosts) but still am not able to connect fully.

hope that helps

avidwriter picture avidwriter on 10 Mar 2020
👍1

I finally got a running mini-cluster deployed on libvirt.

The Workaround with using OCP images was the key:

  1. Put every suggestion from https://github.com/openshift/installer/blob/master/docs/dev/libvirt/README.md in place
  2. Patch Installer source for RAM on bootstrap to 16GB (installer/pkg/asset/machines/libvirt/machines.go)
  3. Build the installer for libvirt and 4.3
  4. Set export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=quay.io/openshift-release-dev/ocp-release:4.3.1-x86_64 before running the installer.

A proper pull secret from RedHat has to be entered for usage of OCP images for install-config.

As a result i have a master and a worker running.

thdevelthreesixtiy picture thdevelthreesixtiy on 12 Mar 2020
👍3

as @thdevelthreesixtiy mentioned above, when waiting fro Kube API times out, from what I've seen, it's because there is an issue pulling the release image, and you can see this when you ssh into the bootstrap machine and look at the journal logs - the fix is to use the
OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=<a payload for which you have access>

sallyom picture sallyom on 23 Mar 2020

I'm hitting this also. I'd love to get some insight into figuring out how to SSH into the VMs created and see if I can get more information on what's failing.

cwawak picture cwawak on 24 Mar 2020

with libvirt - I usually do this
$ sudo virsh list
$ sudo domifaddr -the-bootstrap-node
$ ssh -i /path/to/private-ssh-key core@ip-addr-from-above
$ run the cmd that is shown upon ssh'ing into bootstrap-it's journalctl
(I don't have a cluster atm, so those are approximate cmds)

On Tue, Mar 24, 2020 at 11:14 AM Christopher Wawak notifications@github.com
wrote:

I'm hitting this also. I'd love to get some insight into figuring out how
to SSH into the VMs created and see if I can get more information on what's
failing.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/openshift/installer/issues/2927#issuecomment-603297873,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACVGC4N7E3WZ22P5L53V47TRJDE5XANCNFSM4KG67RIQ
.

--

Sally O'Malley

Software Engineer | OpenShift
https://www.redhat.com/

sallyom picture sallyom on 24 Mar 2020

Thanks! The piece I was missing was adding an sshKey entry to an installer-config.yaml so I could SSH in in the first place. :)

cwawak picture cwawak on 24 Mar 2020

@sallyom Hi Sally, I hit the same issue :)
Error:error pulling image "quay.io/openshift-release-dev/ocp-release:4.3.1-x86_6"

Silly question, how do I know which payload I have access to?
I saw your comment:
OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=<a payload for which you have access>

I did set the env:
OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=quay.io/openshift-release-dev/ocp-release:4.3.1-x86_6

seems I dont have access to this image..

I manually execute this script on my bootstrap VM:
[root@test1-qbzll-bootstrap ~]# export RELEASE_IMAGE=quay.io/openshift-release-dev/ocp-release:4.3.1-x86_6

[root@test1-qbzll-bootstrap ~]# podman pull --quiet "$RELEASE_IMAGE"

Error: error pulling image "quay.io/openshift-release-dev/ocp-release:4.3.1-x86_6": unable to pull quay.io/openshift-release-dev/ocp-release:4.3.1-x86_6: unable to pull image: Error initializing source docker://quay.io/openshift-release-dev/ocp-release:4.3.1-x86_6: error pinging docker registry quay.io: Get https://quay.io/v2/: dial tcp: lookup quay.io on 192.168.126.1:53: read udp 192.168.126.10:38579->192.168.126.1:53: i/o timeout

Wonder something different is happening here.
This port 192.168.126.10:38579 is actually not opened up:
[root@test1-qbzll-bootstrap ~]# netstat -tnlpn

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/systemd

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1675/sshd

tcp 0 0 0.0.0.0:37957 0.0.0.0:* LISTEN 1628/rpc.statd

tcp6 0 0 :::19531 :::* LISTEN 1/systemd

tcp6 0 0 :::111 :::* LISTEN 1/systemd

tcp6 0 0 :::22 :::* LISTEN 1675/sshd

tcp6 0 0 :::44833 :::* LISTEN 1628/rpc.statd

cakebrick picture cakebrick on 25 Mar 2020

Hi, I'm not sure what you're trying to do, exactly, but if you go to
https://www.openshift.com/try you can register and get access to images.
You can download an installer, from, say, 'Get Started With
OpenShift->Create your own cluster-> Try it in the cloud
->aws->User-provisioned infrastructure'
and from there, once you download an installer (can use the aws or gcp link
there, whatever - the installer is the same regardless)
$ ./openshift-install version will give you the payload image (for me,
the latest from try.openshift.com gives me
quay.io/openshift-release-dev/ocp-release@sha256:a414f6308db72f88e9d2e95018f0cc4db71c6b12b2ec0f44587488f0a16efc42
)
Once you're registered with try.openshift.com, that image should be
accessible. : )

On Wed, Mar 25, 2020 at 9:43 AM Xin notifications@github.com wrote:

@sallyom https://github.com/sallyom Hi Sally, I hit the same issue :)
Error:error pulling image "
quay.io/openshift-release-dev/ocp-release:4.3.1-x86_6"

Silly question, how do I know which payload I have access to?
I saw your comment:
OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE= access>

I did set the env:
OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=
quay.io/openshift-release-dev/ocp-release:4.3.1-x86_6

seems I dont have access to this image..

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openshift/installer/issues/2927#issuecomment-603846537,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACVGC4IUNMTZAW436RULAU3RJIC6BANCNFSM4KG67RIQ
.

--

Sally O'Malley

Software Engineer | OpenShift
https://www.redhat.com/

sallyom picture sallyom on 25 Mar 2020

/cc

zhijianli88 picture zhijianli88 on 17 Apr 2020

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot picture openshift-bot on 16 Jul 2020

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

openshift-bot picture openshift-bot on 15 Aug 2020

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

openshift-bot picture openshift-bot on 14 Sep 2020

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot picture openshift-ci-robot on 14 Sep 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

lowerorbit picture lowerorbit  Â·  4Comments
alexouzounis picture alexouzounis  Â·  12Comments
raesene picture raesene  Â·  12Comments
glennodickson picture glennodickson  Â·  5Comments
davivcgarcia picture davivcgarcia  Â·  5Comments