Installer: Waiting 30m0s for the kubernetes api

Created on 13 May 2019 · 19Comments · Source: openshift/installer

Version

$ openshift-install version ./openshift-install v4.1.0-201905091432-dirty
built from commit 969b7ef6a763d1a4af658f3f19eb1c2a85d573f0
release image quay.io/openshift-release-dev/ocp-release@sha256:713aae8687cf8a3cb5c2c504f65532dfe11e1b3534448ea9eeef5b0931d3e208
<your output here>

Platform (aws|libvirt|openstack): AWS

Enter text here.

What happened?

Time out at waiting for the kubernetes api to enable

What you expected to happen?

Sucessfull installation of the OCP4 console.

How to reproduce it (as minimally and precisely as possible)?

$ ./openshift-install create cluster --dir=/downloads/openshift4

Anything else we need to know?

Unable to log in to get logs as it doesn't give me the password needed and I don't have a SSH key for the server.

References

Source

matty1979

Most helpful comment

After deleting the whole cluster and installing it again two times it worked, no idea what the problem was.

SonOfTheChief on 14 May 2019

🎉1 👍1

All 19 comments

We do not setup passwords. All debugging efforts require that you provide installer an ssh key when prompted.

openshift-install gather bootstap should provide details on how to gather logs.

abhinavdahiya on 13 May 2019

interesting, I tried running it on a local laptop and it asked for an ssh key this time. It didn't do that when I ran the installer on a cloud server ssh window.
That said, the
INFO Creating infrastructure resources...
INFO Waiting up to 30m0s for the Kubernetes API
The .openshift_install.logs says "no such host"

matty1979 on 13 May 2019

time="2019-05-13T17:34:56-04:00" level=debug msg="Still waiting for the Kubernetes API: Get https://api.os4.xxx.xxx.chat:6443/version?timeout=32s: dial tcp x.x.x.x:6443: connect: connection refused"

matty1979 on 13 May 2019

I have the same problem. The installer waited 30mins for the Kubernetes API and then failed with:
FATAL waiting for Kubernetes API: context deadline exceeded
Right before that, these INFO messages were shown:

INFO Make sure ssh-agent is running, env SSH_AUTH_SOCK is set to the ssh-agent's UNIX socket and your private key is added to the agent.
INFO Use the following commands to gather logs from the cluster
INFO ssh -A [email protected] '/usr/local/bin/installer-gather.sh XX.XX.XX.XX XX.XX.XX.XX XX.XX.XX.XX'
INFO scp [email protected]:~/log-bundle.tar.gz .

And yes, I selected my ssh key when terraform asked for it. And I can gather the logs, so the connection via ssh works.

SonOfTheChief on 14 May 2019

After deleting the whole cluster and installing it again two times it worked, no idea what the problem was.

SonOfTheChief on 14 May 2019

🎉1 👍1

I've destroyed and reinstalled numerous times, I still receive the same errors. I can SSH to the machines so there's that, but nothing else is working within the Kubernetes API.

matty1979 on 14 May 2019

Can you provide us the log tar file that installer recommends, so we can help debug.

Also see https://github.com/openshift/installer/blob/master/docs/user/troubleshooting.md#kubernetes-api-is-unavailable

abhinavdahiya on 14 May 2019

oc --config=${INSTALL_DIR}/auth/kubeconfig --namespace=openshift-machine-api get deployments
error: stat /auth/kubeconfig: no such file or directory

matty1979 on 15 May 2019

Matthew-King-MacBook-Pro:openshift4 matthewking$ oc status The connection to the server api.os.dev.openshift.octo.chat:6443 was refused - did you specify the right host or port?

matty1979 on 15 May 2019

$ oc --config=${INSTALL_DIR}/auth/kubeconfig --namespace=openshift-machine-api get deployments
error: stat /auth/kubeconfig: no such file or directory

Looks like your INSTALL_DIR was empty. You need to fill in whatever directory you used for assets when you called openshift-install create ... (/downloads/openshift4 in your original post here). But oc commands in general are not going to work without a Kubernetes API. You need to gather bootstrap logs over ssh as folks keep suggesting above.

wking on 15 May 2019

log-bundle.tar.gz

matty1979 on 15 May 2019

log-bundle.tar.gz

based on the file attached, it looks like the bootstrapping did finish...
Can you try again and provide, both INSTALL_DIR/.openshift_install.log and log-bundle.tar.gz from the run if it fails again.

abhinavdahiya on 20 May 2019

log-bundle.tar.gz
openshift_install.log

matty1979 on 21 May 2019

I also encountered the same problem as you, have you solved it?

yangjinsay on 4 Jun 2019

No it’s still an issue for me. I’m going to try again today to see if I can get it to install.

matty1979 on 4 Jun 2019

👀1

Using the newest version, I noticed something with the elb creation phase as well as an error while tail -f the .openshift_install.log

time="2019-06-04T17:12:14-04:00" level=debug msg="Still waiting for the Kubernetes API: Get https://api.os4.openshiftdev.octo.chat:6443/version?timeout=32s: dial tcp: lookup api.os4.openshiftdev.octo.chat on 75.75.75.75:53: no such host"

Even with changing off of Comcast DNS to google, I receive the same error.

During the ELB creation phase, without fail one instance will not become healthy in one AZ. So I'm not quite sure what the issue is but, us-east-2b and us-east-2c are fine, us-east-2a is unhealthy.

I did this earlier today and 2b was unhealthy.

matty1979 on 4 Jun 2019

I tired this again today in us-west-2 and it worked correctly. No idea why this won’t work in us-east-2 though.

matty1979 on 8 Jun 2019

Please re-open if issue re-appears.

/close

abhinavdahiya on 20 Jun 2019

@abhinavdahiya: Closing this issue.

In response to this:

Please re-open if issue re-appears.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.