Kops: The connection to the server [redacted] was refused - did you specify the right host or port?

[ramonkeypr]

I have been following the kops guide to the letter. I setup my Route 53 zone, etc. I can ssh into my newly created hosts, and the necessary k8s daemons/etc. appear to be running.

Here is what I am experiencing:

$ kubectl get nodes
The connection to the server [redacted] was refused - did you specify the right host or port?

Here is the command I used to create my cluster:

$ kops create cluster --cloud=aws --zones=us-east-1c,us-east-1a,us-east-1b ${NAME} --master-zones=us-east-1a,us-east-1b,us-east-1c --target=terraform --node-count=4 --node-size=m3.medium --master-size=m3.medium --dns-zone=dev.keypr.io --state=${KOPS_STATE_STORE}

Then, I built the infra.:

${GOPATH}/bin/kops update cluster ${NAME} --yes

I have waited quite a while, and still can't get my nodes. I do not have kubectl proxy running.

Please help!

[ramonkeypr]

When logging onto one my masters, I run the following:

$ ps aux | grep kubernetes
root       759  1.0  1.7 388564 68428 ?        Ssl  02:03   0:13 /usr/local/bin/kubelet --allow-privileged=true --api-servers=http://127.0.0.1:8080 --babysit-daemons=true --cgroup-root=docker --cloud-provider=aws --cluster-dns=100.64.0.10 --cluster-domain=cluster.local --config=/etc/kubernetes/manifests --configure-cbr0=true --enable-debugging-handlers=true --non-masquerade-cidr=100.64.0.0/10 --pod-cidr=10.123.45.0/30 --reconcile-cidr=true --register-schedulable=false --v=2
root       762  0.1  0.0  21776  3344 ?        Ss   02:03   0:01 /bin/bash /etc/kubernetes/kube-addons.sh
root      2793  0.0  0.0   1184     4 ?        Ss   02:04   0:00 /bin/sh -c /usr/local/bin/kube-controller-manager --allocate-node-cidrs=true --cloud-provider=aws --cluster-cidr=100.96.0.0/11 --cluster-name=k8s1.dev.keypr.io --configure-cloud-routes=true --leader-elect=true --master=127.0.0.1:8080 --root-ca-file=/srv/kubernetes/ca.crt --service-account-private-key-file=/srv/kubernetes/server.key --v=2 1>>/var/log/kube-controller-manager.log 2>&1
root      2807  0.0  0.8  66404 34172 ?        Sl   02:04   0:00 /usr/local/bin/kube-controller-manager --allocate-node-cidrs=true --cloud-provider=aws --cluster-cidr=100.96.0.0/11 --cluster-name=k8s1.dev.keypr.io --configure-cloud-routes=true --leader-elect=true --master=127.0.0.1:8080 --root-ca-file=/srv/kubernetes/ca.crt --service-account-private-key-file=/srv/kubernetes/server.key --v=2
admin    21513  0.0  0.0  12732  2156 pts/0    S+   02:26   0:00 grep kubernetes

I expect the API to listening on http://127.0.0.1:8080

$ curl http://127.0.0.1:8080
curl: (7) Failed to connect to 127.0.0.1 port 8080: Connection refused

$ sudo curl http://127.0.0.1:8080
curl: (7) Failed to connect to 127.0.0.1 port 8080: Connection refused

Could this be part of the problem?

[ramonkeypr]

I can confirm that this master is not running the API:

$ sudo netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:38818           0.0.0.0:*               LISTEN      357/rpc.statd   
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      759/kubelet     
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      2407/kube-proxy 
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      348/rpcbind     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      539/sshd        
tcp6       0      0 :::4001                 :::*                    LISTEN      22341/etcd      
tcp6       0      0 :::4002                 :::*                    LISTEN      21906/etcd      
tcp6       0      0 :::4194                 :::*                    LISTEN      759/kubelet     
tcp6       0      0 :::10245                :::*                    LISTEN      2005/aws-controller
tcp6       0      0 :::10250                :::*                    LISTEN      759/kubelet     
tcp6       0      0 :::10251                :::*                    LISTEN      2906/kube-scheduler
tcp6       0      0 :::48363                :::*                    LISTEN      357/rpc.statd   
tcp6       0      0 :::2380                 :::*                    LISTEN      22341/etcd      
tcp6       0      0 :::10252                :::*                    LISTEN      2807/kube-controlle
tcp6       0      0 :::2381                 :::*                    LISTEN      21906/etcd      
tcp6       0      0 :::10255                :::*                    LISTEN      759/kubelet     
tcp6       0      0 :::111                  :::*                    LISTEN      348/rpcbind     
tcp6       0      0 :::22                   :::*                    LISTEN      539/sshd 

[ramonkeypr]

@justinsb what else can I check?

[justinsb]

Sorry for the trouble. I suspect one of two things:

1) Using a version of terraform older than 0.7 but not using the workaround. So your volumes aren't labelled correctly. Easy way to check is to look at your AWS volumes; they should have a tag like `k8s.io/role/master' = '1'

2) The DNS records are being created, but not in the correct hosted zone. Can you verify that DNS names are present, and that you are able to resolve them (e.g. that ping api.${name} works if you see that record)

[ramonkeypr]

Thank you so much for the reply. I will check both points. Assuming that
all routing and networking is correct, does kops have any problems using
private zones instead of public zones on Route 53?

Thanks.

Regards,
Ramon

On Aug 31, 2016 8:49 PM, "Justin Santa Barbara" [email protected]
wrote:

Sorry for the trouble. I suspect one of two things:

1) Using a version of terraform older than 0.7 but not using the
workaround. So your volumes aren't labelled correctly. Easy way to check is
to look at your AWS volumes; they should have a tag like `
k8s.io/role/master' = '1'

2) The DNS records are being created, but not in the correct hosted zone.
Can you verify that DNS names are present, and that you are able to resolve
them (e.g. that ping api.${name} works if you see that record)

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kops/issues/391#issuecomment-243968726,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AT8TkyIFQtJ6-yejP0lrJh6dDqg8SD8Pks5qlktSgaJpZM4JxKXZ
.

[justinsb]

It does work, although it is harder because you then have to link the private zone to a VPC, which means you have to create the VPC first, which means you then have to specify the VPC when creating your cluster like this: https://github.com/kubernetes/kops/blob/master/docs/run_in_existing_vpc.md

I want to make this easier - I opened #396 for that.

[ramonkeypr]

Thanks.

On Aug 31, 2016 9:26 PM, "Justin Santa Barbara" [email protected]
wrote:

It does work, although it is harder because you then have to link the
private zone to a VPC, which means you have to create the VPC first, which
means you then have to specify the VPC when creating your cluster like
this: https://github.com/kubernetes/kops/blob/master/docs/run_in_
existing_vpc.md

I want to make this easier - I opened #396
https://github.com/kubernetes/kops/issues/396 for that.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kops/issues/391#issuecomment-243972765,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AT8Tk3JQg2r45ZyICiZkpJRhRQ7D-P7-ks5qllPlgaJpZM4JxKXZ
.

[ramonkeypr]

@justinsb when deploying to an existing VPC, it seems kops wants to rename the existing VPC to match the name of the k8s cluster. Am I interpreting this correctly, and if so, why does kops need to rename the existing VPC?

[ramonkeypr]

@justinsb I was able to resolve my own issue by fixing my DNS configuration. Thanks for all of your help.

[zkmoney]

@ramonkeypr I am experiencing the exact same thing you were after following the install instructions to the letter, I can ssh onto the master, see that there are no processes listening to either 8080 or 443, and I get the same The connection to the server api.<name> was refused - did you specify the right host or port? when running kubectl cluster-info. How did you end up fixing this?

@justinsb, to your point about being able to ping api.<name>, I just get request timeouts there. I do have a route 53 A record set for api.<name> that is pointing to the public ip of the master. Is this a common issue? Any insight would be huge, thanks!

[ramonkeypr]

@zkmoney I will try to look through my notes for some documentation, but essentially, I let Terraform own the entire zone I wanted to use. Also, I had to update the information at my registrar to use the name servers used by the newly created zone (which was a subdomain of an existing zone).

[justinsb]

@zkmoney is your master node an instance with a single vCPU (i.e. a m3.medium or similar?). I think there's a bug I introduced, which I am about to fix. A simple workaround you can do (more to verify that this is in fact the issue) is to edit /etc/kubernetes/manifests/kube-apiserver and change the cpu request from 250m to 50m. Then sudo systemctl restart kubelet and the system should come up. If that's the case, this will be fixed in an hour or two. If not please find me on slack and we can figure it out.

[zkmoney]

@justinsb It is a single vCPU yes. I will pull the latest and rerun, and if that doesn't work I will try your other fix. Thanks for the offer regarding contacting you on Slack!

@ramonkeypr Good to know as well. I'll keep that in mind as I am troubleshooting.

[vkumar2808]

Hello,
Changing to cpu from 250m to 50m didnt fix the problem. do we have any other solution?

[pranayhere]

I'm facing the same issue,

kubectl cluster-info
Kubernetes master is running at http://localhost:8080

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
The connection to the server localhost:8080 was refused - did you specify the right host or port?

how do i fix this, i don't want to delete cluster and rebuild it as it is running production workload.
Please suggest

[k]

I'm experiencing this issue on a previous cluster after attempting to open a feature gates and enable an admission controller. I am running kops 1.10.0 k8s 1.10.6 on AWS

[nicholaison]

I'm facing the same issue,

kubectl cluster-info
Kubernetes master is running at http://localhost:8080

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
The connection to the server localhost:8080 was refused - did you specify the right host or port?

how do i fix this, i don't want to delete cluster and rebuild it as it is running production workload.
Please suggest

Make sure your config file has been setup correctly - Alternatively, use --kubeconfig to specify the config file