Kops: kops installation on aws ::::: kubectl command not working after installing and creating cluster

[mahavirteraiya]

root@ip-172-31-46-246:/usr/local/bin# kubectl get nodes
Unable to connect to the server: dial tcp: lookup api.uswest2.kuber.shi.com on 172.31.0.2:53: no such host

I have followed 100% correct documentation to install kops.
https://kubernetes.io/docs/getting-started-guides/kops/

I have used this command for ROute53
aws route53 create-hosted-zone --name kuber.shi.com --caller-reference 3

S3
aws s3 mb s3://clusters.kuber.shi.com

Cluster get created I can also see created cluster in AWS dashboard

When I try below command :+1:

root@ip-172-31-46-246:/usr/local/bin# kubectl get nods
Unable to connect to the server: dial tcp: lookup api.uswest2.kuber.shi.com on 172.31.0.2:53: no such host

When i connect to Master through SSH -i
and do following command I get
admin@ip-172-20-41-222:/usr/local/bin$ kubectl get no
The connection to the server localhost:8080 was refused - did you specify the right host or port?

[justinsb]

We have some additional docs in review that may be clearer.

Can you check in particular this one: https://github.com/kubernetes/kops/pull/1538/files#diff-5803986eae4dd3a69a265d54997b0040

It sounds as if your NS records are not set up.

Another possibility: if you are using a private hostedzone, you may need to attach that HostedZone to your VPC. We should do this automatically for you in kops 1.5.0-alpha4 if you specify --dns private though.

Which version of kops is this (kops version)0?

[mahavirteraiya]

Hi justinsb ,

Thanks for reply...

I dont have real domain , can u please suggest me what to do in that case.

I have v1.4.0 kops version.

If need i can try 1.5-alpha also.

Please suggest.

[chrislovecnm]

You need a domain. We have had some thoughts about using instance names and such, but currently you need a real domain.

[mahavirteraiya]

So is there any other way to create kubernetes clusterin AWS ?
Please suggest.

[chrislovecnm]

There are. Can you swing by sig-aws?

[natemurthy]

I'm experiencing a similar issue. I'm using a Route 53 private hosted zone associated with a VPC where both DNS resolution and hostname are enabled. The first thing that I noticed was odd was that after running kops create cluster successfully, the A name record for the master k8s node was incorrect so I had to manually change the IP address to the correct one that appeared in the EC2 dashboard for that node. That got my DNS resolving correctly after the newly updated A record propagated.

The next issue I found was that I could not run any kubectl commands from my client to the k8s master node, more specifically the node addressed by api.k8s.isyslabs.net. I kept getting connection refused messages like this:

$ kubectl get nodes
The connection to the server api.k8s.isyslabs.net was refused - did you specify the right host or port?

I knew I had the right host because I could telnet and ssh to the server on port 22. So maybe I had the port incorrect. It seemed that by default kubectl wanted to connect on port 443. I checked my Security Groups and all the inbound entries were correct. However, two observations:

$ telnet api.k8s.isyslabs.net 22
Trying VV.XX.YY.ZZ...
Connected to api.k8s.isyslabs.net.
Escape character is '^]'.
SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3

$ telnet api.k8s.isyslabs.net 443
Trying VV.XX.YY.ZZ...



md5-55719b936880a6b620cc311c94441800



$ netstat -tln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:39856           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp6       0      0 :::4194                 :::*                    LISTEN     
tcp6       0      0 :::10250                :::*                    LISTEN     
tcp6       0      0 :::10251                :::*                    LISTEN     
tcp6       0      0 :::10252                :::*                    LISTEN     
tcp6       0      0 :::45005                :::*                    LISTEN     
tcp6       0      0 :::10255                :::*                    LISTEN     
tcp6       0      0 :::111                  :::*                    LISTEN  



md5-07183338078e8e6d305ec4d91ba6d455



$ ps ax | grep kub
 1222 ?        Ssl    0:00 /usr/bin/docker run -v /:/rootfs/ -v /var/run/dbus:/var/run/dbus -v /run/systemd:/run/systemd --net=host --privileged protokube:1.5.3 /usr/bin/protokube --channels=s3://natemurthy-artifacts/kops-state/k8s.isyslabs.net/addons/bootstrap-channel.yaml --cloud=aws --containerized=true --dns-internal-suffix=.internal.k8s.isyslabs.net --dns=aws-route53 --master=true --v=4 --zone=k8s.isyslabs.net
 1246 ?        Ssl    0:00 /usr/bin/protokube --channels=s3://natemurthy-artifacts/kops-state/k8s.isyslabs.net/addons/bootstrap-channel.yaml --cloud=aws --containerized=true --dns-internal-suffix=.internal.k8s.isyslabs.net --dns=aws-route53 --master=true --v=4 --zone=k8s.isyslabs.net
 1293 ?        Ssl    0:20 /usr/local/bin/kubelet --allow-privileged=true --api-servers=http://127.0.0.1:8080 --babysit-daemons=true --cgroup-root=docker --cloud-provider=aws --cluster-dns=100.64.0.10 --cluster-domain=cluster.local --enable-debugging-handlers=true --eviction-hard=memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5% --hostname-override=ip-172-20-53-185.us-west-2.compute.internal --network-plugin=kubenet --node-labels=kubernetes.io/role=master --non-masquerade-cidr=100.64.0.0/10 --pod-manifest-path=/etc/kubernetes/manifests --register-schedulable=false --v=2
 1568 ?        Ss     0:00 /bin/sh -c echo -998 > /proc/$$/oom_score_adj && kube-proxy --kubeconfig=/var/lib/kube-proxy/kubeconfig --resource-container="" --master=http://127.0.0.1:8080 --v=2 1>>/var/log/kube-proxy.log 2>&1
 1578 ?        Sl     0:01 kube-proxy --kubeconfig=/var/lib/kube-proxy/kubeconfig --resource-container= --master=http://127.0.0.1:8080 --v=2
 1621 ?        Ss     0:00 /bin/sh -c /usr/local/bin/kube-scheduler --leader-elect=true --master=127.0.0.1:8080 --v=2 1>>/var/log/kube-scheduler.log 2>&1
 1633 ?        Sl     0:04 /usr/local/bin/kube-scheduler --leader-elect=true --master=127.0.0.1:8080 --v=2
 1809 ?        Ss     0:00 /bin/sh -c /usr/local/bin/kube-controller-manager --allocate-node-cidrs=true --attach-detach-reconcile-sync-period=1m0s --cloud-provider=aws --cluster-cidr=100.96.0.0/11 --cluster-name=k8s.isyslabs.net --configure-cloud-routes=true --leader-elect=true --master=127.0.0.1:8080 --root-ca-file=/srv/kubernetes/ca.crt --service-account-private-key-file=/srv/kubernetes/server.key --v=2 1>>/var/log/kube-controller-manager.log 2>&1
 1819 ?        Sl     0:00 /usr/local/bin/kube-controller-manager --allocate-node-cidrs=true --attach-detach-reconcile-sync-period=1m0s --cloud-provider=aws --cluster-cidr=100.96.0.0/11 --cluster-name=k8s.isyslabs.net --configure-cloud-routes=true --leader-elect=true --master=127.0.0.1:8080 --root-ca-file=/srv/kubernetes/ca.crt --service-account-private-key-file=/srv/kubernetes/server.key --v=2



md5-b8b703dab48134f93b4fbbdc4da63127



$ telnet localhost 8080
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

and no surprises.

So what's the deal?

[richburdon]

I'm having the same issues. I've double checked the R53 recordset matches the entries I added to my domain.

The nodes show up in EC2 (per my config), but there's no master.

[richburdon]

In my case, It seems that although "us-east-1a" shows up for aws ec2 describe-availability-zones --region us-east-1, it doesn't actually work. "us-east-1d" did. I don't understand why.

[jorge07]

Same issue

kops 1.6.0-beta1
k8s 1.6.2 & 1.5.4

Protokube logs loop in:

I0509 09:47:35.558077       1 channels.go:50] apply channel output was: Error: error querying kubernetes version: Get https://127.0.0.1/version: dial tcp 127.0.0.1:443: getsockopt: connection refused
Usage:
  channels apply channel [flags]

Flags:
  -f, --filename stringSlice   Apply from a local file
      --yes                    Apply update

Global Flags:
      --alsologtostderr                  log to standard error as well as files
      --config string                    config file (default is $HOME/.channels.yaml)
      --log_backtrace_at traceLocation   when logging hits line file:N, emit a stack trace (default :0)
      --log_dir string                   If non-empty, write log files in this directory
      --logtostderr                      log to standard error instead of files (default false)
      --stderrthreshold severity         logs at or above this threshold go to stderr (default 2)
  -v, --v Level                          log level for V logs (default 0)
      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging


error querying kubernetes version: Get https://127.0.0.1/version: dial tcp 127.0.0.1:443: getsockopt: connection refused
W0509 09:47:35.558136       1 kube_boot.go:145] error applying channel "s3://kubernetes-staging-kubernetes-state-store/kubernetes.staging.porntube.com/addons/bootstrap-channel.yaml": error running channels: exit status 1

[cirulls]

I had exactly the same issue @richburdon reported in his comment: starting the cluster in us-east-1a doesn't seem to work (I get an error message saying cannot get nodes when issuing kops validate cluster) but starting the cluster in us-east-1d with the same configuration works like a charm. Both availability zones show up when issuing aws ec2 describe-availability-zones --region us-east-1.

[chrislovecnm]

Do you have m3 instance types in us-east-1a? Change the master instance type to a m4 in us-east-1a

[cirulls]

Yes, I had m3 instances types in us-east-1a for master and changing it to m4 works, thanks a lot.

[chrislovecnm]

Closing ;)

[deadpoet6]

@natemurthy have you resolved your issue if so I would like to know how you did that. I am still facing the same issue. I am performing kops validate cluster but ending up with connection refused error. It is the same with "kubectl get nodes".
The connection to the server api.xxxxx.xxxx.xxx was refused - did you specify the right host or port?
@chrislovecnm any suggestions please?

[natemurthy]

I've upgraded to kops 1.6.1, and I'm still experiencing this issue.

@chrislovecnm Can we please re-open?

[natemurthy]

This time I used m4.large instances, and got a bit further. I noticed that kops will create 4 additional A records in my hosted zone for the names:

api.sub.domain.com
api.internal.sub.domain.com
etcd-b.internal.sub.domain.com
etcd-events-b.internal.sub.domain

However, the IP addresses assigned to these records all appear to be bogus. There are no VMs that have these IP addresses. I did however change the IP address of api.sub.domain.com to the IP address of the master. This got me as far as being able view just the master node, but none of the worker nodes show up (even though the EC2 dashboard shows they are running):

$ kubectl get nodes
NAME                                          STATUS         AGE       VERSION
ip-xxx-xx-xx-xxx.us-west-2.compute.internal   Ready,master   9m        v1.6.2

So I've made a bit of progress, but it still appears that I have more manual work (not exactly sure what though) to do to get this fully working. What should the IP address of the other names be set to? I've tried setting them to both the private and public IP addresses, neither of those values worked -- still couldn't view any of the kubelet nodes.

[natemurthy]

Cross-linking with previously reported DNS-related issue https://github.com/kubernetes/kops/issues/1386

[deadpoet6]

I have found a solution at least for my issue. The actual problem is, I was using a private image from our corporate due to the way it was developed it has some issues when installed and running kops ( I was unable to ssh and ping the private IP at least of the master node running in aws). Later I have changed the image to a public one (ubuntu) and began the entire process again debugging the issue.
`kops create cluster

--cloud=aws
--zones=eu-central-1a
--node-size=t2.small
--master-size=t2.small
--node-count=2
--name=gave a name here
--dns-zone=my DNS name
--dns private
--vpc=vpc-xxxxxx
--network-cidr=172.xx.xx.xx/xx
--state=s3://waterinthebucket
--image ami-xxxxx
--ssh-public-key=~/keys/id_rsa.pub`
I have launched the cluster with above configuration but I still had the issue. When I looked into my aws account I found that the above script is creating a new route table for an existing VPC. I have an existing route table previously for that VPC with defined rules but the new one flushed all of them and have only two rules in that. So I've manually changed my route table configurations and also changed the PlaceholdereIP=203.0.113.123 of all 4 sub domains (api,api,etcd,etcd) under hosted zones to my private Ip address of the master node and waited for 5 mins for DNS to reconfigure. Now everything works perfectly. @natemurthy thanks for bringing #1386 it has set a correct path.
edit: here they have mentioned about the creation of shred subnets https://github.com/kubernetes/kops/blob/master/docs/run_in_existing_vpc.md#running-in-a-shared-subnet

[natemurthy]

Aha! I needed to set the --dns=private flag and that got things working with my Route53 setup.