Kind: Can't connect to API server when using docker-in-docker

[fiunchinho]

What happened:
I'm trying to use kind to run some tests during CI in our CI server. The CI server runs jobs inside Docker containers. I'm creating the kind cluster from my job's container, and that works just fine. But when I try to connect to the API server using kubectl, I get "connection refused".
To make it clear, the scenario is a host running the Docker daemon, where I start a container for my CI job using docker-in-docker. From that container I create the kind container.

What you expected to happen:
Being able to connect to the API server and issue kubectl commands.

How to reproduce it (as minimally and precisely as possible):
Create "docker-in-docker" Docker container and try to create a kind cluster and execute kubectl commands.
Anything else we need to know?:

Environment:

• kind version: (use kind version): 0.3.0-alpha
• Kubernetes version: (use kubectl version): 1.13.4
• Docker version: (use docker info): 18.09.6
• OS (e.g. from /etc/os-release):

[aojea]

/assign

[aojea]

It seems that you need a way to expose the port of the kind cluster in your host, since this is inside docker I guess that we need to do some kind of tunnel or port mapping over port mapping, so you can connect directly to the API port.

You can tell kind in which port you want the API server to listen using the API, this way you can do a port forwarding of that port in the outer docker

kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
networking:
  apiServerPort: 19090
  apiServerAddress: 0.0.0.0
nodes:
# the control plane node
- role: control-plane

with that config you can reach your API on port 19090, so you only need to forward that port to your host and point your kubectl config to it

[BenTheElder]

So replace "docker in docker" with "docker in a VM" for the mental model for a moment:

Since docker is on another host, and we're binding / port forwarding the apiServer on localhost by default, this means that you cannot talk to it.

This can be fixed by configuring the networking.apiServerAddress config field as @aojea mentioned above. (though probably it should be the dind container IP, rather than 0.0.0.0, so we correctly set it in the kuebeconfig... :thinking: )

docker-in-docker is just like this case, the containers are in a different network namespace and the apiServer is only available on that loopback (which is NOT the hosts' loopback).

The fix for this should be configuring that field.

/remove-kind bug
/kind documentation
/triage support

[aojea]

/unassign @aojea

[fiunchinho]

I'm trying to pass the config file. The docs say To specify a configuration file when creating a cluster, use the --config flag, but I'm getting an error

kind create --config kind-config.yaml
Error: unknown flag: --config

Ok, I submitted a PR to clarify how to pass a custom config file.

[fiunchinho]

I can't make it work.

I used this config file

kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
networking:
  apiServerPort: 19090
  apiServerAddress: 127.0.0.1

So the generated kubeconfig contains this

server: https://127.0.0.1:19090

But I'm still getting The connection to the server 127.0.0.1:19090 was refused - did you specify the right host or port?. This is the output for docker ps

7e0f9c6c6d19        kindest/node:v1.14.1                                              "/usr/local/bin/entr…"   About a minute ago   Up 56 seconds       19090/tcp, 127.0.0.1:19090->6443/tcp   kind-control-plane

My DIND container is running on networking host mode, its IP address is the same as the host IP address.
Any ideas?

[aojea]

Where are you running kubectl ?
with that config you have to run inside the host with the dind container.

If you want to use kubectl outside of the host I guess you have to use apiServerAddress: 0.0.0.0 and use the ip address of your host in your kubeconfig
server: https://IP_HOST:19090

[fiunchinho]

I'm running kubectl inside the DIND container.

[aojea]

Then check if you have the socket open

 ss -apn | grep 19090
tcp    LISTEN     0      128    127.0.0.1:19090              0.0.0.0:*

if it's open you have to check if something is blocking it, maybe an iptables rule.

[fiunchinho]

Not sure if that's the same as the following

$ netstat -lntp

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.11:42514        0.0.0.0:*               LISTEN      -

Does this mean that the socket is not open?

[aojea]

yeah, can you paste your docker info on the host?
That doesn't seem a _normal_ host 🤔

[fiunchinho]

Containers: 3
Running: 3
Paused: 0
Stopped: 0
Images: 64
Server Version: 18.03.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 18ebad5d2d2b7486c6dbb66fbb8b4e3e063b53c3 (expected: 4fc53a81fb7c994640722ac585fa9ca548971871)
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-1013-aws
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.675GiB
Name: ip-10-0-11-251
ID: O2PQ:USX3:5P6Y:X6CB:MGHP:J2MB:SL3T:PAWJ:5L6A:KPUW:B4WY:PBN7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

[BenTheElder]

I'm running kubectl inside the DIND container.

how exactly? if this is done via another container inside the dind container note that any container will have it's own loopback (localhost) interface. it has to be actually in the dind container specifically to get the same interface.

[BenTheElder]

My DIND container is running on networking host mode, its IP address is the same as the host IP address.

could you try not doing host network and forwarding a port instead?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[angelotinho]

I hit the same issue. I started a Pod containing a docker in docker container on an eks cluster.
Within that container I ran kind create cluster. The cluster was created. I set the KUBECONFIG. Ran kubectl get nodes....Got a 500 internal server response from the server.

[k8s-ci-robot]

@angelotinho: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

I hit the same issue. I started a Pod containing a docker in docker container on an eks cluster.
Within that container I ran kind create cluster. The cluster was created. I set the KUBECONFIG. Ran kubectl get nodes....Got a 500 internal server response from the server.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[BenTheElder]

please don't reopen this.
you observed the same symptom / a similar issue, but with different details, notably you're either using a much newer version of kind, or we're not doing support for it.

If you're trying to run kind in a kubernetes cluster, I'm going to suggest you don't and then point you to https://github.com/kubernetes-sigs/kind/issues/303 with discussion of the various issues you need to work around in that type of environment.

[dgajendran]

I have the same exact problem. I am using docker compose with two services: a dind and a kind. I run docker daemon in dind and expose that in kind service to create a cluster. Cluster created successfully but it seems kubedns is not running. I get the same The connection to the server docker:19090 was refused - did you specify the right host or port?

[BenTheElder]

"docker" is not a name kind would use natively. that sounds like a problem
with how you're managing dind.

On Sat, Sep 12, 2020 at 12:18 AM dgajendran [e-mail]
wrote:

I have the same exact problem. I am using docker compose with two
services: a dind and a kind. I run docker daemon in dind and expose that in
kind service to create a cluster. Cluster created successfully but it seems
kubedns is not running. I get the same The connection to the server
docker:19090 was refused - did you specify the right host or port?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-sigs/kind/issues/523#issuecomment-691432657,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHADK7JBWGWXULZCRNK45TSFMODZANCNFSM4HMO46HQ
.