Kind: Cannot create cluster by kind 0.3.0 with docker 18.09.6

[wanghh2000]

What happened:
Create cluster failed by kind 0.3.0 on Ubuntu.

What you expected to happen:
Create cluster success.

How to reproduce it (as minimally and precisely as possible):
1) Install docker 18.09.6 and kind 0.3.0 on Ubuntu.
2) Run the command.

# ./kind-linux-amd64 create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.14.2) 🖼
ERRO[22:52:13] 0cd93e6e3b3a28c4216a3fa7b0d75337e83ca32f5e4095629c75a472b2ee89a6
ERRO[22:52:13] docker: Error response from daemon: driver failed programming external connectivity on endpoint kind-control-plane (1229f3b0af4456532d4a8cf9ae274c0c03441da448de535ee94a1a6e25148d05):  (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 127.0.0.1 --dport 46796 -j DNAT --to-destination 172.17.0.2:6443 ! -i docker0: iptables: No chain/target/match by that name.
ERRO[22:52:13]  (exit status 1)).
 ✗ Preparing nodes 📦
ERRO[22:52:13] docker run error: exit status 125
Error: failed to create cluster: docker run error: exit status 125

Anything else we need to know?:

Environment:

# docker version
Client:
 Version:           18.09.6
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        481bc77
 Built:             Sat May  4 02:35:27 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.6
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       481bc77
  Built:            Sat May  4 01:59:36 2019
  OS/Arch:          linux/amd64
  Experimental:     false

# cat /etc/issue
Ubuntu 16.04.6 LTS \n \l

# uname -p
x86_64

• kind version: (use kind version): 0.3.0
• Kubernetes version: (use kubectl version):
• Docker version: (use docker info): docker-ce 18.09.6
• OS (e.g. from /etc/os-release): Ubuntu 16.04.6 LTS

[tao12345666333]

this log looks like the docker's error. please use --loglevel=debug to get more detail.

[tao12345666333]

and can you show me your dockerd's configuration? this error is mainly in the Docker network, i.e. userland-proxy

[aojea]

also check if the iptables table for docker exists with iptables -L -t nat, you should have something like this

linux-6my5:~ # iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination

@wanghh2000 I have kind working with docker version 18.09.6, the error complains about the DOCKER iptables chain missing

````
linux-6my5:~ # docker version
Client:
Version: 18.09.6
API version: 1.39
Go version: go1.10.8
Git commit: 481bc7715621
Built: Mon May 6 12:00:00 2019
OS/Arch: linux/amd64
Experimental: false

Server:
Engine:
Version: 18.09.6
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: 481bc7715621
Built: Mon May 6 12:00:00 2019
OS/Arch: linux/amd64
Experimental: false
```

[wanghh2000]

@tao12345666333

root@ouch1:~# ./kind-linux-amd64 create cluster --loglevel=debug
DEBU[00:01:54] Running: /usr/bin/docker [docker ps -q -a --no-trunc --filter label=io.k8s.sigs.kind.cluster --format {{.Names}}\t{{.Label "io.k8s.sigs.kind.cluster"}}]
Creating cluster "kind" ...
DEBU[00:01:55] Running: /usr/bin/docker [docker inspect --type=image kindest/node:v1.14.2]
INFO[00:01:55] Image: kindest/node:v1.14.2 present locally
 ✓ Ensuring node image (kindest/node:v1.14.2) 🖼
DEBU[00:01:55] Running: /usr/bin/docker [docker info --format '{{json .SecurityOptions}}']
DEBU[00:01:55] Running: /usr/bin/docker [docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname kind-control-plane --name kind-control-plane --label io.k8s.sigs.kind.cluster=kind --label io.k8s.sigs.kind.role=control-plane --expose 40599 -p 127.0.0.1:40599:6443 kindest/node:v1.14.2@sha256:33539d830a6cf20e3e0a75d0c46a4e94730d78c7375435e6b49833d81448c319]
ERRO[00:01:58] 1649627c4b99b35837961e12db348b4e573c24ef4a2bd58ee2ec7db59e50aecc
ERRO[00:01:58] docker: Error response from daemon: driver failed programming external connectivity on endpoint kind-control-plane (e02afba3af22ef3be710c99105f1fbb949f275b3229eaac48388e503445ecf58):  (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 127.0.0.1 --dport 40599 -j DNAT --to-destination 172.17.0.2:6443 ! -i docker0: iptables: No chain/target/match by that name.
ERRO[00:01:58]  (exit status 1)).
 ✗ Preparing nodes 📦
ERRO[00:01:58] docker run error: exit status 125
DEBU[00:01:58] Running: /usr/bin/docker [docker ps -q -a --no-trunc --filter label=io.k8s.sigs.kind.cluster --format {{.Names}}\t{{.Label "io.k8s.sigs.kind.cluster"}} --filter label=io.k8s.sigs.kind.cluster=kind]
DEBU[00:01:58] Running: /usr/bin/docker [docker rm -f -v kind-control-plane]
⠈⠁ Preparing nodes 📦 Error: failed to create cluster: docker run error: exit status 125

@aojea Seems no found any for docker.

# iptables -L -t nat | grep DOCKER
#

[aojea]

@wanghh2000 that's the problem, please check your docker installation, kind uses the default docker network setups and needs that those tables exist.
There are similar errors reported https://github.com/moby/moby/issues/1871

[tao12345666333]

@wanghh2000 dockerd has one --iptables flag to enable addition of iptables rules (default true) .
If you accidentally cleaned out the default iptables rules, you can restart docker directly.

[wanghh2000]

@tao12345666333 @aojea By ticket #1871
Thanks! Problem solved after execute command:

iptables -t nat -N DOCKER
iptables -t nat -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
iptables -t nat -A PREROUTING -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER

systemctl restart docker

[wanghh2000]

Close issue.