Weave: Weave on a single centos7 host, Containers cant resolve each other
I have a single weave host, with two containers running on it.
I started weave
weave launch && weave launch-dns && weave launch-proxy
and then started the containers with docker commands,
docker run -d -ti -h webnode1 --name webnode1 -p 80:80 docker.io/httpd:latest
docker run -d -ti -h webnode2 --name webnode2 -p 8080:80 docker.io/httpd:latest
but i am not able to ping them from one another and then the traffic to the outside world is not passed through.
docker exec webnode1 ping -c 5 webnode2 results in unknown host webnode2
Running the latest version of weave 1.0.1, docker is also latest version.
miztiik
All 67 comments
You've missed out the eval $(weave proxy-env) step.
rade
on 6 Aug 2015
I did that as well
eval $(weave proxy-env)
Should i be using docker run or should i always be using
weave run <IP> -ti -h webnode2 --name webnode2 -p 8080:80 docker.io/httpd:latest
miztiik
on 6 Aug 2015
I see what the problem is. You are setting the hostnames to webnode1 and webnode2. That is lacking the weave.local domain and hence will not be registered in weavedns.
You shouldn't need to set the hostname at all since weave will derive it automatically from the container name, placing it in the weave.local domain. So just dropping the -h should fix your problem.
rade
on 6 Aug 2015
docker run -d -ti --name webnode1 -p 80:80 docker.io/httpd:latest
docker run -d -ti --name webnode2 -p 8080:80 docker.io/httpd:latest
[root@dockerHostCentOS7 docker-dns-build]# docker exec webnode1 ping -c 5 webnode2
ping: unknown host
Still no luck!!, not sure if this helps
[root@dockerHostCentOS7 docker-dns-build]# weave status
weave router 1.0.1
Our name is ce:48:64:23:08:43(dockerHostCentOS7)
Encryption off
Peer discovery on
Sniffing traffic on &{10 65535 ethwe a6:10:9d:1a:4a:76 up|broadcast|multicast}
MACs:
a6:10:9d:1a:4a:76 -> ce:48:64:23:08:43(dockerHostCentOS7) (2015-08-06 12:47:08.450437604 +0000 UTC)
ce:48:64:23:08:43 -> ce:48:64:23:08:43(dockerHostCentOS7) (2015-08-06 12:47:08.939177819 +0000 UTC)
ee:f9:e3:f2:91:ce -> ce:48:64:23:08:43(dockerHostCentOS7) (2015-08-06 12:47:11.06701967 +0000 UTC)
6a:d1:6f:fb:34:9a -> ce:48:64:23:08:43(dockerHostCentOS7) (2015-08-06 12:47:12.425499074 +0000 UTC)
Peers:
ce:48:64:23:08:43(dockerHostCentOS7) (v0) (UID 15219084075392424760)
Routes:
unicast:
ce:48:64:23:08:43 -> 00:00:00:00:00:00
broadcast:
ce:48:64:23:08:43 -> []
Direct Peers:
Reconnects:
Allocator range [10.128.0.0-10.192.0.0)
Owned Ranges:
10.128.0.0 -> ce:48:64:23:08:43 (dockerHostCentOS7) (v1)
Allocator default subnet: 10.128.0.0/10
weave DNS 1.0.1
Listen address :53
Fallback DNS config &{[8.8.8.8 8.8.4.4] 53 1 5 2}
Local domain weave.local.
Interface &{14 65535 ethwe ee:f9:e3:f2:91:ce up|broadcast|multicast}
Zone database:
weave proxy is running
are you sure you have run the eval $(weave proxy-env) _in the same shell where you execute docker run_? What does echo $DOCKER_HOST report in that shell?
rade
on 6 Aug 2015
Oops, i wrote a small bash script to start weave (and its containers) so the eval gets executed within the bash script. so in essence the docker run shell doesn't see the eval values
echo $DOCKER_HOST is empty, let me try it manually.
miztiik
on 6 Aug 2015
still no luck
[root@dockerHostCentOS7 docker-dns-build]# weave launch && weave launch-dns && weave launch-proxy
25be7916063b545ebdc7a0410a7cf6b140c126b8a373c81ef72261c698ec78b3
48c527c7203438f7ded1d3a84eb2324e9b7b3a19c7e94f80b83070a0204e2239
c962d4d085edb05d1838e1fc1726fee4fe3185f0de0b5c7c519eec003d81597e
[root@dockerHostCentOS7 docker-dns-build]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c962d4d085ed weaveworks/weaveexec:1.0.1 "/home/weave/weavepr 3 seconds ago Up 2 seconds weaveproxy
48c527c72034 weaveworks/weavedns:1.0.1 "/home/weave/weavedn 5 seconds ago Up 4 seconds 172.17.42.1:53->53/udp weavedns
25be7916063b weaveworks/weave:1.0.1 "/home/weave/weaver 9 seconds ago Up 8 seconds 0.0.0.0:6783->6783/tcp, 0.0.0.0:6783->6783/udp weave
[root@dockerHostCentOS7 docker-dns-build]# eval $(weave proxy-env)
[root@dockerHostCentOS7 docker-dns-build]# echo ^C
[root@dockerHostCentOS7 docker-dns-build]# echo $DOCKER_HOST
tcp://127.0.0.1:12375
[root@dockerHostCentOS7 docker-dns-build]#
[root@dockerHostCentOS7 docker-dns-build]# docker run -d -ti --name webnode1 -p 80:80 docker.io/httpd:latest
ac7c814b22a980d20bcea1ad2ea9a8bc4d0b371608a56bf110ea25bdcf824d70
[root@dockerHostCentOS7 docker-dns-build]# docker run -d -ti --name webnode2 -p 8080:80 docker.io/httpd:latest
b300d603b09831702207633c33a87f7fd8f1434669493b0b7dc97d36615896f3
[root@dockerHostCentOS7 docker-dns-build]# docker exec webnode1 ping -c 5 webnode2
ping: unknown host
[root@dockerHostCentOS7 docker-dns-build]#
[root@dockerHostCentOS7 docker-dns-build]# docker exec webnode1 more /etc/resolv.conf
::::::::::::::
/etc/resolv.conf
::::::::::::::
nameserver 172.17.42.1
[root@dockerHostCentOS7 docker-dns-build]# docker exec webnode2 more /etc/resolv.conf
::::::::::::::
/etc/resolv.conf
::::::::::::::
nameserver 172.17.42.1
Is it in any case related to this issue - https://github.com/weaveworks/weave/issues/1033
miztiik
on 6 Aug 2015
The above works absolutely fine for me; the ping succeeds :(
rade
on 6 Aug 2015
What do weave status and weave ps say?
rade
on 6 Aug 2015
I dont know if this has any bearing i start docker with the google dns servers
OPTIONS='--dns 8.8.8.8 --dns 8.8.4.4'
here is the weave status
[root@dockerHostCentOS7 docker-dns-build]# weave status
weave router 1.0.1
Our name is ce:48:64:23:08:43(dockerHostCentOS7)
Encryption off
Peer discovery on
Sniffing traffic on &{30 65535 ethwe 1a:e7:07:e3:50:8f up|broadcast|multicast}
MACs:
Peers:
ce:48:64:23:08:43(dockerHostCentOS7) (v0) (UID 7977121167346644011)
Routes:
unicast:
ce:48:64:23:08:43 -> 00:00:00:00:00:00
broadcast:
ce:48:64:23:08:43 -> []
Direct Peers:
Reconnects:
Allocator range [10.128.0.0-10.192.0.0)
Owned Ranges:
10.128.0.0 -> ce:48:64:23:08:43 (dockerHostCentOS7) (v3)
Allocator default subnet: 10.128.0.0/10
weave DNS 1.0.1
Listen address :53
Fallback DNS config &{[8.8.8.8 8.8.4.4] 53 1 5 2}
Local domain weave.local.
Interface &{34 65535 ethwe 66:6d:cf:aa:f1:85 up|broadcast|multicast}
Zone database:
ac7c814b22a9: webnode1.weave.local.[10.128.0.2]
b300d603b098: webnode2.weave.local.[10.128.0.3]
weave proxy is running
Here is the output for weave ps
[root@dockerHostCentOS7 docker-dns-build]# weave ps
weave:expose ce:48:64:23:08:43
b300d603b098 1e:12:b7:6d:51:7f 10.128.0.3/10
ac7c814b22a9 fe:99:4e:ee:e4:80 10.128.0.2/10
48c527c72034 66:6d:cf:aa:f1:85 10.128.0.1/10
25be7916063b 1a:e7:07:e3:50:8f
[root@dockerHostCentOS7 docker-dns-build]#
That status/ps output looks fine.
Please repeat the test, but this time launch weavedns with weave launch weavedns -debug. And then, after the failed ping, post the output of docker logs weavedns.
rade
on 6 Aug 2015
docker is also latest version
i.e. 1.7.1?
rade
on 6 Aug 2015
root@dockerHostCentOS7 ~]# weave launch && weave launch-dns -debug && weave launch-proxy
aeb80ba06ba4c423ef72620add40af015a3ea3e277fd8b7d3b913c330ab1e046
61c844154d7ec0e727dc948535eea0ae341f2ca9c92afe98cd1089c305164d14
1af9023b53e8e348d28ac27b1d8c392e53a25b840165b9893f8c76f951066bb5
[root@dockerHostCentOS7 ~]# eval $(weave proxy-env)
[root@dockerHostCentOS7 ~]#
[root@dockerHostCentOS7 ~]# docker run -d -ti --name webnode1 -p 80:80 docker.io/httpd:latest
58735e07ff849baecc11c1f65db3051cd5d159cc90771c4342d9559e3d5b48d6
[root@dockerHostCentOS7 ~]# docker run -d -ti --name webnode2 -p 8080:80 docker.io/httpd:latest
95f61e13db4f2ab592b2be68045d7ada39ceef5d0f27140d7328d2800be01086
[root@dockerHostCentOS7 ~]#
[root@dockerHostCentOS7 ~]# docker exec webnode1 ping -c 5 webnode2
ping: unknown host
Here are the debug messages for weavedns
[root@dockerHostCentOS7 ~]# docker logs weavedns
INFO: 2015/08/06 13:28:04.626552 [main] WeaveDNS version 1.0.1
INFO: 2015/08/06 13:28:04.630798 [main] Waiting for mDNS interface ethwe to come up
INFO: 2015/08/06 13:28:05.043068 [main] Interface ethwe is up
INFO: 2015/08/06 13:28:05.043106 [main] Waiting for HTTP interface eth0 to come up
INFO: 2015/08/06 13:28:05.043238 [main] Interface eth0 is up
INFO: 2015/08/06 13:28:05.043671 [zonedb] Using mDNS on &{Index:82 MTU:65535 Name:ethwe HardwareAddr:ae:26:cd:0c:59:f1 Flags:up|broadcast|multicast}
INFO: 2015/08/06 13:28:05.063381 [docker] Using Docker API on unix:///var/run/docker.sock: &[ApiVersion=1.18 Arch=amd64 GitCommit=ba1f6c3/1.6.2 GoVersion=go1.4.2 KernelVersion=3.10.0-229.11.1.el7.x86_64 Os=linux Version=1.6.2]
INFO: 2015/08/06 13:28:05.070432 [main] HTTP API listening on 172.17.0.11:6785
DEBUG: 2015/08/06 13:28:05.080515 [dns] Creating UDP DNS client with 5s timeout
DEBUG: 2015/08/06 13:28:05.080574 [dns] Creating UDP DNS client with 5s timeout
DEBUG: 2015/08/06 13:28:05.080793 [dns] Creating TCP DNS client with 5s timeout
DEBUG: 2015/08/06 13:28:05.080816 [dns] Creating TCP DNS client with 5s timeout
INFO: 2015/08/06 13:28:05.080145 [dns] Upstream server(s): &{Servers:[8.8.8.8 8.8.4.4] Search:[in.ibm.com] Port:53 Ndots:1 Timeout:5 Attempts:2}
INFO: 2015/08/06 13:28:05.080199 [dns] Cache: 8192 entries
INFO: 2015/08/06 13:28:05.107393 [dns] Listening for DNS on :53 (UDP)
INFO: 2015/08/06 13:28:05.125025 [dns] Listening for DNS on :53 (TCP)
INFO: 2015/08/06 13:28:05.662558 [zonedb] Container bb567ae2d11d653085aafab5b22f657bae6d05ccfc4317165b96bb7015bd5c3b down. Removing records
INFO: 2015/08/06 13:28:07.286273 [zonedb] Container 59d47b61cda90b6abadbd98d678a7ce51c566bce2d91b3bbc415d8c4cc6aa7f9 down. Removing records
INFO: 2015/08/06 13:28:08.158429 [zonedb] Container a43cd000785a08f9381a7a950f7e4bb327405a7fa2b79f43301cd7de8185ac59 down. Removing records
INFO: 2015/08/06 13:28:13.038294 [http] Adding webnode1.weave.local -> 10.128.0.2
DEBUG: 2015/08/06 13:28:13.038341 [zonedb] Adding record: '58735e07ff849baecc11c1f65db3051cd5d159cc90771c4342d9559e3d5b48d6'/'webnode1.weave.local'[10.128.0.2]
DEBUG: 2015/08/06 13:28:13.038378 [zonedb] Touching name webnode1.weave.local.
INFO: 2015/08/06 13:28:14.666202 [http] Adding webnode2.weave.local -> 10.128.0.3
DEBUG: 2015/08/06 13:28:14.666252 [zonedb] Adding record: '95f61e13db4f2ab592b2be68045d7ada39ceef5d0f27140d7328d2800be01086'/'webnode2.weave.local'[10.128.0.3]
DEBUG: 2015/08/06 13:28:14.666277 [zonedb] Touching name webnode2.weave.local.
Docker version
[root@dockerHostCentOS7 ~]# docker version
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): ba1f6c3/1.6.2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): ba1f6c3/1.6.2
OS/Arch (server): linux/amd64
So it looks like the DNS queries never make it to weavedns.
On the host, run dig +short @$(weave docker-bridge-ip) webnode2.weave.local. What does that return? It _should_ return the weave IP of webnode2.
1.6.2
That is _not_ the latest version. Any chance you could try with 1.7.1?
rade
on 6 Aug 2015
Ok got 1.7.1 installed and re did the test with debug option
[root@dockerHostCentOS7 ~]# weave status
weave router 1.0.1
Our name is ce:48:64:23:08:43(dockerHostCentOS7)
Encryption off
Peer discovery on
Sniffing traffic on &{94 65535 ethwe be:9d:1b:59:4b:4d up|broadcast|multicast}
MACs:
be:9d:1b:59:4b:4d -> ce:48:64:23:08:43(dockerHostCentOS7) (2015-08-06 13:41:14.202195777 +0000 UTC)
b2:32:d0:ac:f5:1e -> ce:48:64:23:08:43(dockerHostCentOS7) (2015-08-06 13:41:21.753432001 +0000 UTC)
6a:9c:bc:70:5f:1b -> ce:48:64:23:08:43(dockerHostCentOS7) (2015-08-06 13:41:33.052924798 +0000 UTC)
e6:02:62:c2:29:7b -> ce:48:64:23:08:43(dockerHostCentOS7) (2015-08-06 13:44:00.081181869 +0000 UTC)
3a:8b:b8:d1:33:a5 -> ce:48:64:23:08:43(dockerHostCentOS7) (2015-08-06 13:44:01.762989946 +0000 UTC)
Peers:
ce:48:64:23:08:43(dockerHostCentOS7) (v0) (UID 16730361611156887707)
Routes:
unicast:
ce:48:64:23:08:43 -> 00:00:00:00:00:00
broadcast:
ce:48:64:23:08:43 -> []
Direct Peers:
Reconnects:
Allocator range [10.128.0.0-10.192.0.0)
Owned Ranges:
10.128.0.0 -> ce:48:64:23:08:43 (dockerHostCentOS7) (v3)
Allocator default subnet: 10.128.0.0/10
weave DNS 1.0.1
Listen address :53
Fallback DNS config &{[8.8.8.8 8.8.4.4] [in.ibm.com] 53 1 5 2}
Local domain weave.local.
Interface &{98 65535 ethwe 6a:9c:bc:70:5f:1b up|broadcast|multicast}
Zone database:
1858f57d3eea: webnode1.weave.local.[10.128.0.2]
629ce08747a6: webnode2.weave.local.[10.128.0.3]
weave proxy is running
Here is the ping test, no luck yet.
[root@dockerHostCentOS7 ~]# docker exec webnode1 ping -c 5 webnode2
ping: unknown host
Docker version
[root@dockerHostCentOS7 ~]# docker version
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/amd64
Here are the logs
[root@dockerHostCentOS7 ~]# docker logs weavedns
INFO: 2015/08/06 13:41:32.195256 [main] WeaveDNS version 1.0.1
INFO: 2015/08/06 13:41:32.195354 [main] Waiting for mDNS interface ethwe to come up
INFO: 2015/08/06 13:41:33.050785 [main] Interface ethwe is up
INFO: 2015/08/06 13:41:33.050817 [main] Waiting for HTTP interface eth0 to come up
INFO: 2015/08/06 13:41:33.050950 [main] Interface eth0 is up
INFO: 2015/08/06 13:41:33.051424 [zonedb] Using mDNS on &{Index:98 MTU:65535 Name:ethwe HardwareAddr:6a:9c:bc:70:5f:1b Flags:up|broadcast|multicast}
INFO: 2015/08/06 13:41:33.163600 [docker] Using Docker API on unix:///var/run/docker.sock: &[GitCommit=786b29d GoVersion=go1.4.2 Os=linux Arch=amd64 KernelVersion=3.10.0-229.11.1.el7.x86_64 Version=1.7.1 ApiVersion=1.19]
INFO: 2015/08/06 13:41:33.178179 [main] HTTP API listening on 172.17.0.2:6785
INFO: 2015/08/06 13:41:33.178376 [dns] Upstream server(s): &{Servers:[8.8.8.8 8.8.4.4] Search:[in.ibm.com] Port:53 Ndots:1 Timeout:5 Attempts:2}
INFO: 2015/08/06 13:41:33.178398 [dns] Cache: 8192 entries
DEBUG: 2015/08/06 13:41:33.195525 [dns] Creating UDP DNS client with 5s timeout
DEBUG: 2015/08/06 13:41:33.195725 [dns] Creating UDP DNS client with 5s timeout
DEBUG: 2015/08/06 13:41:33.196103 [dns] Creating TCP DNS client with 5s timeout
DEBUG: 2015/08/06 13:41:33.196128 [dns] Creating TCP DNS client with 5s timeout
INFO: 2015/08/06 13:41:33.236883 [dns] Listening for DNS on :53 (UDP)
INFO: 2015/08/06 13:41:33.240972 [dns] Listening for DNS on :53 (TCP)
INFO: 2015/08/06 13:41:35.020809 [zonedb] Container 396466ab071c8b0559dde0cbc039f1c20aca0230b474c3b219dc76c249d8207f down. Removing records
INFO: 2015/08/06 13:41:39.637666 [zonedb] Container 8e0fa47c784572f3bf9d65314dd8f44f999f32a745114aa398b3bccde6db9cf4 down. Removing records
INFO: 2015/08/06 13:41:41.583835 [zonedb] Container 73299a674914d85b06b6e7a6217ddd6bfd78e13d7c85036a0f6e1ebdf37eefea down. Removing records
INFO: 2015/08/06 13:42:37.717582 [zonedb] Container 601e164f0ec4b643927afa525b9b1173570945ec9f51655d564051be8206af56 down. Removing records
INFO: 2015/08/06 13:43:53.309294 [zonedb] Container c2c57784018df5f0c53dd984f7e299e3730ad30f9a0a95552135a87ebc08c0f9 down. Removing records
INFO: 2015/08/06 13:43:53.603372 [zonedb] Container 064d78dcfe3d916549b59f1ea156f08d2e6a25f6d935e1f25818dbcb8c93c6ac down. Removing records
INFO: 2015/08/06 13:44:00.413339 [http] Adding webnode1.weave.local -> 10.128.0.2
DEBUG: 2015/08/06 13:44:00.413405 [zonedb] Adding record: '1858f57d3eeaec73c09c82c7680cd3784c7d89f15e044ac2e9636f5e48ce5b25'/'webnode1.weave.local'[10.128.0.2]
DEBUG: 2015/08/06 13:44:00.413450 [zonedb] Touching name webnode1.weave.local.
INFO: 2015/08/06 13:44:02.102818 [http] Adding webnode2.weave.local -> 10.128.0.3
DEBUG: 2015/08/06 13:44:02.102882 [zonedb] Adding record: '629ce08747a6f41e48e57808f509f8b75c51d7fdb7ac83a1c8e3eb5121221bd9'/'webnode2.weave.local'[10.128.0.3]
DEBUG: 2015/08/06 13:44:02.102923 [zonedb] Touching name webnode2.weave.local.
INFO: 2015/08/06 13:44:08.185641 [zonedb] Container 95b8d1db7c1a059a76103096924bc09a516c0cb9526d3897baf412a431b02d4a down. Removing records
[root@dockerHostCentOS7 ~]#
This is a minimal install of centos doesn't have dig, let me install and try that command.
miztiik
on 6 Aug 2015
Here is the output of the dig command.
[root@dockerHostCentOS7 ~]# dig +short @$(weave docker-bridge-ip) webnode2.weave.local
10.128.0.3
dig for webnode2 get the NXDOMAIN
[root@dockerHostCentOS7 ~]# dig webnode2.weave.local
; <<>> DiG 9.9.4-RedHat-9.9.4-18.el7_1.3 <<>> webnode2.weave.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 53133
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;webnode2.weave.local. IN A
;; AUTHORITY SECTION:
. 1799 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2015080600 1800 900 604800 86400
;; Query time: 383 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Aug 06 19:21:08 IST 2015
;; MSG SIZE rcvd: 124
output of the dig command
That looks good.
What do you get from docker exec webnode1 getent hosts webnode2 and docker exec webnode1 getent hosts webnode2.weave.local?
rade
on 6 Aug 2015
not sure i did it right, there was no output
[root@dockerHostCentOS7 ~]# docker exec webnode1 getent hosts webnode2
[root@dockerHostCentOS7 ~]# docker exec webnode1 getent hosts webnode2.weave.local
[root@dockerHostCentOS7 ~]#
So somehow name resolution from inside the containers just doesn't work.
does docker exec webnode1 ping -c1 172.17.42.1 work?
the output of iptables-save would be interesting too.
rade
on 6 Aug 2015
[root@dockerHostCentOS7 ~]# docker exec webnode1 ping -c1 172.17.42.1
PING 172.17.42.1 (172.17.42.1): 56 data bytes
64 bytes from 172.17.42.1: icmp_seq=0 ttl=64 time=0.198 ms
--- 172.17.42.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.198/0.198/0.198/0.000 ms
The output of iptables-save
[root@dockerHostCentOS7 ~]# iptables-save
# Generated by iptables-save v1.4.21 on Thu Aug 6 19:31:28 2015
*nat
:PREROUTING ACCEPT [22:1981]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [76:4704]
:POSTROUTING ACCEPT [77:4781]
:DOCKER - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_ZONES - [0:0]
:POSTROUTING_ZONES_SOURCE - [0:0]
:POSTROUTING_direct - [0:0]
:POST_public - [0:0]
:POST_public_allow - [0:0]
:POST_public_deny - [0:0]
:POST_public_log - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
:WEAVE - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -j OUTPUT_direct
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -j POSTROUTING_direct
-A POSTROUTING -j POSTROUTING_ZONES_SOURCE
-A POSTROUTING -j POSTROUTING_ZONES
-A POSTROUTING -j WEAVE
-A POSTROUTING -s 172.17.0.1/32 -d 172.17.0.1/32 -p tcp -m tcp --dport 6783 -j MASQUERADE
-A POSTROUTING -s 172.17.0.1/32 -d 172.17.0.1/32 -p udp -m udp --dport 6783 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p udp -m udp --dport 53 -j MASQUERADE
-A POSTROUTING -s 172.17.0.5/32 -d 172.17.0.5/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -s 172.17.0.6/32 -d 172.17.0.6/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 6783 -j DNAT --to-destination 172.17.0.1:6783
-A DOCKER ! -i docker0 -p udp -m udp --dport 6783 -j DNAT --to-destination 172.17.0.1:6783
-A DOCKER -d 172.17.42.1/32 ! -i docker0 -p udp -m udp --dport 53 -j DNAT --to-destination 172.17.0.2:53
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.17.0.5:80
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 8080 -j DNAT --to-destination 172.17.0.6:80
-A POSTROUTING_ZONES -o eth1 -g POST_public
-A POSTROUTING_ZONES -o eth0 -g POST_public
-A POSTROUTING_ZONES -g POST_public
-A POST_public -j POST_public_log
-A POST_public -j POST_public_deny
-A POST_public -j POST_public_allow
-A PREROUTING_ZONES -i eth1 -g PRE_public
-A PREROUTING_ZONES -i eth0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 6 19:31:28 2015
# Generated by iptables-save v1.4.21 on Thu Aug 6 19:31:28 2015
*mangle
:PREROUTING ACCEPT [25477:176361095]
:INPUT ACCEPT [25436:176358324]
:FORWARD ACCEPT [20:1905]
:OUTPUT ACCEPT [21775:2086208]
:POSTROUTING ACCEPT [21795:2088113]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -j POSTROUTING_direct
-A PREROUTING_ZONES -i eth1 -g PRE_public
-A PREROUTING_ZONES -i eth0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 6 19:31:28 2015
# Generated by iptables-save v1.4.21 on Thu Aug 6 19:31:28 2015
*security
:INPUT ACCEPT [25288:176349541]
:FORWARD ACCEPT [20:1905]
:OUTPUT ACCEPT [21775:2086208]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
COMMIT
# Completed on Thu Aug 6 19:31:28 2015
# Generated by iptables-save v1.4.21 on Thu Aug 6 19:31:28 2015
*raw
:PREROUTING ACCEPT [25478:176361671]
:OUTPUT ACCEPT [21775:2086208]
:OUTPUT_direct - [0:0]
:PREROUTING_direct - [0:0]
-A PREROUTING -j PREROUTING_direct
-A OUTPUT -j OUTPUT_direct
COMMIT
# Completed on Thu Aug 6 19:31:28 2015
# Generated by iptables-save v1.4.21 on Thu Aug 6 19:31:28 2015
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1813:240660]
:DOCKER - [0:0]
:FORWARD_IN_ZONES - [0:0]
:FORWARD_IN_ZONES_SOURCE - [0:0]
:FORWARD_OUT_ZONES - [0:0]
:FORWARD_OUT_ZONES_SOURCE - [0:0]
:FORWARD_direct - [0:0]
:FWDI_public - [0:0]
:FWDI_public_allow - [0:0]
:FWDI_public_deny - [0:0]
:FWDI_public_log - [0:0]
:FWDO_public - [0:0]
:FWDO_public_allow - [0:0]
:FWDO_public_deny - [0:0]
:FWDO_public_log - [0:0]
:INPUT_ZONES - [0:0]
:INPUT_ZONES_SOURCE - [0:0]
:INPUT_direct - [0:0]
:IN_public - [0:0]
:IN_public_allow - [0:0]
:IN_public_deny - [0:0]
:IN_public_log - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -p icmp -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -i docker0 -o weave -j DROP
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct
-A FORWARD -j FORWARD_IN_ZONES_SOURCE
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD -j FORWARD_OUT_ZONES_SOURCE
-A FORWARD -j FORWARD_OUT_ZONES
-A FORWARD -p icmp -j ACCEPT
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -i weave -o weave -j ACCEPT
-A OUTPUT -j OUTPUT_direct
-A DOCKER -d 172.17.0.1/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 6783 -j ACCEPT
-A DOCKER -d 172.17.0.1/32 ! -i docker0 -o docker0 -p udp -m udp --dport 6783 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p udp -m udp --dport 53 -j ACCEPT
-A DOCKER -d 172.17.0.5/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -d 172.17.0.6/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT
-A FORWARD_IN_ZONES -i eth1 -g FWDI_public
-A FORWARD_IN_ZONES -i eth0 -g FWDI_public
-A FORWARD_IN_ZONES -g FWDI_public
-A FORWARD_OUT_ZONES -o eth1 -g FWDO_public
-A FORWARD_OUT_ZONES -o eth0 -g FWDO_public
-A FORWARD_OUT_ZONES -g FWDO_public
-A FWDI_public -j FWDI_public_log
-A FWDI_public -j FWDI_public_deny
-A FWDI_public -j FWDI_public_allow
-A FWDO_public -j FWDO_public_log
-A FWDO_public -j FWDO_public_deny
-A FWDO_public -j FWDO_public_allow
-A INPUT_ZONES -i eth1 -g IN_public
-A INPUT_ZONES -i eth0 -g IN_public
-A INPUT_ZONES -g IN_public
-A IN_public -j IN_public_log
-A IN_public -j IN_public_deny
-A IN_public -j IN_public_allow
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
COMMIT
# Completed on Thu Aug 6 19:31:28 2015
[root@dockerHostCentOS7 ~]#
[root@dockerHostCentOS7 ~]# exec webnode1 ping -c1 172.17.42.1
-bash: exec: webnode1: not found
_docker_ exec ...
rade
on 6 Aug 2015
I corrected it again. ping to host IP is working
miztiik
on 6 Aug 2015
I corrected it again. ping with IP is working
I want to know whether pinging _that specific IP_, which is the docker bridge IP, is working from inside containers. Does it?
rade
on 6 Aug 2015
yes
[root@dockerHostCentOS7 ~]# docker exec webnode1 ping -c5 172.17.42.1
PING 172.17.42.1 (172.17.42.1): 56 data bytes
64 bytes from 172.17.42.1: icmp_seq=0 ttl=64 time=0.189 ms
64 bytes from 172.17.42.1: icmp_seq=1 ttl=64 time=0.194 ms
64 bytes from 172.17.42.1: icmp_seq=2 ttl=64 time=0.189 ms
64 bytes from 172.17.42.1: icmp_seq=3 ttl=64 time=0.190 ms
64 bytes from 172.17.42.1: icmp_seq=4 ttl=64 time=0.194 ms
--- 172.17.42.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.189/0.191/0.194/0.000 ms
[root@dockerHostCentOS7 ~]#
Any chance you can either a) give me access to that machine, or b) tell me how to create a VM with the exact same configuration as you have?
rade
on 6 Aug 2015
If you have teamviewer, 291 280 615 & ``
miztiik
on 6 Aug 2015
tcpdumping docker0 while running docker exec webnode1 getent hosts webnode2.weave.local revealed some ICMP packets getting rejected. I dropped the following rules:
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
made things work!
So the question is a) why are those rules there, and b) how exactly do they break things.
rade
on 6 Aug 2015
Thank you very much Rade for working with me to fix this issue.
I have uploaded the PCAP file here - https://github.com/miztiik/cloudera-On-Docker/blob/master/issues/dump1.pcap
Let me know if you have any problem accessing it.
miztiik
on 6 Aug 2015
Here's the text version of the pcap:
15:41:43.206673 IP 172.17.42.1.59352 > 172.17.0.2.6785: Flags [.], ack 2837717397, win 140, options [nop,nop,TS val 6827392 ecr 6797312], length 0
15:41:43.206846 IP 172.17.0.2.6785 > 172.17.42.1.59352: Flags [.], ack 1, win 114, options [nop,nop,TS val 6827392 ecr 6616976], length 0
15:41:48.214650 ARP, Request who-has 172.17.42.1 tell 172.17.0.2, length 28
15:41:48.214707 ARP, Reply 172.17.42.1 is-at 56:84:7a:fe:97:99, length 28
15:42:03.761886 IP 172.17.0.5.43258 > 172.17.42.1.53: 11374+ AAAA? webnode2.weave.local. (38)
15:42:03.762023 IP 172.17.42.1 > 172.17.0.5: ICMP host 172.17.42.1 unreachable - admin prohibited, length 74
15:42:03.762267 IP 172.17.0.5.39505 > 172.17.42.1.53: 11374+ AAAA? webnode2.weave.local. (38)
15:42:03.762314 IP 172.17.42.1 > 172.17.0.5: ICMP host 172.17.42.1 unreachable - admin prohibited, length 74
15:42:03.762555 IP 172.17.0.5.58101 > 172.17.42.1.53: 20584+ AAAA? webnode2.weave.local.weave.local. (50)
15:42:03.762591 IP 172.17.42.1 > 172.17.0.5: ICMP host 172.17.42.1 unreachable - admin prohibited, length 86
15:42:03.762744 IP 172.17.0.5.57628 > 172.17.42.1.53: 20584+ AAAA? webnode2.weave.local.weave.local. (50)
15:42:03.762781 IP 172.17.42.1 > 172.17.0.5: ICMP host 172.17.42.1 unreachable - admin prohibited, length 86
15:42:03.763057 IP 172.17.0.5.43729 > 172.17.42.1.53: 40840+ A? webnode2.weave.local. (38)
15:42:03.763097 IP 172.17.42.1 > 172.17.0.5: ICMP host 172.17.42.1 unreachable - admin prohibited, length 74
15:42:03.763199 IP 172.17.0.5.36321 > 172.17.42.1.53: 40840+ A? webnode2.weave.local. (38)
15:42:03.763230 IP 172.17.42.1 > 172.17.0.5: ICMP host 172.17.42.1 unreachable - admin prohibited, length 74
15:42:03.763327 IP 172.17.0.5.52089 > 172.17.42.1.53: 32570+ A? webnode2.weave.local.weave.local. (50)
15:42:08.768780 IP 172.17.0.5.52089 > 172.17.42.1.53: 32570+ A? webnode2.weave.local.weave.local. (50)
15:42:08.768879 IP 172.17.42.1 > 172.17.0.5: ICMP host 172.17.42.1 unreachable - admin prohibited, length 86
15:42:08.774670 ARP, Request who-has 172.17.0.5 tell 172.17.42.1, length 28
15:42:08.774662 ARP, Request who-has 172.17.42.1 tell 172.17.0.5, length 28
15:42:08.774713 ARP, Reply 172.17.42.1 is-at 56:84:7a:fe:97:99, length 28
15:42:08.774724 ARP, Reply 172.17.0.5 is-at 02:42:ac:11:00:05, length 28
15:42:13.287582 IP 172.17.42.1.59352 > 172.17.0.2.6785: Flags [.], ack 1, win 140, options [nop,nop,TS val 6857472 ecr 6827392], length 0
15:42:13.287680 IP 172.17.0.2.6785 > 172.17.42.1.59352: Flags [.], ack 1, win 114, options [nop,nop,TS val 6857473 ecr 6616976], length 0
I have used WeaveDNS on CentOS 7 without any issues, but I have SELinux disabled. The message unreachable - admin prohibited suggest to me that disabling SELinux might do the trick.
errordeveloper
on 13 Aug 2015
I have used WeaveDNS on CentOS 7 without any issues, but I have SELinux disabled. The message unreachable - admin prohibited suggest to me that disabling SELinux might do the trick.
Nope. I have reproduced this. It's due to how firewalld sets up iptables, not SELinux. That error message is indicating an ICMP destination unreachable, code 13.
dpw
on 13 Aug 2015
which begs the question why weavedns works for @errordeveloper.
rade
on 13 Aug 2015
[vagrant@mesos-00 ~]$ sudo systemctl status firewalld
firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled)
Active: inactive (dead)
So @errordeveloper, if you systemd enable firewalld, does weavedns stop working for you?
dpw
on 13 Aug 2015
So @errordeveloper, if you systemd enable firewalld, does weavedns stop working for you?
Yes.
[vagrant@mesos-01 ~]$ docker -H localhost:12375 run -ti --name=pinger ubuntu
root@pinger:/# ping pingme
PING pingme.weave.local (10.160.0.1) 56(84) bytes of data.
64 bytes from pingme.weave.local (10.160.0.1): icmp_seq=1 ttl=64 time=1.85 ms
64 bytes from pingme.weave.local (10.160.0.1): icmp_seq=2 ttl=64 time=0.867 ms
64 bytes from pingme.weave.local (10.160.0.1): icmp_seq=3 ttl=64 time=1.22 ms
^C
--- pingme.weave.local ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 0.867/1.317/1.857/0.409 ms
root@pinger:/# exit
[vagrant@mesos-01 ~]$ sudo systemctl enable firewalld
ln -s '/usr/lib/systemd/system/firewalld.service' '/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service'
ln -s '/usr/lib/systemd/system/firewalld.service' '/etc/systemd/system/basic.target.wants/firewalld.service'
[vagrant@mesos-01 ~]$ sudo systemctl start firewalld
[vagrant@mesos-01 ~]$ docker -H localhost:12375 rm -f pinger
pinger
[vagrant@mesos-01 ~]$ docker -H localhost:12375 run -ti --name=pinger ubuntu
root@pinger:/# ping pingme
ping: unknown host pingme
root@pinger:/#
@errordeveloper How come firewalld is off on those machines? Did disabling selinux turn it off?
rade
on 13 Aug 2015
@rade I wasn't able to find where build scripts leave, but I think the motivation was to avoid it getting in a way in development, which makes sense as it's a Vagrant box and development is what it's for.
errordeveloper
on 14 Aug 2015
The box I'm using is from Chef folks, which is a popular one basically.
errordeveloper
on 14 Aug 2015
@errordeveloper do you have an example of how to fix this via firewalld / firewall-cmd?
I ask because EL7 based distro's are heading in this direction so I would like to know how to "properly" fix this instead of reverting those systems back to iptables management via iptables-save etc, since that's basically working against the direction the distro's are taking
alex-sherwin
on 8 Oct 2015
Even kubernetes, a project to which RH contributes, currently requires firewalld to be disabled. See:
https://access.redhat.com/articles/1198103
https://github.com/kubernetes/kubernetes/issues/10398
While firewalld does seem to have support for adding arbitrary iptables rules (https://fedoraproject.org/wiki/FirewallD#Direct_options), that suggests you invoke firewall-cmd. This is awful from the perspective of weave or anything else that wants to run in a container. You can manipulate iptables from a container that has --net=host. But unless there is some kind of cross-container dbus integration (a prospect that makes me shudder), you can't integrate with firewalld from inside a container.
So the firewalld cure seems worse than the disease right now. If RHEL insists on going down that road, maybe we'll have to do it at some point. But it looks unpleasant, and I'd like to see some more projects like k8s go through the pain first. Maybe the folks inside RH who do that will be able to convince their colleagues responsible for firewalld to make it less objectionable.
dpw
on 9 Oct 2015
Ok, I just read through this: https://github.com/kubernetes/kubernetes/pull/12396
In particular:
For the second part, we can listen for D-Bus messages indicating that firewalld has restarted or reloaded, and then re-apply all of our rules when it does. (This is what the docker firewalld patches do as well.)
So what docker (and now k8s) have done is more like tolerating firewalld than supporting it. They are not using it in a way that yields the benefits firewalld is supposed to provide (i.e. coordination of access to iptables).
But even that requires use of dbus, which is a problem for us.
dpw
on 9 Oct 2015
I'm fine with the warning, and applying a fix manually
I'm just saying the lack of offering a resolution using the tools at hand (firewalld/firewall-cmd) either alongside the warning message or in your documentation is problematic
In the scenario where I just want to go get up and running with Docker + Weave, it turns into a two-command install into a multi-hour foray of researching WTF firewalld is and how it interop's with iptables
I'm just saying, if you know a way to manually fix this using firewalld/firewall-cmd, you should make that info easy to find
Thanks,
alex-sherwin
on 9 Oct 2015
I'm fine with the warning, and applying a fix manually
I'm just saying the lack of offering a resolution using the tools at hand (firewalld/firewall-cmd) either alongside the warning message or in your documentation is problematic
In the scenario where I just want to go get up and running with Docker + Weave, it turns into a two-command install into a multi-hour foray of researching WTF firewalld is and how it interop's with iptables
I'm just saying, if you know a way to manually fix this using firewalld/firewall-cmd, you should make that info easy to find
I don't think there is a simple manual resolution other than disabling firewalld, and I appreciate that is not very satisfying to you. It's not satisfying to me either.
And I take your point that the warning message is not very specific. But the obvious fix for the would be to make it say how to disabled firewalld.
dpw
on 11 Oct 2015
Can you add the command to send to firewalld to remove the 2 offending rules?
Thanks
gioppoluca
on 16 Oct 2015
Set the iptables to allow for weave to work - https://github.com/weaveworks/weave/issues/1266
iptables -D INPUT -j REJECT --reject-with icmp-host-prohibited
iptables -D FORWARD -j REJECT --reject-with icmp-host-prohibited
Can you add the command to send to firewalld to remove the 2 offending rules?
We didn't find any way to ask firewalld to do this. And if you remove them with iptables, but you don't disable firewalld, then it will put them back when it restarts.
dpw
on 16 Oct 2015
If the new rules are saved, they would persist,
# Reload the firewall configuration and make it permenent
firewall-cmd --reload
firewall-cmd --permanent
@miztiik: Are you suggesting that weave should use firewalld to add the rules?
dpw
on 16 Oct 2015
The commands
firewall-cmd --reload
firewall-cmd --permanent
do NOT make permanent changes made by iptables command, tryed and does not work if someone knows how to make those two commands permanent could be helpful
Thanks
gioppoluca
on 16 Oct 2015
do NOT make permanent changes made by iptables command, tryed and does not work if someone knows how to make those two commands permanent could be helpful
I don't think there is any way to do this.
dpw
on 16 Oct 2015
@dpw We can't do it automatically, but it could be put in an init script on the host, right?
squaremo
on 16 Oct 2015
It means that the firewalld command is not able to remove some of the rule it manages????
Seems very strange.
There HAS to be a way!!!
gioppoluca
on 16 Oct 2015
@dpw We can't do it automatically, but it could be put in an init script on the host, right?
It could. But that's not necessary a robust approach: If firewalll d restarts for any reason, it will restore the rules.
(I expect that any system with firewalld is using systemd, so strictly speaking it would involve a service file, not an init script.)
dpw
on 16 Oct 2015
No I mean that there should be a way to remove permanently that rules using firewalld.
As is possible to open a port it should also be possible to do that.
If is possible to do it with iptables it HAS to be possible with firewalld.
Any tool should enable you to set all the rules you want, my problem is that I'm not familiar with it and have not found a way to do it.
gioppoluca
on 19 Oct 2015
@gioppoluca I think we're all agreed that it _should_ allow these things, but unfortunately at present it appears not to. Do RedHat have anything to say on the matter?
awh
on 19 Oct 2015
In case anyone else comes across this, and has the same issue and desire:
"As someone using firewalld, I want to use weaveDNS, without disabling firewalld, and i am willing to run a manual command to set up the firewall to make it work"
Here is the firewall-cmd answer:
First, you select a zone, and add your weave and docker interfaces to it:
sudo firewall-cmd --zone=internal --add-interface=weave --permanent
sudo firewall-cmd --zone=internal --add-interface=docker --permanent
Then, add the dns service to those interfaces:
sudo firewall-cmd --zone=internal --add-service=dns --permanent
and finally reload your configuation ( you'll probably have to reload docker too )
sudo firewall-cmd --reload
sudo service docker restart
when you launch weave, you will still get the message complaining about the reject rule. But in fact, it will still work because firewall-cmd has inserted a rule that will allow the DNS traffic.
Hope this helps someone else!
dcowden
on 17 Mar 2016
@dcowden dude still not working, weaveDNS cant resolve a name on my containers, any ideas?
joserivca
on 20 Apr 2016
Hmm, sorry no other ideas... does it start working when you disable firewalld completely?
sudo systemctl disable firewalld
That test will tell u if it is a firewall issue or not
dcowden
on 20 Apr 2016
yeah! with the firewalld is down works fine!, but in your comment of 17 MAR, you wrote that works for you ?
joserivca
on 20 Apr 2016
@joserivca did you follow the exact steps @dcowden posted?
rade
on 20 Apr 2016
@rade yes, but still appears that message
WARNING: existing iptables rule
'-A FORWARD -j REJECT --reject-with icmp-host-prohibited'
will block name resolution via weaveDNS - please reconfigure your firewall.
'-A FORWARD -j REJECT --reject-with icmp-host-prohibited'
will block name resolution via weaveDNS - please reconfigure your firewall.
this is my iptables-save
Generated by iptables-save v1.4.21 on Wed Apr 20 16:42:34 2016
*nat
:PREROUTING ACCEPT [854:108328]
:INPUT ACCEPT [1:1460]
:OUTPUT ACCEPT [52:5086]
:POSTROUTING ACCEPT [52:5086]
:DOCKER - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_ZONES - [0:0]
:POSTROUTING_ZONES_SOURCE - [0:0]
:POSTROUTING_direct - [0:0]
:POST_internal - [0:0]
:POST_internal_allow - [0:0]
:POST_internal_deny - [0:0]
:POST_internal_log - [0:0]
:POST_public - [0:0]
:POST_public_allow - [0:0]
:POST_public_deny - [0:0]
:POST_public_log - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_internal - [0:0]
:PRE_internal_allow - [0:0]
:PRE_internal_deny - [0:0]
:PRE_internal_log - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -j OUTPUT_direct
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -j POSTROUTING_direct
-A POSTROUTING -j POSTROUTING_ZONES_SOURCE
-A POSTROUTING -j POSTROUTING_ZONES
-A POSTROUTING_ZONES -o ens32 -g POST_public
-A POSTROUTING_ZONES -o weave -g POST_internal
-A POSTROUTING_ZONES -o docker -g POST_internal
-A POSTROUTING_ZONES -g POST_public
-A POST_internal -j POST_internal_log
-A POST_internal -j POST_internal_deny
-A POST_internal -j POST_internal_allow
-A POST_public -j POST_public_log
-A POST_public -j POST_public_deny
-A POST_public -j POST_public_allow
-A PREROUTING_ZONES -i ens32 -g PRE_public
-A PREROUTING_ZONES -i weave -g PRE_internal
-A PREROUTING_ZONES -i docker -g PRE_internal
-A PREROUTING_ZONES -g PRE_public
-A PRE_internal -j PRE_internal_log
-A PRE_internal -j PRE_internal_deny
-A PRE_internal -j PRE_internal_allow
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
Completed on Wed Apr 20 16:42:34 2016
Generated by iptables-save v1.4.21 on Wed Apr 20 16:42:34 2016
*mangle
:PREROUTING ACCEPT [3682:1393387]
:INPUT ACCEPT [2893:1305495]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2631:1320416]
:POSTROUTING ACCEPT [2631:1320416]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_internal - [0:0]
:PRE_internal_allow - [0:0]
:PRE_internal_deny - [0:0]
:PRE_internal_log - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -j POSTROUTING_direct
-A PREROUTING_ZONES -i ens32 -g PRE_public
-A PREROUTING_ZONES -i weave -g PRE_internal
-A PREROUTING_ZONES -i docker -g PRE_internal
-A PREROUTING_ZONES -g PRE_public
-A PRE_internal -j PRE_internal_log
-A PRE_internal -j PRE_internal_deny
-A PRE_internal -j PRE_internal_allow
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
Completed on Wed Apr 20 16:42:34 2016
Generated by iptables-save v1.4.21 on Wed Apr 20 16:42:34 2016
*security
:INPUT ACCEPT [2829:1286519]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2631:1320416]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
COMMIT
Completed on Wed Apr 20 16:42:34 2016
Generated by iptables-save v1.4.21 on Wed Apr 20 16:42:34 2016
*raw
:PREROUTING ACCEPT [3682:1393387]
:OUTPUT ACCEPT [2631:1320416]
:OUTPUT_direct - [0:0]
:PREROUTING_direct - [0:0]
-A PREROUTING -j PREROUTING_direct
-A OUTPUT -j OUTPUT_direct
COMMIT
Completed on Wed Apr 20 16:42:34 2016
Generated by iptables-save v1.4.21 on Wed Apr 20 16:42:34 2016
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2158:1249707]
:DOCKER - [0:0]
:FORWARD_IN_ZONES - [0:0]
:FORWARD_IN_ZONES_SOURCE - [0:0]
:FORWARD_OUT_ZONES - [0:0]
:FORWARD_OUT_ZONES_SOURCE - [0:0]
:FORWARD_direct - [0:0]
:FWDI_internal - [0:0]
:FWDI_internal_allow - [0:0]
:FWDI_internal_deny - [0:0]
:FWDI_internal_log - [0:0]
:FWDI_public - [0:0]
:FWDI_public_allow - [0:0]
:FWDI_public_deny - [0:0]
:FWDI_public_log - [0:0]
:FWDO_internal - [0:0]
:FWDO_internal_allow - [0:0]
:FWDO_internal_deny - [0:0]
:FWDO_internal_log - [0:0]
:FWDO_public - [0:0]
:FWDO_public_allow - [0:0]
:FWDO_public_deny - [0:0]
:FWDO_public_log - [0:0]
:INPUT_ZONES - [0:0]
:INPUT_ZONES_SOURCE - [0:0]
:INPUT_direct - [0:0]
:IN_internal - [0:0]
:IN_internal_allow - [0:0]
:IN_internal_deny - [0:0]
:IN_internal_log - [0:0]
:IN_public - [0:0]
:IN_public_allow - [0:0]
:IN_public_deny - [0:0]
:IN_public_log - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -p icmp -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct
-A FORWARD -j FORWARD_IN_ZONES_SOURCE
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD -j FORWARD_OUT_ZONES_SOURCE
-A FORWARD -j FORWARD_OUT_ZONES
-A FORWARD -p icmp -j ACCEPT
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -j OUTPUT_direct
-A FORWARD_IN_ZONES -i ens32 -g FWDI_public
-A FORWARD_IN_ZONES -i weave -g FWDI_internal
-A FORWARD_IN_ZONES -i docker -g FWDI_internal
-A FORWARD_IN_ZONES -g FWDI_public
-A FORWARD_OUT_ZONES -o ens32 -g FWDO_public
-A FORWARD_OUT_ZONES -o weave -g FWDO_internal
-A FORWARD_OUT_ZONES -o docker -g FWDO_internal
-A FORWARD_OUT_ZONES -g FWDO_public
-A FWDI_internal -j FWDI_internal_log
-A FWDI_internal -j FWDI_internal_deny
-A FWDI_internal -j FWDI_internal_allow
-A FWDI_public -j FWDI_public_log
-A FWDI_public -j FWDI_public_deny
-A FWDI_public -j FWDI_public_allow
-A FWDO_internal -j FWDO_internal_log
-A FWDO_internal -j FWDO_internal_deny
-A FWDO_internal -j FWDO_internal_allow
-A FWDO_public -j FWDO_public_log
-A FWDO_public -j FWDO_public_deny
-A FWDO_public -j FWDO_public_allow
-A INPUT_ZONES -i ens32 -g IN_public
-A INPUT_ZONES -i weave -g IN_internal
-A INPUT_ZONES -i docker -g IN_internal
-A INPUT_ZONES -g IN_public
-A IN_internal -j IN_internal_log
-A IN_internal -j IN_internal_deny
-A IN_internal -j IN_internal_allow
-A IN_internal_allow -d 224.0.0.251/32 -p udp -m udp --dport 5353 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p udp -m udp --dport 631 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p tcp -m tcp --dport 53 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p udp -m udp --dport 53 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p udp -m udp --dport 137 -m conntrack --ctstate NEW -j ACCEPT
-A IN_internal_allow -p udp -m udp --dport 138 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public -j IN_public_log
-A IN_public -j IN_public_deny
-A IN_public -j IN_public_allow
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 6784 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 6783 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p udp -m udp --dport 6784 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 53 -m conntrack --ctstate NEW -j ACCEPT
-A IN_public_allow -p udp -m udp --dport 6783 -m conntrack --ctstate NEW -j ACCEPT
COMMIT
Completed on Wed Apr 20 16:42:34 2016
joserivca
on 20 Apr 2016
still appears that message
@dcowden wrote:
when you launch weave, you will still get the message complaining about the reject rule. But in fact, it will still work because firewall-cmd has inserted a rule that will allow the DNS traffic.
rade
on 20 Apr 2016
one note-- even in my
I dont know iptables well enough to really dive through your entire list above..
dcowden
on 20 Apr 2016
@dcowden @rade , i know about the message , thats why im tried that solution,
and i can ping between containers in different host , but just with the ip container, but when im using the name of container is not working
for example
i making this
[root@pmbrklnxd02 S56490]# /database/weave/weave status dns
hdfs-namenode-main 10.32.0.1 fa2a7a7225ef 96:5b:5c:68:7c:31
hdfs-namenode-passive 10.40.0.0 19437347f412 22:d1:4b:bd:77:30
[root@hdfs-namenode-main /]# telnet 10.40.0.0 22
Trying 10.40.0.0...
Connected to 10.40.0.0.
Escape character is '^]'.
SSH-2.0-OpenSSH_6.6.1
^[
Protocol mismatch.
Connection closed by foreign host.
[root@hdfs-namenode-main /]# telnet hdf-namenode-passive 22
telnet: hdf-namenode-passive: Name or service not known
hdf-namenode-passive: Host name lookup failure
joserivca
on 21 Apr 2016
Note in https://github.com/weaveworks/weave/issues/1266#issuecomment-198005108 the interface docker should be replaced by whatever your Docker bridge is named.
You can find this name by running:
docker network inspect -f '{{index .Options "com.docker.network.bridge.name"}}' bridge
On many systems the name is docker0
bboreham
on 21 Apr 2016
OK, really thanks for the help , but my solution was put this on iptables
/sbin/iptables -D FORWARD -j REJECT --reject-with icmp-host-prohibited
/sbin/iptables -D INPUT -j REJECT --reject-with icmp-host-prohibited
iptables-save
joserivca
on 25 Apr 2016
Related issues
bergtwvd
路
11Comments
jeroenjacobs79
路
7Comments
cwhenderson20
路
10Comments
rachit1arora
路
7Comments
brb
路
3Comments