Hi.
In https://docs.docker.com/v1.6/installation/centos/#installing-docker-centos-7 it is documented that docker can be run together with firewalld if you respect the order in which services are started, however I seem to have problems getting that to run. As far as the documentation says I would assume that the following would result in a clean state.
[root@App1 ~]# systemctl stop docker
[root@App1 ~]# systemctl stop firewalld
[root@App1 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain DOCKER (1 references)
target prot opt source destination
[root@App1 ~]# systemctl start firewalld
[root@App1 ~]# systemctl start docker
Everything went fine up to now, I also can run containers, they have network connectivity etc.. The thing which does not seem to work is inter container communication. While debugging this, I came across the following lines in /var/log/firewalld, which are a direct result of the [root@App1 ~]# systemctl start docker command above:
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory
Try `iptables -h' or 'iptables --help' for more information.
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory
Try `iptables -h' or 'iptables --help' for more information.
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory
Try `iptables -h' or 'iptables --help' for more information.
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -F DOCKER' failed: iptables: No chain/target/match by that name.
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -X DOCKER' failed: iptables: No chain/target/match by that name.
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C POSTROUTING -s 172.17.42.1/16 ! -o docker0 -j MASQUERADE' failed: iptables: No chain/target/match by that name.
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -n -L DOCKER' failed: iptables: No chain/target/match by that name.
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed: iptables: No chain/target/match by that name.
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -n -L DOCKER' failed: iptables: No chain/target/match by that name.
2015-09-08 07:27:13 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -o docker0 -j DOCKER' failed: iptables: No chain/target/match by that name.
My current thesis is, that a running firewalld causes a kind of a race condition which causes that the docker chain is not yet existing when docker already tries to add rules to it. When researching that issue I found other people having trouble with CentOS 7 and Docker, but no one so far described that behavior.
docker version:
Client version: 1.7.1
Client API version: 1.19
Package Version (client): docker-1.7.1-108.el7.centos.x86_64
Go version (client): go1.4.2
Git commit (client): 3043001/1.7.1
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Package Version (server): docker-1.7.1-108.el7.centos.x86_64
Go version (server): go1.4.2
Git commit (server): 3043001/1.7.1
OS/Arch (server): linux/amd64
docker info:
Containers: 8
Images: 155
Storage Driver: devicemapper
Pool Name: docker-253:0-3146366-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 4.95 GB
Data Space Total: 107.4 GB
Data Space Available: 40.16 GB
Metadata Space Used: 9.466 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.138 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.11.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 12
Total Memory: 31.22 GiB
Name: App1
ID: SHSS:NWHQ:V3C7:KUCO:VUH6:4VH3:AOYV:N3TN:YTLI:7M5F:OKSZ:6FWY
uname -a
Linux USCL-HDApp1 3.10.0-229.11.1.el7.x86_64 #1 SMP Thu Aug 6 01:06:18 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
stephanadler
All 96 comments
ping @mavenugo has this come up before i think it may be a dup idk
jessfraz
on 8 Sep 2015
@stephanadler
@mavenugo
I have the same/similar issues as mentioned by @stephanadler .
Attempted both with docker upstream package (docker-engine 1.8.1) and CentOS provided packages (docker 1.7.1) on CentOS 7.1 - I see unpredictable behavior with docker-engine/docker client startup. This happens consistently with firewalld running and sporadically with firewalld stopped/disabled. I am happy to provide addition details, but I do not want to clutter this post.
smuthali
on 8 Sep 2015
The same issue exists (unsurprisingly) on Oracle Linux 7 as well.
Djelibeybi
on 8 Sep 2015
i had the same issue , Through this article solve this problem (the article Chinese not English,who can translate that )
http://www.lxy520.net/2015/09/24/centos-7-docker-qi-dong-bao/
ystyle
on 24 Sep 2015
@stephanadler I'm having this exact issue, was there any solution to this?
fud
on 29 Sep 2015
@fud right now I am running the system with disabled firewalld. That's quite ugly, but up to now I tend to think this is a real bug and therefore everything which can be used as workarround is somewhat ugly. The chinese article posted by @ystyle seems to indicate to let the firewall management (in our case firewalld) create the DOCKER chain. I did not yet play arround with that option, probably will in the next days.
stephanadler
on 29 Sep 2015
@fud @stephanadler - even with firewalld disabled I have trouble with docker-engine starting. Apparently docker-engine 1.9 will address this issue.
smuthali
on 29 Sep 2015
@stefanberger @smuthali Thanks, I will await for a solution from docker.
fud
on 30 Sep 2015
This is still broken in CentOS 7
[vagrant@myprecise ~]$ uname -a
Linux myprecise.box 3.10.0-123.el7.x86_64 #1 SMP Mon Jun 30 12:09:22 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
[vagrant@myprecise ~]$ sudo docker version
Client:
Version: 1.8.2
API version: 1.20
Package Version: docker-1.8.2-7.el7.centos.x86_64
Go version: go1.4.2
Git commit: bb472f0/1.8.2
Built:
OS/Arch: linux/amd64
Server:
Version: 1.8.2
API version: 1.20
Package Version:
Go version: go1.4.2
Git commit: bb472f0/1.8.2
Built:
OS/Arch: linux/amd64
[vagrant@myprecise ~]$ sudo service firewalld status -l
Redirecting to /bin/systemctl status -l firewalld.service
firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
Active: active (running) since Mon 2015-11-09 16:20:28 UTC; 2min 15s ago
Main PID: 22098 (firewalld)
CGroup: /system.slice/firewalld.service
└─22098 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
Nov 09 16:20:39 myprecise.box firewalld[22098]: 2015-11-09 16:20:39 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C POSTROUTING -s 172.17.42.1/16 ! -o docker0 -j MASQUERADE' failed: iptables: No chain/target/match by that name.
Nov 09 16:20:39 myprecise.box firewalld[22098]: 2015-11-09 16:20:39 ERROR: COMMAND_FAILED: '/sbin/iptables -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 09 16:20:39 myprecise.box firewalld[22098]: 2015-11-09 16:20:39 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 09 16:20:39 myprecise.box firewalld[22098]: 2015-11-09 16:20:39 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 09 16:20:39 myprecise.box firewalld[22098]: 2015-11-09 16:20:39 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 09 16:20:39 myprecise.box firewalld[22098]: 2015-11-09 16:20:39 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -n -L DOCKER' failed: iptables: No chain/target/match by that name.
Nov 09 16:20:39 myprecise.box firewalld[22098]: 2015-11-09 16:20:39 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
Nov 09 16:20:39 myprecise.box firewalld[22098]: 2015-11-09 16:20:39 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed: iptables: No chain/target/match by that name.
Nov 09 16:20:39 myprecise.box firewalld[22098]: 2015-11-09 16:20:39 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -n -L DOCKER' failed: iptables: No chain/target/match by that name.
Nov 09 16:20:39 myprecise.box firewalld[22098]: 2015-11-09 16:20:39 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -o docker0 -j DOCKER' failed: iptables: No chain/target/match by that name.
AndreaGiardini
on 9 Nov 2015
@AndreaGiardini have you tested with docker 1.9 as well?
thaJeztah
on 11 Nov 2015
@thaJeztah I just installed a fresh CentOS 7 and installed the current docker release:
[root@localhost ~]# uname -a
Linux localhost.localdomain 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Sep 15 15:05:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# docker version
Client:
Version: 1.9.0
API version: 1.21
Go version: go1.4.2
Git commit: 76d6bc9
Built: Tue Nov 3 18:00:05 UTC 2015
OS/Arch: linux/amd64
Server:
Version: 1.9.0
API version: 1.21
Go version: go1.4.2
Git commit: 76d6bc9
Built: Tue Nov 3 18:00:05 UTC 2015
OS/Arch: linux/amd64
Nov 12 10:03:07 localhost.localdomain firewalld[572]: 2015-11-12 10:03:07 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -n -L DOCKER' failed: iptables: No chain/target/match by that name.
Nov 12 10:03:07 localhost.localdomain firewalld[572]: 2015-11-12 10:03:07 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -n -L DOCKER' failed: iptables: No chain/target/match by that name.
Nov 12 10:03:07 localhost.localdomain firewalld[572]: 2015-11-12 10:03:07 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE' failed: iptables: No chain/target/match by that name.
Nov 12 10:03:07 localhost.localdomain firewalld[572]: 2015-11-12 10:03:07 ERROR: COMMAND_FAILED: '/sbin/iptables -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 12 10:03:07 localhost.localdomain firewalld[572]: 2015-11-12 10:03:07 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 12 10:03:07 localhost.localdomain firewalld[572]: 2015-11-12 10:03:07 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 12 10:03:07 localhost.localdomain firewalld[572]: 2015-11-12 10:03:07 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Nov 12 10:03:07 localhost.localdomain firewalld[572]: 2015-11-12 10:03:07 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
Nov 12 10:03:07 localhost.localdomain firewalld[572]: 2015-11-12 10:03:07 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed: iptables: No chain/target/match by that name.
Nov 12 10:03:07 localhost.localdomain firewalld[572]: 2015-11-12 10:03:07 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -o docker0 -j DOCKER' failed: iptables: No chain/target/match by that name.
I confirm as well
AndreaGiardini
on 12 Nov 2015
[vagrant@myprecise yum.repos.d]$ sudo service firewalld status
Redirecting to /bin/systemctl status firewalld.service
firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
Active: active (running) since Mon 2015-11-16 14:57:31 UTC; 1s ago
Main PID: 9963 (firewalld)
CGroup: /system.slice/firewalld.service
├─ 9963 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
└─10319 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
Nov 16 14:57:28 myprecise.box systemd[1]: Starting firewalld - dynamic firewall daemon...
Nov 16 14:57:31 myprecise.box systemd[1]: Started firewalld - dynamic firewall daemon.
[vagrant@myprecise yum.repos.d]$ sudo service docker start
Redirecting to /bin/systemctl start docker.service
[vagrant@myprecise yum.repos.d]$ sudo service firewalld status
Redirecting to /bin/systemctl status firewalld.service
firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
Active: active (running) since Mon 2015-11-16 14:57:31 UTC; 8s ago
Main PID: 9963 (firewalld)
CGroup: /system.slice/firewalld.service
└─9963 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
Nov 16 14:57:38 myprecise.box firewalld[9963]: 2015-11-16 14:57:38 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -n -L DOCKER' failed: iptables: No chain/tar...that name.
Nov 16 14:57:38 myprecise.box firewalld[9963]: 2015-11-16 14:57:38 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -n -L DOCKER' failed: iptables: No chain/...that name.
Nov 16 14:57:38 myprecise.box firewalld[9963]: 2015-11-16 14:57:38 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0...that name.
Nov 16 14:57:38 myprecise.box firewalld[9963]: 2015-11-16 14:57:38 ERROR: COMMAND_FAILED: '/sbin/iptables -D FORWARD -i docker0 -o docker0 -j DROP' failed: i...t chain?).
Nov 16 14:57:38 myprecise.box firewalld[9963]: 2015-11-16 14:57:38 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -i docker0 -o docker0 -j ACCEP...t chain?).
Nov 16 14:57:38 myprecise.box firewalld[9963]: 2015-11-16 14:57:38 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -i docker0 ! -o docker0 -j ACC...t chain?).
Nov 16 14:57:38 myprecise.box firewalld[9963]: 2015-11-16 14:57:38 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -o docker0 -m conntrack --ctst...t chain?).
Nov 16 14:57:38 myprecise.box firewalld[9963]: 2015-11-16 14:57:38 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C PREROUTING -m addrtype --dst-type LOCAL -...that name.
Nov 16 14:57:38 myprecise.box firewalld[9963]: 2015-11-16 14:57:38 ERROR: COMMAND_FAILED: '/sbin/iptables -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DO...that name.
Nov 16 14:57:38 myprecise.box firewalld[9963]: 2015-11-16 14:57:38 ERROR: COMMAND_FAILED: '/sbin/iptables -t filter -C FORWARD -o docker0 -j DOCKER' failed: ...that name.
Hint: Some lines were ellipsized, use -l to show in full.
[vagrant@myprecise yum.repos.d]$ sudo service docker status
Redirecting to /bin/systemctl status docker.service
docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled)
Active: active (running) since Mon 2015-11-16 14:57:38 UTC; 19s ago
Docs: https://docs.docker.com
Main PID: 10356 (docker)
CGroup: /system.slice/docker.service
└─10356 /usr/bin/docker daemon -H fd://
Nov 16 14:57:36 myprecise.box docker[10356]: time="2015-11-16T14:57:36.744523806Z" level=warning msg="Usage of loopback devices is strongly discouraged for p... section."
Nov 16 14:57:37 myprecise.box docker[10356]: time="2015-11-16T14:57:37.062164989Z" level=info msg="API listen on /var/run/docker.sock"
Nov 16 14:57:37 myprecise.box docker[10356]: time="2015-11-16T14:57:37.103601717Z" level=info msg="[graphdriver] using prior storage driver \"devicemapper\""
Nov 16 14:57:37 myprecise.box docker[10356]: time="2015-11-16T14:57:37.137470288Z" level=info msg="Firewalld running: true"
Nov 16 14:57:38 myprecise.box docker[10356]: time="2015-11-16T14:57:38.145007406Z" level=info msg="Default bridge (docker0) is assigned with an IP address 17...P address"
Nov 16 14:57:38 myprecise.box docker[10356]: time="2015-11-16T14:57:38.304092328Z" level=info msg="Loading containers: start."
Nov 16 14:57:38 myprecise.box docker[10356]: time="2015-11-16T14:57:38.304312664Z" level=info msg="Loading containers: done."
Nov 16 14:57:38 myprecise.box docker[10356]: time="2015-11-16T14:57:38.304336576Z" level=info msg="Daemon has completed initialization"
Nov 16 14:57:38 myprecise.box docker[10356]: time="2015-11-16T14:57:38.304360310Z" level=info msg="Docker daemon" commit=76d6bc9 execdriver=native-0.2 graphd...sion=1.9.0
Nov 16 14:57:38 myprecise.box systemd[1]: Started Docker Application Container Engine.
Hint: Some lines were ellipsized, use -l to show in full.
[vagrant@myprecise yum.repos.d]$ sudo docker version
Client:
Version: 1.9.0
API version: 1.21
Go version: go1.4.2
Git commit: 76d6bc9
Built: Tue Nov 3 18:00:05 UTC 2015
OS/Arch: linux/amd64
Server:
Version: 1.9.0
API version: 1.21
Go version: go1.4.2
Git commit: 76d6bc9
Built: Tue Nov 3 18:00:05 UTC 2015
OS/Arch: linux/amd64
ping @mavenugo @mrjana could you have a look at this one? some people running into this on 1.9.0 (see the discussion above)
thaJeztah
on 16 Nov 2015
We are also seeing a similar issue with Docker on CentOS 7 with firewalld running. Intermittently we will see networking issues with container either being inaccessible or unable to communicate with external services. I will try to get more details the next time the issue arises.
InfoSec812
on 25 Nov 2015
@jfrazelle @thaJeztah we had issues with selinux package in 1.8.x which caused issues in docker to firewalld interaction. But this seems to be a different issue. It is not very clear what is the actual issue seen in 1.9.0 (other than the error messages seen in the firewalld service).
@mrjana the only thing i could think of that is different in 1.9.0 is the way we handle the docker0 bridge cleanup and restart during daemon restarts. Do you think this could cause any problems with firewalld ?
mavenugo
on 25 Nov 2015
@aboch this could be a bridge driver specific initializing issue. can you PTAL ?
mavenugo
on 25 Nov 2015
I’m not certain, but it sounds like I have the same problem as what I read about above.
I have a fresh CentOS 7 install, turned off SELinux, turned off firewalld, and when I try to start my Docker container, I get:
Error response from daemon: Cannot start container FOO: failed to create endpoint BAR on network bridge: COMMAND_FAILED: '/sbin/iptables -t nat -A DOCKER -p tcp -d 0/0 --dport 9292 -j DNAT --to-destination 172.17.0.2:9292 ! -i docker0' failed: iptables: No chain/target/match by that name.
Version:
$ docker --version
Docker version 1.9.1, build a34a1d5
If it helps, this is what iptables shows me:
$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:bootps
Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere 192.168.122.0/24 ctstate RELATED,ESTABLISHED
ACCEPT all -- 192.168.122.0/24 anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:bootpc
filethis-dev-site
on 30 Nov 2015
@filethis-dev-site
iptables-save > /etc/sysconfig/iptables
modify this file like this
*nat
:PREROUTING ACCEPT [27:11935]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [598:57368]
:POSTROUTING ACCEPT [591:57092]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
COMMIT
# Completed on Sun Sep 20 17:35:31 2015
# Generated by iptables-save v1.4.21 on Sun Sep 20 17:35:31 2015
*filter
:INPUT ACCEPT [139291:461018923]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [127386:5251162]
:DOCKER - [0:0]
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
COMMIT
# Completed on Sun Sep 20 17:35:31 2015
restart iptables
systemctl restart iptables.service
ystyle
on 30 Nov 2015
@ystyle Thank you for your solution, but I failed in last step:
[lei@tang workspace]$ sudo systemctl restart iptables.service
Failed to issue method call: Unit iptables.service failed to load: No such file or directory.
and I solved this issue by this http://stackoverflow.com/questions/24756240/how-can-i-use-iptables-on-centos-7
LeoShi
on 30 Nov 2015
Would you gents, please, mind adding a note to the documentation about how Docker + CentOS/RHEL in its default configuration is a very bad idea right now?
A lot of things suggest that el7 is a pretty good platform for a host that is supposed to do nothing apart from docker. (advertizing from RH, documentation from RH, being #2 after ubuntu in the docker documentation, RH deprecating LXC in favor of docker, the mere fact that RH "supports" docker, as it does not strive to support more than is good).
It is not a good platform.
This is reproducible on all our CentOS hosts. All our Ubuntu and Arch Linux hosts are still standing strong with docker. It's all deployed using ansible. This does cause actual issues (at the very least after a reboot without any docker autostart.)
SAKUJ0
on 1 Feb 2016
@SAKUJ0 which version of docker are you installing? The Red Hat maintained packages, or the packages from https://yum.dockerproject.org? Could you explain what you mean with "at the very least after a reboot without any docker autostart"?
thaJeztah
on 1 Feb 2016
Edit It does not appear to be a MSS issue but a Docker with CentOS 7 related issue. I will try out the CentOS package (despite people saying it would not change anything) and mess around with some more containers. But without masquerading involved, those kind of issues tend to always be firewall(read: iptables) related on the client that can't be reached (after all the entire rest of the network can be reached). Locally it works.
@thaJeztah
To answer your question: We use our (read the official docker) yum repo following the instructions from "Installing Docker" in our official documentation.
It's a bit messy. I have dockera and dockerb. dockera was running docker for now ~ 4 months or so. This Sunday, the web application had issues (potentially unrelated with docker but instead related with Fragment and MSS).
After the reboot I moved all the volumes from dockera to dockerb and re-deployed. They are symmetrical hosts, but I keep dockerb perfectly up-to-date and dockera I keep quite stagnant (it's local only).
Then I re-installed dockera (again all using ansible) and it solved the issues. Until I rebooted. What both hosts have in common are the error messages described in this thread (and it does not look like with those chains missing forwarding can work decently).
Here are the logs.
http://hastebin.com/wavobicexi.vbs
http://hastebin.com/qobopuqaqa.vbs
(it's a pastebin without all the bloat).
While going through them, I think I was dead wrong when I said "actual issues". It seems they are just error messages with no other things related and the VPN issues were just coincidence.
Though, quite reproducible for me. Host b was naked and fully updated and running a functional DNS server only for the longest amount of time. You can see the issue on host b only once. That is because I never rebooted it.
This could be a bit instructive as I could grab out the ansible playbooks used to deploy the docker and firewalld portions.
SAKUJ0
on 1 Feb 2016
Just installed Docker 1.10 (upgrade but I destroyed all the existing configuration/LVM/etc).
# docker version
Client:
Version: 1.10.0
API version: 1.22
Go version: go1.5.3
Git commit: 590d5108
Built: Thu Feb 4 18:34:50 2016
OS/Arch: linux/amd64
Server:
Version: 1.10.0
API version: 1.22
Go version: go1.5.3
Git commit: 590d5108
Built: Thu Feb 4 18:34:50 2016
OS/Arch: linux/amd64
Got the following messages from tail -f /var/log/firewalld on service docker start.
2016-02-09 12:52:02 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
2016-02-09 12:52:02 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2016-02-09 12:52:02 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2016-02-09 12:52:03 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -X DOCKER' failed: iptables: Too many links.
2016-02-09 12:52:03 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -X DOCKER-ISOLATION' failed: iptables: Too many links.
2016-02-09 12:52:03 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -n -L DOCKER' failed: iptables: No chain/target/match by that name.
2016-02-09 12:52:03 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C DOCKER-ISOLATION -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2016-02-09 12:52:03 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2016-02-09 12:52:03 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2016-02-09 12:52:03 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
2016-02-09 12:52:03 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed: iptables: No chain/target/match by that name.
I can appreciate there must be quite some work in making docker play nice with firewalld (I'm perfectly happy with iptables command myself) - but it would be nice to put a big warning on the page indicating that the two services (docker and firewalld) don't play together particularly well.
binarytemple
on 9 Feb 2016
+1 to @binarytemple's doc suggestions.
To be clear - the only functional solution (AFAIK) on CentOS now is to use docker 1.8 from CentOS extras (which still generates warnings, but firewalld works - not sure if docker networking is fully functional) OR mask firewalld and use iptables?
davclark
on 9 Feb 2016
+1 for @binarytemple's documentation
willseward
on 9 Feb 2016
We have opted to disable firewalld for all of our environments and it is
very stable.
Deven
On Feb 9, 2016 1:59 PM, "Wills Ward" [email protected] wrote:
+1 for @binarytemple https://github.com/binarytemple's documentation
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/16137#issuecomment-182004259.
InfoSec812
on 9 Feb 2016
FWIW, I'm running FirewallD on Ubuntu and I'm seeing exactly the same issue.
Disabling firewalld is not an option here as it is the mandated way of managing server firewalls across this particular infrastructure.
Can any provide me with more details on what it is that Docker is trying to do when it stands up the iptables Chain so I can try and work out a way of fixing this via firewalld?
proffalken
on 16 Feb 2016
Try this - it works for me.
greenrd
on 2 Mar 2016
That would work, but resetting your network connections is a reactive
instead of proactive fix. The problem shouldn't happen in the first place.
Deven
On Mar 2, 2016 6:42 AM, "Robin Green" [email protected] wrote:
Try this http://stackoverflow.com/a/33604859/495796 - it works for me.
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/16137#issuecomment-191202042.
InfoSec812
on 2 Mar 2016
Yes. In my experience I've seen a lot (at least 6) of these sort of high level firewall abstractions come and go over the last 10 years or so. Learning the syntax/configuration just shoehorns you into some particular distribution. My personal advice would be to just bite the bullet and learn iptables command syntax. It will stand you in much better stead in the long run. +1 to @InfoSec812 - resetting your network connections is not a durable solution to the issue - better to remove firewalld (strikes me it's tailored towards desktop users anyway) and create your own iptables rules file.
binarytemple
on 2 Mar 2016
Oh, we've already done that on all of our Docker hosts. We updated our
Puppet config to disable/remove firewalld and enable iptables/ipmasq
capabilities in Docker's daemon.
On Wed, Mar 2, 2016 at 10:22 AM, Bryan Hunt [email protected]
wrote:
Yes. In my experience I've seen a lot (at least 6) of these sort of high
level firewall abstractions come and go over the last 10 years or so.
Learning the syntax/configuration just shoehorns you into some particular
distribution. My personal advice would be to just bite the bullet and learn
iptables command syntax. It will stand you in much better stead in the long
run. +1 to @InfoSec812 https://github.com/InfoSec812 - resetting your
network connections is not a durable solution to the issue - better to
remove firewalld (strikes me it's tailored towards desktop users anyway)
and create your own iptables rules file.—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/16137#issuecomment-191284603.
InfoSec812
on 2 Mar 2016
@binarytemple I completely agree with you and @InfoSec812 on this, however, the easiest way to manage firewall rules via Ansible across multiple customers and multiple distribution families is to install one of these high-level firewall rule generators and have it as a common platform.
Centos does not support UFW, so Firewalld it is... :(
I'm hoping I might get time in the next few weeks to solve this, as it's being driven by a particular client need at work at the moment.
proffalken
on 2 Mar 2016
FWIW, I now don't believe this is a docker issue, I believe it's a flaw in the way that firewalld doesn't let you add new chains, but I'm happy to be corrected :)
proffalken
on 2 Mar 2016
Disabling Docker's iptables management capability and have firewalld manage masquerading and docker-proxy for port forwarding also works atleast when you're not trying to do anything fancy such as overlay networking.
ghost
on 6 Mar 2016
Today also hit this problem, with version:
Client:
Version: 1.11.1
API version: 1.23
Go version: go1.5.4
Git commit: 5604cbe
Built: Wed Apr 27 00:34:42 2016
OS/Arch: linux/amd64
Server:
Version: 1.11.1
API version: 1.23
Go version: go1.5.4
Git commit: 5604cbe
Built: Wed Apr 27 00:34:42 2016
OS/Arch: linux/amd64
Disabled firewalld for now...rather would have it running though...
jortkoopmans
on 14 May 2016
We can't disable firewalld in our environment, so this is a show-stopper issue for us.
@UserTaken Can you explain how to disable Docker's iptables management capability? We are only using bridge networks.
ToBeReplaced
on 18 May 2016
@ToBeReplaced Running Docker daemon with --iptables=false will not insert any iptables rules.
ghost
on 19 May 2016
@UserTaken what's the impact on Docker when using this flag from an Operations point of view? Does it mean that all firewall rules (between swarm, containers and docker networks etc) need to be added by the configuration management tooling (or manually :( ) instead?
proffalken
on 19 May 2016
guys, i found out when i execute "firewall-cmd --add-forward-port" then i got the same error, i must restart docker daemon to resume. what should i do ? stop using firewall is not an option.
luost
on 27 May 2016
Guys... seriously you need to learn iptables and integrate manually. Also,
it's madness to directly expose a machine running Docker engine directly to
the internet. I know you're all looking for a technical solution on a plate
but this is a problem that requires an organisational/operational solution
from the user. The best the Docker guys could ever do is formalise the
required ruleset and extension points.
On Fri, 27 May 2016, 14:16 Luo SongTao, [email protected] wrote:
guys, i found out when i execute "firewall-cmd --add-forward-port" then i
got the same error, i must restart docker daemon to resume. what should i
do ? stop using firewall is not an option.—
You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub
https://github.com/docker/docker/issues/16137#issuecomment-222142859,
or mute the thread
https://github.com/notifications/unsubscribe/AAZk5lBKvQskFLx093nfnqMksGbTZthuks5qFu6egaJpZM4F5b7M
.
binarytemple
on 27 May 2016
@binarytemple I agree to an extent that this is not a Docker problem, however, it would be nice to see the docker community working with the FirewallD community to find a solution.
It's true that on most platforms IP Tables is the default firewall, however on all RHEL variants FirewallD is rapidly taking over.
Unlike UFW on Ubuntu, FirewallD is _not_ a way of controlling IP Tables (although it can do that) and the advice from the community is _do not_ run IP Tables and FirewallD on the same host.
proffalken
on 27 May 2016
I "fixed" this on our centos7 server by adding these to lines to the [Unit] section of/usr/lib/systemd/system/docker-latest.service:
After=firewalld.service
PartOf=firewalld.service
This should assure that docker gets restartet everytime firewalld starts.
I'm afraid that this does not solve the issue when firewalld --reload. The user would need to restart docker manually in that case.
This is not a great or the correct solution (imho, docker should talk to firewalld when its used instead of using iptables directly) but its an easy work around for the time beeing.
themasch
on 14 Jun 2016
I'm getting this error in CentOS 7
[root@devlinux linux]# uname -a
Linux devlinux.medbpro.net 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 23 17:05:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@devlinux linux]# docker version
Client:
Version: 1.10.3
API version: 1.22
Package version: docker-common-1.10.3-44.el7.centos.x86_64
Go version: go1.4.2
Git commit: 9419b24-unsupported
Built: Fri Jun 24 12:09:49 2016
OS/Arch: linux/amd64
Server:
Version: 1.10.3
API version: 1.22
Package version: docker-common-1.10.3-44.el7.centos.x86_64
Go version: go1.4.2
Git commit: 9419b24-unsupported
Built: Fri Jun 24 12:09:49 2016
OS/Arch: linux/amd64
[root@devlinux linux]# service firewalld status -l
Redirecting to /bin/systemctl status -l firewalld.service
â—Ź firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2016-07-05 09:36:04 IST; 1h 27min ago
Main PID: 734 (firewalld)
CGroup: /system.slice/firewalld.service
└─734 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
Jul 05 09:56:49 devlinux.localhost.net firewalld[734]: 2016-07-05 09:56:49 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Jul 05 09:56:49 devlinux.localhost.net firewalld[734]: 2016-07-05 09:56:49 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Jul 05 09:56:49 devlinux.localhost.net firewalld[734]: 2016-07-05 09:56:49 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -X DOCKER' failed: iptables: Too many links.
Jul 05 09:56:49 devlinux.localhost.net firewalld[734]: 2016-07-05 09:56:49 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -X DOCKER-ISOLATION' failed: iptables: Too many links.
Jul 05 09:56:49 devlinux.localhost.net firewalld[734]: 2016-07-05 09:56:49 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -n -L DOCKER' failed: iptables: No chain/target/match by that name.
Jul 05 09:56:49 devlinux.localhost.net firewalld[734]: 2016-07-05 09:56:49 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C DOCKER-ISOLATION -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Jul 05 09:56:49 devlinux.localhost.net firewalld[734]: 2016-07-05 09:56:49 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Jul 05 09:56:49 devlinux.localhost.net firewalld[734]: 2016-07-05 09:56:49 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Jul 05 09:56:49 devlinux.localhost.net firewalld[734]: 2016-07-05 09:56:49 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
Jul 05 09:56:49 devlinux.localhost.net firewalld[734]: 2016-07-05 09:56:49 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed: iptables: No chain/target/match by that name.
bkmagnetron
on 5 Jul 2016
i'm having the similar issue like described above (CentOS Linux release 7.2.1511 (Core), Docker Server Version: 1.10.3 coming from CentOS extras repository).
Docker as such DOES work(!) but the firewall is not 100% working as expected :/
below are my comments, but please correct me if i'm wrong :)
AFAIK Firewalld is using the iptables command itself but not the iptables-service - correct?
So installing the iptables-service (same as using the _iptables-save_) on a system with active firewalld is no good idea !
As well: Deactivating firewalld is no option (for me:)
I'm now wondering the following: Where do these rules (which fail during docker daemon startup) come from at all ? Are these hardcoded inside the docker code ???
I wasn't able to determine any configuration file which may hold such rules !?
If these rules really do come from docker side, then IMHO it's up to the docker people to find out what they're doing wrong when using firewalld, ... and come up with a solution (e.g. under cooperation/support with/of the firewalld guys(?))
I mean: for now I can simply live with the fact that these errors appear in the systems log - and that the firewall isn't 100% working as expected. At least for the major aspects I have, it luckily does ...
And for now I'm not in a production system as such (and anyway behind a corporate firewall)
one last word: I find it unbelievable that this problem exists that long and still nobody has come up with a reasonable solution - but instead only pinpointing about responsibility(?)
yogitcs
on 6 Jul 2016
another thing I just came accross:
if you start the daemon using the '_--iptabels=false_' option, then obviously the service start doesn't try to insert the iptables rules anymore !?!
seems to me a clear sign of the fact that the "offending" rules come from docker side - and are hard-coded (?).
if so: why not put that stuff into a config-file ?
thus, ...
i - so far - haven't checked yet completely for the implications of using this option.
but if this is the "solution" I have been looking for, then there's a clear question coming to the docker-community: obviously the docker can determine on startup the OS flavour it's running upon.
thus why not use the '_--iptabels=false_' per default on the CentOS (and alike)?
this doesn't seem to be documented ???
like said: I don't know about possible implications coming from the option so far ...
yogitcs
on 6 Jul 2016
The "issue" with iptables=false is that exposing ports doesn't work anymore because thats done via iptables. We are currently running with iptables=false & using nginx as a reverse proxy (what we do anyway). For other rules are configured manually using firewalld...
themasch
on 6 Jul 2016
okay, ... :|
_"exposing ports"_, what exactly you mean with this ? you're talking about the accessibility of a port of a docker-container from the outside world ? that's where one should anyway proxy the connection (e.g. with nginx - like you mention)
and: if really a direct port exposure without proxy would be required: this could anyway be done by "firewall-cmd --zone=
but if that's the only implication, ... one could possibly live with this :)
if all this is true: YES, use the '--iptables=false' option per default if you detect firewalld ! state it in the documentation (and e.g. in the log :)
[EDIT] using this option is NO option !!! see post below ...
that's IMO way better than issuing the errors about the failure of application of iptables rules ...
in the end you could as well (on detection of firewalld) issue the compatible firewall-cmd commands instead - couldn't you ?
still my "complaint" remains about the hard-coding of such rules ... or do you disagree on that ?
yogitcs
on 6 Jul 2016
@yogitcs sry. I used the wrong terminology. Docker calls it "publish". Its what the "-p" parameter for docker create does.
AFAIK this parameter creates a port forwarding rule using iptables for the given ports.
and: if really a direct port exposure without proxy would be required: this could anyway be done by "firewall-cmd --zone=
--add-forward-port=port= :proto=tcp:toport= "
Thats pretty much exactly what we are doing now ;).
On your last question: I don't know how much effort it would be to implement a firewall-cmd handler next to the current iptables implementation. But I guess anyone who wants to use docker in production should manage their firewall/port-forwarding manually anyway so using --iptables=false in production seems reasonable to me.
themasch
on 6 Jul 2016
if i recall correctly the publishing of the container ports specified with "-p" seems to have been working even with '--iptables=false', but i'll have to double check that again ... just not now as i'm scratching all to reproduce another issue of mine. then restart, rebuild images, containers, etc.
=> i'll come back to that (probably only tomorrow)
[...] so using --iptables=false in production seems reasonable to me.
YES, to me too :D [EDIT] it is NOT ! => see below !
although, ... i can't remember to have found a corresponding remark in the documentation so far.
but this could be due to the fact that i may belong to the population of _"the blind and the (nearly) old"_ who isn't good at reading fineprint anymore :P
thus: if such a remark is already in, then okay. if not: it should be added.
about the effort to implement a "firewall-cmd handler": one would "just" need to translate the iptables rules into the corresponding firewall-cmd commands. so far i have no idea ... but - if familiar with firewall-cmd - this possibly shouldn't bee too tricky (my assumption, while only having _"dangerous half-knowledge"_ expertise on firewalld so far :)
yogitcs
on 6 Jul 2016
It's worse than expected
observations when building images:
after a system reboot ... if the docker daemon is started with '--iptables==false' the containers can NOT connect to the outside world - as to be expected.
=> so my previous statement was false due to fact 'II.' (see below)
facts:
I: (after a fresh boot) starting docker.service _without_ the option resembles to
a) you get the errors about failed iptables commands
b) still containers CAN connect to the outside world.
(e.g. run an apt-get update|install and similar)
II: stopping the docker service (after it once has been started according to 'I.') and starting it again WITH the option resembles to
c) you get no errors about failed iptables commands anymore
d) containers CAN connect to the outside world (!?!?)
III: (after a fresh boot) starting docker.service WITH the option resembles to
e) you get no errors about failed iptables commands
f) containers can NOT connect to the outside world. (as to be expected)
this leads to the conclusion that (at the moment) in the firewalld context
1st) docker can not handle firewalld in a 100% reliable way at all !
2nd) obviously some of the hard-coded docker-iptables "magic" succeeds but not all (?)
3rd) the "magic" doesn't get removed on service shutdown, but persists
so there is no real compatibility of docker with CentOS (and the likes) while firewalld is used then ... :|
... although (as far as i have yet tested) the firewall seems to work as expected - but still no reliability in terms of services need to be restarted, etc.
there seems to be some work pending !!!
yogitcs
on 8 Jul 2016
ping @aboch ^^
thaJeztah
on 8 Jul 2016
If iptables is installed after the docker, you can try reInstalled the docker
ystyle
on 12 Jul 2016
@ystyle: what you try to say with this ???
the talk here is about using docker in a (semi-)production context. so any kind of "reinstallation" is out of question! ... same as using the iptables service (if one wants/needs to use firewalld).
if the docker-community wants to keep up the claim of being compatible to CentOS and the like, the firewalld issue needs to be resolved in a reasonable way - e.g. that the restart of services doesn't lead to an undetermined state, that no error messages occur on service start, ... and so on !
yogitcs
on 13 Jul 2016
FYI, regarding the [...] failed: iptables: No chain/target/match by that name. error, when due to firewalld reload, docker 1.11.0 has a fix which would allow the docker chains to be recreated when firewalld reloads: https://github.com/docker/libnetwork/pull/947/commits/dbc253f732f8f1d90306e4128fb05bdc59f61544
For the people who were hitting this issue on firewalld reload running older docker version, I think it is worth to try 1.11.2 and see if the problem is gone.
Thanks.
aboch
on 13 Jul 2016
FYI, since somebody had questions about docker/firewalld interaction, at boot docker checks whether firewalld is running. If so, it will defer to firewalld the iptables manipulation.
See https://github.com/docker/libnetwork/blob/v0.8.0-dev.2/iptables/iptables.go#L351
The firewalld support was added by: https://github.com/docker/docker/pull/9397
aboch
on 13 Jul 2016
the question is: how do the packages provided from the 'docker-repo' (so, from https://yum.dockerproject.org/repo/main/centos/7) cooperate/coexist with the ones provided by the CentOS 'extra' repository ?
the CentOS guys seem to have done quite some repackaging ... obviously (more/other/additional rpms)
e.g. they have no 'docker-engine' package anymore, etc.
it's a bit of an effort to fiddle the things apart (i may give this a try - as soon as i find some time here for - but i don't want to break my currently running _"semi-production"_ environment :)
probably one would have to remove the CentOS-provided packages first (?)
anyone who already has experience with this ?
yogitcs
on 14 Jul 2016
Having installed the 1.12rc4 via https://yum.dockerproject.org/repo/experimental/fedora/24 , seems like docker vs. firewalld conflict still has not been solved on latest release.
Since firewalld is now default package on red hat distribution, any chances to fix this issue ?
docker version
Client:
Version: 1.12.0-rc4
API version: 1.24
Go version: go1.6.2
Git commit: e4a0dbc
Built:
OS/Arch: linux/amd64
Experimental: true
Server:
Version: 1.12.0-rc4
API version: 1.24
Go version: go1.6.2
Git commit: e4a0dbc
Built:
OS/Arch: linux/amd64
Experimental: true
cat /etc/redhat-release
Fedora release 24 (Twenty Four)
systemctl status firewalld
â—Ź firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2016-07-13 11:10:53 CST; 1 day 22h ago
Main PID: 718 (firewalld)
Tasks: 2 (limit: 512)
Memory: 9.1M
CPU: 5.029s
CGroup: /system.slice/firewalld.service
└─718 /usr/bin/python3 -Es /usr/sbin/firewalld --nofork --nopid
Jul 15 09:59:34 /firewalld[718]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w -t nat -C DOCKER -i docker0 -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Jul 15 09:59:34 /firewalld[718]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Jul 15 09:59:34 /firewalld[718]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that
Jul 15 09:59:34 /firewalld[718]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in th
Jul 15 09:59:34 /firewalld[718]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed: iptables: Bad rule (do
Jul 15 09:59:34 /firewalld[718]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
Jul 15 09:59:34 /firewalld[718]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed: iptables: No chain/target/match
Jul 15 09:59:34 /firewalld[718]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w -t filter -C FORWARD -o docker0 -j DOCKER' failed: iptables: No chain/target/match by that name.
Jul 15 09:59:34 /firewalld[718]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w -t filter -C FORWARD -j DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
Jul 15 09:59:34 /firewalld[718]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
kj54321
on 15 Jul 2016
_USER POLL_
_The best way to get notified of updates is to use the _Subscribe_ button on this page._
Please don't use "+1" or "I have this too" comments on issues. We automatically
collect those comments to keep the thread short.
The people listed below have upvoted this issue by leaving a +1 comment:
@virtuman
GordonTheTurtle
on 21 Jul 2016
CentOS 7.2 has the same issue:
systemctl status firewalld
Jul 21 02:58:26 host8.air.com firewalld[959]: 2016-07-21 02:58:26 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -o docker_gwbridge -j DOCKER' failed: iptables: No chain/target/match by that name.
Jul 21 02:58:26 host8.air.com firewalld[959]: 2016-07-21 02:58:26 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -j DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
Jul 21 02:58:26 host8.air.com firewalld[959]: 2016-07-21 02:58:26 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE' failed: iptables: No chain/target/match by that name.
Jul 21 02:58:26 host8.air.com firewalld[959]: 2016-07-21 02:58:26 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Jul 21 02:58:27 host8.air.com firewalld[959]: 2016-07-21 02:58:27 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Jul 21 02:58:27 host8.air.com firewalld[959]: 2016-07-21 02:58:27 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Jul 21 02:58:27 host8.air.com firewalld[959]: 2016-07-21 02:58:27 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Jul 21 02:58:27 host8.air.com firewalld[959]: 2016-07-21 02:58:27 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Jul 21 02:58:27 host8.air.com firewalld[959]: 2016-07-21 02:58:27 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -j DOCKER' failed: iptables: No chain/target/match by that name.
Jul 21 02:58:27 host8.air.com firewalld[959]: 2016-07-21 02:58:27 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
virtuman
on 21 Jul 2016
I use CentOS 7 and docker version 1.11.2. I found if firewalld.service starts after docker.service, the firewalld would work properly. I just changed firewalld.service file and let it start after docker.service. Hope it will help.
tonny1983
on 21 Jul 2016
If anyone finds this page and is still having difficulty with Docker and FirewallD.
Depending upon your deployment architecture, a small sort-of workaround to this is to bind your containers to ONLY the IPs or interfaces they will use. For example:
docker run --name ubuntu --net docker-network -it -p 172.22.144.17:20080:80 ubuntu /bin/bash
This opens an Ubuntu container up but the port forwarding rule is bound only to my 172.22.144.17 address (private VPN VLAN). This means that the port 20080 is NOT being offered to any other interface (for a start you can't route 172.22 outside a local network, internet facing routers drop it). It should work fine with public IP's. Netstat example below:
[root@splendor cellardoor]# netstat -tulpn | grep 20080
tcp 0 0 172.22.144.17:20080 0.0.0.0:* LISTEN 29173/docker-proxy
[root@splendor cellardoor]#
So start FirewallD first and let it do it's thing. Then start Docker and let it punch holes in your underlying iptables config which FirewallD set up (and allows internal container DNS etc to work). Then create containers but bind them only on specific IP&Port combinations (technical word being, sockets).
Not the best solution but it's what I'm doing until some wonderful person somewhere figures out how to properly address this problem.
xcellardoor
on 22 Oct 2016
@kj54321 and others
All the firewalld logged errors of this kind
Jul 15 09:59:34 /firewalld[718]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w -t nat -C [...]' failed: iptables: Bad rule (does a matching rule exist in that chain?).
are not a sign of the original issue. They are expected.
When iptables supports the -C parameter, docker first checks if a certain rule exists before deleting it or vice-versa, with the iptables -C command.
If the rule exists, iptables -C returns no error. If it does not not exist, iptables -C responds with that known error message. We should just read it as that rule did not exist when we checked. Docker handles it that way.
Also, a similar failure in deleting a rule/chain (-D) or flushing/destroying a chain (-F/-X) is not that bad if it happens at docker boot. It could just be the rule/chain docker is unconditionally deleting (it wants to start from a clean state) is not present (if this is the first time docker is started on this host).
aboch
on 28 Dec 2016
Any news on this?
exislow
on 10 Jan 2017
Note: How I fixed it is a section below.
I was having issues with firewalld and Docker, specifically with Rancher v1.2 in CentOS 7.
Rancher needs a container to be able to communicate to the host via it's external (possibly local) IP address.
It was a requirement to have firewalld enabled and working.
Update 2017-01-13
My previous fix would work only for the current session but not after a restart or after restarting or reconnecting interfaces in some specific order. This is due to a misbehavior of firewalld.
It is reported in the firewalld repo with a lot of details, here: ~https://github.com/t-woerner/firewalld/issues/195~ https://github.com/firewalld/firewalld/issues/195 If you want to know and understand all the details read that issue.
If you want a quick and simple fix just for Docker (that now works after system restarts, etc), read the updated fix below.
Environment
OS: CentOS Linux release 7.3.1611 (Core)
Kernel updated manually using "elrepo" to be able to use Overlay2 (devicemapper gave us several problems), although I don't think the kernel or the back end store is related to the problem: 4.9.0-1.el7.elrepo.x86_64
Docker:
Containers: 9
Running: 9
Paused: 0
Stopped: 0
Images: 29
Server Version: 1.12.5
Storage Driver: overlay2
Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge null host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.9.0-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 56
Total Memory: 125.7 GiB
Name: node19
ID: 3SC3:MCMU:PYSL:IBPE:4LPY:OYAG:RUUF:QMQ7:JWDS:5VRI:V7IS:KUYC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8
[root@node19 cerebro]# uname -r
4.9.0-1.el7.elrepo.x86_64
[root@node19 cerebro]# docker info
Containers: 9
Running: 9
Paused: 0
Stopped: 0
Images: 29
Server Version: 1.12.5
Storage Driver: overlay2
Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.9.0-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 56
Total Memory: 125.7 GiB
Name: node19
ID: 3SC3:MCMU:PYSL:IBPE:4LPY:OYAG:RUUF:QMQ7:JWDS:5VRI:V7IS:KUYC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8
firewalld is enabled:
systemctl status firewalld.service
â—Ź firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
Active: active (running) since jue 2017-01-05 18:04:12 COT; 4 days ago
Docs: man:firewalld(1)
Main PID: 24584 (firewalld)
Memory: 46.5M
CGroup: /system.slice/firewalld.service
└─24584 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
ene 10 10:08:23 node19 firewalld[24584]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D PREROUTING' failed:
ene 10 10:08:23 node19 firewalld[24584]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT' failed:
ene 10 10:08:23 node19 firewalld[24584]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -X DOCKER' failed:
ene 10 10:08:23 node19 firewalld[24584]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -X DOCKER-ISOLATION' failed:
ene 10 10:08:23 node19 firewalld[24584]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -n -L DOCKER' failed:
ene 10 10:08:23 node19 firewalld[24584]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C DOCKER-ISOLATION -j RETURN' failed:
ene 10 10:08:23 node19 firewalld[24584]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed:
ene 10 10:08:23 node19 firewalld[24584]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed:
ene 10 10:08:23 node19 firewalld[24584]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed:
ene 10 10:08:23 node19 firewalld[24584]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed:
...but those errors in firewalld are expected as pointed out in: https://github.com/docker/docker/issues/16137#issuecomment-269398488
The problem (updated 2017-01-13)
firewalld uses iptables and IPtables rules under the hood, but the way it "works" is with different "zones", with different levels of access (as I understand, different sets of iptables rules for each "zone").
The docker0 network interface is not assigned to any zone, so, some of its communications are blocked.
In my case, running the Rancher "agent" (a Docker container) with firewalld active gives errors about not being able to connect to the Rancher server (via it's external IP, in where the Rancher server, another container, is listening to). But with firewalld inactive, it works normally.
To make it worse, in all the systems I have tested (bare metal and Linode VPS) the docker0 interface's firewalld zone cannot be modified consistently using firewalld commands (firewall-cmd), because it is "controlled" by the "NetworkManager".
...and there's more, firewalld is currently misbehaving in its communication with "NetworkManager" right after a boot, giving as a result that the configurations are "lost" after rebooting the system.
But firewalld interacts with zone configurations in 3 places, ifcfg files in /etc/sysconfig/network-scripts/, .xml files in /etc/firewalld/zones/ and listen to messages given by NetworkManager via "D-Bus" (a Linux inter-process message system).
If NetworkManager is running, firewalld will try to read NetworkManager's configurations via D-Bus, and when setting zone configurations it will write them to ifcfg files.
But when NetworkManager is not running it will write (and use) the configurations in /etc/firewalld/zones/ as .xml files.
And for networks not controlled by NetworkManager (or if it is not running), firewalld will also overwrite some settings in those ifcfg files, even if it is reading and writing its rules to .xml files in /etc/firewalld/zones/.
All this gets messed up at boot time (here's the misbehavior), as firewalld seems to be unable to communicate with NetworkManager and assume it is not running, using only the files in /etc/firewalld/zones/ and overwriting all the ifcfg files in /etc/sysconfig/network-manager/... and those are the files that NetworkManager reads to set its configurations.
And this is just the short version... to know all the details read that issue in firewalld: https://github.com/t-woerner/firewalld/issues/195
The (new) fix (updated 2017-01-13) (updated 2017-01-16)
The ultra short version of the fix
- Run all these commands:
nmcli connection modify docker0 connection.zone trusted
systemctl stop NetworkManager.service
firewall-cmd --permanent --zone=trusted --change-interface=docker0
systemctl start NetworkManager.service
nmcli connection modify docker0 connection.zone trusted
systemctl restart docker.service
The explained version and how to check everything worked
The current workaround that seems to work ends up creating a trusted.xml file AND a ifcfg-docker0 file. The trusted.xml file would set the zone after a reboot (read and used by firewalld) and the ifcfg-docker0 would set the zone after reload or restart of services and interface or connections restarted (read and used mainly by NetworkManager).
To achieve that:
- After having the new interface (e.g. after installing Docker) and having FirewallD enabled and started, set the zone of the interface with NetworkManager's
nmcli:
nmcli connection modify docker0 connection.zone trusted
...that would set the zone in NetworkManager and FirewallD for the current session and will create the ifcfg-docker0 file for services, network or interfaces' restarts and reloads.
- Check that the file was created with:
cat /etc/sysconfig/network-scripts/ifcfg-docker0
...it should output something like:
DEVICE=docker0
STP=no
BRIDGING_OPTS=ageing_time=299
TYPE=Bridge
BOOTPROTO=none
IPADDR=172.17.0.1
PREFIX=16
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
NAME=docker0
UUID=5ccc8292-95a2-40d5-9ed6-ab6202fa629e
ONBOOT=no
ZONE=trusted
...specifically, it should have a:
ZONE=trusted
- Now we need FirewallD to generate that
trusted.xmlfile so that it uses it while booting, but for FirewallD to write that file it must think that NetworkManager is not active, so stop NetworkManager:
systemctl stop NetworkManager.service
- Now set the zone with FirewallD's
firewall-cmd:
firewall-cmd --permanent --zone=trusted --change-interface=docker0
- As NetworkManager is stopped, it won't modify (or even try to create) an
ifcfg-docker0file, if NetworkManager was running it would try to create that same file and wouldn't work after reboot. But this time, as Networkmanager is stopped, it will create a file in the other place for configurations, we can see it with:
cat /etc/firewalld/zones/trusted.xml
...outputs:
<?xml version="1.0" encoding="utf-8"?>
<zone target="ACCEPT">
<short>Trusted</short>
<description>All network connections are accepted.</description>
<interface name="docker0"/>
</zone>
...we can see that the docker0 interface was added to this trusted zone by the:
<interface name="docker0"/>
- And now we can start NetworkManager again:
systemctl start NetworkManager.service
- It is possible that you need to set the zone with NetworkManager again as
firewalldmight have "forgotten" the zone settings, it won't do any harm:
nmcli connection modify docker0 connection.zone trusted
- We can check that FirewallD thinks that the
docker0is in thetrustedzone. Check the zone of thedocker0interface as seen by FirewallD:
firewall-cmd --get-zone-of-interface=docker0
...outputs:
- And NetworkManager also thinks that it is in the
trustedzone. Check the zone of thedocker0interface as seen by NetworkManager:
nmcli connection show docker0 | grep zone
...outputs something like:
connection.zone: trusted
We can restart the system and check that the zone will persist, for both FirewallD and NetworkManager.
If you already checked that it worked and don't want to restart the system, you still will have to restart the Docker service for it to re-create it's
ipatablesrules:
systemctl restart docker.service
- If you need to change more things with FirewallD and NetworkManager, or if something doesn't seem to be working, please read that issue in FirewallD, as here I'm not showing a lot of the details: https://github.com/t-woerner/firewalld/issues/195
(obsolete - read the updated fix section above) The fix
Note: This fix won't work after a restart. Read the new fix above.
* Check the `firewalld` "zone" asigned to the Docker interface:* If it says "no zone", you need to add it to the `trusted` zonefirewall-cmd --get-zone-of-interface=docker0* If the previous command returns a message like:firewall-cmd --permanent --zone=trusted --change-interface=docker0...it means that `firewalld` cannot control the zone of the interface (even if it says "success"). You can check that the zone was not assigned by running again:The interface is under control of NetworkManager, setting zone to 'trusted'. success* If that's the case, you need to use the "NetworkManager" and restart the `docker0` interface, so first stop the `docker` daemon:firewall-cmd --get-zone-of-interface=docker0* Now use the "NetworkManager" command line interface to set the zone of the `docker0` interface to `trusted`:systemctl stop docker* Now turn the `docker0` interface off:nmcli connection modify docker0 connection.zone trusted* And turn it on again:ifdown docker0* And check that the `firewalld` zone assigned to the interface `docker0` is "trusted":ifup docker0...now it should return `trusted` instead of `no zone` * Now you can start Docker again:firewall-cmd --get-zone-of-interface=docker0* Check if the problem you had was solvedsystemctl start docker
References
http://unix.stackexchange.com/a/225845/90829
Introduction to FirewallD on CentOS - Linode
RedHat forums (in the comments is mentioned the nmcli)
New References
FirewallD docs
FirewallD description of behavior with NetworkManager in its docs
NetworkManager docs
FirewallD source code
tiangolo
on 10 Jan 2017
Thank you. It worked! But my docker container is exposed to the internet now. I just wanted to establish a communication from the container to dockerhost. Is this possible?
exislow
on 13 Jan 2017
My previous fix to solve the problem with Docker and FirewallD in CentOS 7 won't work after a restart or after restarting some services.
This is due to a misbehavior in how FirewallD communicates with NetworkManager.
Please read the updated fix in that same comment: https://github.com/docker/docker/issues/16137#issuecomment-271615192
tiangolo
on 13 Jan 2017
@exislow:
Your Docker container can only be exposed to the internet if you set the ports to do so.
For example, if you run:
docker run -d --name test-nginx -p 80:80 nginx
It will bind that nginx container to the public port 80.
But if you run it like this (without the -p 80:80):
docker run -d --name test-nginx nginx
It won't bind a public port.
Now, if you want to run a container that actually does bind to a port in your host, but only on a specific interface and not all interfaces (not public) you can specify the IP that it should bind to, for example:
docker run -d --name test-nginx -p 127.0.0.1:80:80 nginx
But if you only need to make your containers communicate internally, you don't need to expose any ports at all.
If the previous doesn't solve your problem, please specify:
- how are you running your container
- what is listening on your host and how are you running that thing on your host
- if it's another Docker container, specify how you are running it
- how do you see that your Docker container is exposed to the internet
tiangolo
on 13 Jan 2017
Hi, after running your fix, i just got errors when restart firewalld as follows:
Jan 24 08:40:30 nifdc1 firewalld[1375]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j...EPT' failed:
Jan 24 08:40:30 nifdc1 firewalld[1375]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -j DOCKER' failed:
Jan 24 08:40:30 nifdc1 firewalld[1375]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C POSTROUTING -s 172.18.0.0/16 ! -o docker_gwbridge -j MASQUERADE' failed:
Jan 24 08:40:30 nifdc1 firewalld[1375]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -i docker_gwbridge -j RETURN' failed:
Jan 24 08:40:30 nifdc1 firewalld[1375]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker_gwbridge -o docker_gwbridge -j ACCEPT' failed:
Jan 24 08:40:30 nifdc1 firewalld[1375]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker_gwbridge -o docker_gwbridge -j DROP' failed:
Jan 24 08:40:30 nifdc1 firewalld[1375]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker_gwbridge ! -o docker_gwbridge -j ACCEPT' failed:
Jan 24 08:40:30 nifdc1 firewalld[1375]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker_gwbridge -m conntrack --ctstate RELATED,ESTABL...EPT' failed:
Jan 24 08:40:30 nifdc1 firewalld[1375]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker_gwbridge -j DOCKER' failed:
Jan 24 08:40:30 nifdc1 firewalld[1375]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed:
is it good?
jstdoit
on 24 Jan 2017
me too
crabbulla
on 7 Feb 2017
Hi
after applying fix (update 2017-01-16) when restarting docker I got similar errors in /var/log/firewalld :
2017-02-08 11:33:04 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER' failed:
2017-02-08 11:33:04 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed:
2017-02-08 11:33:04 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D PREROUTING' failed:
2017-02-08 11:33:04 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT' failed:
2017-02-08 11:33:04 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -F DOCKER' failed:
2017-02-08 11:33:04 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -X DOCKER' failed:
2017-02-08 11:33:04 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -F DOCKER' failed:
2017-02-08 11:33:04 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -X DOCKER' failed:
2017-02-08 11:33:04 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -F DOCKER-ISOLATION' failed:
2017-02-08 11:33:04 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -X DOCKER-ISOLATION' failed:
2017-02-08 11:33:04 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -n -L DOCKER' failed:
2017-02-08 11:33:04 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -n -L DOCKER' failed:
2017-02-08 11:33:05 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -n -L DOCKER-ISOLATION' failed:
2017-02-08 11:33:05 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C DOCKER-ISOLATION -j RETURN' failed:
2017-02-08 11:33:05 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE' failed:
2017-02-08 11:33:05 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed:
2017-02-08 11:33:05 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed:
2017-02-08 11:33:05 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed:
2017-02-08 11:33:05 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed:
2017-02-08 11:33:05 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed:
2017-02-08 11:33:05 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed:
2017-02-08 11:33:05 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed:
2017-02-08 11:33:05 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -j DOCKER' failed:
2017-02-08 11:33:05 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -j DOCKER-ISOLATION' failed:
2017-02-08 11:33:05 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE' failed:
2017-02-08 11:33:05 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed:
2017-02-08 11:33:05 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed:
2017-02-08 11:33:05 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed:
2017-02-08 11:33:05 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed:
2017-02-08 11:33:05 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed:
2017-02-08 11:33:05 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -j DOCKER' failed:
After starting container for NGINX Proxy using port 80:80 and 443:443 and IP =172.17.0.2, errors in /var/log/firewalld :
2017-02-08 11:33:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -p tcp -d 0/0 --dport 443 -j DNAT --to-destination 172.17.0.2:433 ! -i docker0' failed:
2017-02-08 11:33:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 433 -j ACCEPT' failed:
2017-02-08 11:33:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C POSTROUTING -p tcp -s 172.17.0.2 -d 172.17.0.2 --dport 433 -j MASQUERADE' failed:
2017-02-08 11:33:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -p tcp -d 0/0 --dport 80 -j DNAT --to-destination 172.17.0.2:80 ! -i docker0' failed:
2017-02-08 11:33:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 80 -j ACCEPT' failed:
2017-02-08 11:33:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C POSTROUTING -p tcp -s 172.17.0.2 -d 172.17.0.2 --dport 80 -j MASQUERADE' failed:
accessing to port 80 is working from public, using eth0 IP and with localhost
accessing to port 443 doesn't work with any destination except when using "curl https://172.17.0.2" on host
Docker isn't able to apply right commands on "iptables"
Rhodan
Rhodan-FR
on 8 Feb 2017
@Rhodan-FR
You can discard all those firewalld reported warnings.
All the failure on -C rules, just means the rule is not present when docker checked if the rule is there for cleanup.
Same for the -L failures, they are because docker checks if the chain is there, if so it would flush it.
For some other ones docker directly attempts a -D to delete a well known rule, w/o first checking if the rule is there.
accessing to port 443 doesn't work with any destination except when using "curl https://172.17.0.2" on host
Did you publish the port 443 when you started your container (ex.: docker run -p 80:80 -p 443:443 nginx )?
aboch
on 8 Feb 2017
@aboch
yes container is started through docker run -p 80:80 -p 443:443 ....
I made another test which surprised me : I change inside NGINX server config and in docker run the port 443 with port 444 and all was working fine
Before starting container I checked if ports 80 and 443 are free
Rhodan-FR
on 9 Feb 2017
Correct me if wrong, but is it not possible to go around many of the problems mentioned in this issue by using the --ip option of the docker daemon documented here? More specifically, you could
PRIVATE_IP=<your_private_ip_here>
sudo echo "{ \"ip\": \"$PRIVATE_IP\" }" | sudo tee /etc/docker/daemon.json
and restart docker. Note that this erases your current daemon.json configuration!
_(If you have a custom DNS server with the right private IP, this could be a way to get it:
$(nslookup `hostname`| awk '/^Address: / { print $2 ; exit }'))_
This would make docker only bind to the private IP (which could -- or not -- correspond to a real private interface) , thus not exposing docker's services to the rest of the world (of course, it would then also only reply to requests for _that_ IP, which may or may not be something you want).
Jorl17
on 26 May 2017
@aboch what do you mean by "discard all those firewalld reported warnings"? Is there a way to suppress or remove these warnings?
2017-08-03 15:17:46 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed:
2017-08-03 15:17:46 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D PREROUTING' failed:
2017-08-03 15:17:46 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT' failed:
2017-08-03 15:17:46 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -X DOCKER' failed:
2017-08-03 15:17:46 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -X DOCKER-ISOLATION' failed:
2017-08-03 15:17:46 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -n -L DOCKER' failed:
2017-08-03 15:17:46 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C DOCKER-ISOLATION -j RETURN' failed:
2017-08-03 15:17:46 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed:
2017-08-03 15:17:46 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed:
2017-08-03 15:17:46 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed:
2017-08-03 15:17:46 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed:
2017-08-03 15:17:46 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE' failed:
2017-08-03 15:17:46 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed:
2017-08-03 15:17:46 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed:
2017-08-03 15:17:46 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed:
2017-08-03 15:17:46 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed:
2017-08-03 15:17:46 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed:
2017-08-03 15:17:46 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -j DOCKER' failed:
RobbieTheK
on 3 Aug 2017
@aboch I would also like to know if I can safely ignore these warnings?
Aug 17 15:39:06 firewalld[4057]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE' failed:
Aug 17 15:39:06 firewalld[4057]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed:
Aug 17 15:39:06 firewalld[4057]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed:
Aug 17 15:39:06 firewalld[4057]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed:
Aug 17 15:39:06 firewalld[4057]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed:
Aug 17 15:39:06 firewalld[4057]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -j DOCKER' failed:
Aug 17 15:39:06 firewalld[4057]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed:
Aug 17 15:39:06 firewalld[4057]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -n -L DOCKER-USER' failed:
Aug 17 15:39:06 firewalld[4057]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C DOCKER-USER -j RETURN' failed:
Aug 17 15:39:07 firewalld[4057]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -j DOCKER-USER' failed:
any advice on whether I need to worry about this or can I ignore?
J77J
on 17 Aug 2017
@J77J
Docker will operate but network functions, particularly communication between containers and traffic inbound from external sources to your containers will likely not work. What Docker is doing to iptables is creating a rule structure which allows this type of connectivity.
xcellardoor
on 17 Aug 2017
@xcellardoor ok, that'll be a no then...
looks like I'll have to switch out firewalld for iptables in that case.
Thanks!
J77J
on 17 Aug 2017
Not sure if this is fixed or not but one cannot have too many logs :smiley:
docker --version
Docker version 1.10.3, build 79ebcd8-unsupported
journalctl _SYSTEMD_UNIT=firewalld.service --no-pager
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table mangle --delete POSTROUTING --out-interface virbr0 --protocol udp --destination-port 68 --jump CHECKSUM --checksum-fill' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 --destination 224.0.0.0/24 --jump RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 --destination 255.255.255.255/32 --jump RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 -p tcp ! --destination 192.168.122.0/24 --jump MASQUERADE --to-ports 1024-65535' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 -p udp ! --destination 192.168.122.0/24 --jump MASQUERADE --to-ports 1024-65535' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -I DOCKER -i docker0 -j RETURN' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 ! --destination 192.168.122.0/24 --jump MASQUERADE' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -A DOCKER -p tcp -d 0/0 --dport 2005 -j DNAT --to-destination 172.17.0.7:22 ! -i docker0' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table filter --delete FORWARD --destination 192.168.122.0/24 --out-interface virbr0 --match conntrack --ctstate ESTABLISHED,RELATED --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -A DOCKER -p tcp -d 0/0 --dport 2300 -j DNAT --to-destination 172.17.0.8:22 ! -i docker0' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table filter --delete FORWARD --source 192.168.122.0/24 --in-interface virbr0 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -A DOCKER -p tcp -d 0/0 --dport 2110 -j DNAT --to-destination 172.17.0.9:22 ! -i docker0' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table filter --delete FORWARD --in-interface virbr0 --out-interface virbr0 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -A DOCKER -p tcp -d 0/0 --dport 2000 -j DNAT --to-destination 172.17.0.2:22 ! -i docker0' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table filter --delete FORWARD --out-interface virbr0 --jump REJECT' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -A DOCKER -p tcp -d 0/0 --dport 2001 -j DNAT --to-destination 172.17.0.3:22 ! -i docker0' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table filter --delete FORWARD --in-interface virbr0 --jump REJECT' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -A DOCKER -p tcp -d 0/0 --dport 2002 -j DNAT --to-destination 172.17.0.4:22 ! -i docker0' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table filter --delete INPUT --in-interface virbr0 --protocol udp --destination-port 53 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -A DOCKER -p tcp -d 0/0 --dport 2003 -j DNAT --to-destination 172.17.0.5:22 ! -i docker0' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table filter --delete INPUT --in-interface virbr0 --protocol tcp --destination-port 53 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -A DOCKER -p tcp -d 0/0 --dport 2004 -j DNAT --to-destination 172.17.0.6:22 ! -i docker0' failed: iptables: No chain/target/match by that name.
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table filter --delete OUTPUT --out-interface virbr0 --protocol udp --destination-port 68 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table filter --delete INPUT --in-interface virbr0 --protocol udp --destination-port 67 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:16:10 PC4 firewalld[11649]: 2017-09-01 12:16:10 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -w --table filter --delete INPUT --in-interface virbr0 --protocol tcp --destination-port 67 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:01 PC4 firewalld[11649]: 2017-09-01 12:46:01 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D DOCKER -p tcp -d 0/0 --dport 2300 -j DNAT --to-destination 172.17.0.8:22 ! -i docker0' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:01 PC4 firewalld[11649]: 2017-09-01 12:46:01 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D DOCKER -p tcp -d 0/0 --dport 2003 -j DNAT --to-destination 172.17.0.5:22 ! -i docker0' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:01 PC4 firewalld[11649]: 2017-09-01 12:46:01 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D DOCKER -p tcp -d 0/0 --dport 2000 -j DNAT --to-destination 172.17.0.2:22 ! -i docker0' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:02 PC4 firewalld[11649]: 2017-09-01 12:46:02 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D DOCKER -p tcp -d 0/0 --dport 2004 -j DNAT --to-destination 172.17.0.6:22 ! -i docker0' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:02 PC4 firewalld[11649]: 2017-09-01 12:46:02 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D DOCKER -p tcp -d 0/0 --dport 2005 -j DNAT --to-destination 172.17.0.7:22 ! -i docker0' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:02 PC4 firewalld[11649]: 2017-09-01 12:46:02 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D DOCKER -p tcp -d 0/0 --dport 2110 -j DNAT --to-destination 172.17.0.9:22 ! -i docker0' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:03 PC4 firewalld[11649]: 2017-09-01 12:46:03 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D DOCKER -p tcp -d 0/0 --dport 2002 -j DNAT --to-destination 172.17.0.4:22 ! -i docker0' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:04 PC4 firewalld[11649]: 2017-09-01 12:46:04 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D DOCKER -p tcp -d 0/0 --dport 2001 -j DNAT --to-destination 172.17.0.3:22 ! -i docker0' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory
Try `iptables -h' or 'iptables --help' for more information.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory
Try `iptables -h' or 'iptables --help' for more information.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory
Try `iptables -h' or 'iptables --help' for more information.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -F DOCKER' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -X DOCKER' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -F DOCKER' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -X DOCKER' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -F DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -n -L DOCKER' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -n -L DOCKER' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -n -L DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:05 PC4 firewalld[11649]: 2017-09-01 12:46:05 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C DOCKER-ISOLATION -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:06 PC4 firewalld[11649]: 2017-09-01 12:46:06 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:06 PC4 firewalld[11649]: 2017-09-01 12:46:06 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:06 PC4 firewalld[11649]: 2017-09-01 12:46:06 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:06 PC4 firewalld[11649]: 2017-09-01 12:46:06 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:06 PC4 firewalld[11649]: 2017-09-01 12:46:06 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Sep 01 12:46:06 PC4 firewalld[11649]: 2017-09-01 12:46:06 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:06 PC4 firewalld[11649]: 2017-09-01 12:46:06 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:06 PC4 firewalld[11649]: 2017-09-01 12:46:06 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -j DOCKER' failed: iptables: No chain/target/match by that name.
Sep 01 12:46:06 PC4 firewalld[11649]: 2017-09-01 12:46:06 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -j DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
MaciejKucia
on 1 Sep 2017
I use an auxiliar script like next:
docker-start.sh
#!/usr/bin/env bash
set -e
set -x
docker-compose up -d
sleep 5
# #Fix1: Fix "iptable service restart" error
echo 'Fix "iptable service restart" error'
echo 'https://github.com/moby/moby/issues/16137#issuecomment-160505686'
for container_id in $(docker ps --filter='ancestor=reduardo7/my-image' -q)
do
docker exec $container_id sh -c 'iptables-save > /etc/sysconfig/iptables'
done
# End #Fix1
echo Done
reduardo7
on 1 Dec 2017
Hey @tiangolo I just tried to do the following on a fresh CentOS box (7.4.1708)
nmcli connection modify docker0 connection.zone trusted
systemctl stop NetworkManager.service
firewall-cmd --permanent --zone=trusted --change-interface=docker0
systemctl start NetworkManager.service
nmcli connection modify docker0 connection.zone trusted
systemctl restart docker.service
Following this comment: https://github.com/moby/moby/issues/16137#issuecomment-271615192
And i still have issues where my docker containers cannot connect to one another using the host IP.
Despite the fact that
firewall-cmd --get-zone-of-interface=docker0
and
nmcli connection show docker0 | grep zone
return trusted
In addition
cat /etc/sysconfig/network-scripts/ifcfg-docker0
DEVICE=docker0
STP=no
TYPE=Bridge
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
IPADDR=172.17.0.1
PREFIX=16
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV4_DNS_PRIORITY=100
IPV6INIT=no
NAME=docker0
ONBOOT=no
ZONE=trusted
sudo cat /etc/firewalld/zones/trusted.xml
<?xml version="1.0" encoding="utf-8"?>
<zone target="ACCEPT">
<short>Trusted</short>
<description>All network connections are accepted.</description>
<interface name="docker0"/>
</zone>
kwojcicki
on 5 Dec 2017
Replying to my previous comment. The issue was our docker-compose specified another network ie
networks:
default:
Therefore all our containers were not on the docker0 bridge but on a randomly generated bridge.
To fix this restructured our docker-compose.yml
networks:
default:
driver_opts:
com.docker.network.bridge.name: "dockernet"
Next ran
firewall-cmd --permanent --zone=trusted --change-interface=dockernet
And worked like a charm
kwojcicki
on 8 Dec 2017
A combination of @kwojcicki's and @tiangolo's comments solved my issue. The published addresses of my containers were unreachable through the host IP from inside of other containers. Disabling firewalld was not an option, as we needed its NAT routing to access external resources.
PMarci
on 23 Jan 2018
@tiangolo thank you so much!!!! You Rock !
Angelinsky7
on 3 Feb 2018
@tiangolo : our team found this issue and applied in our environment: CentOs 7.4.1708 / docker-ce-17.12.1.ce-1.el7.centos.x86_64 / Swarm classic 1.2.8.
Do you think this workaround still applies now or is this deprecated? This is strange we have to work around firewall rules as this is something docker should do, isn't it? Thank you.
antoinetran
on 4 Jun 2018
@antoinetran yes, Docker should handle firewall rules, but this was a bug in RedHat, inherited by CentOS, it was not misbehavior of Docker but a bug in RedHat.
It was supposedly fixed in RedHat and there was supposedly an update / fix in CentOS. You can read the last comments in the issue in Firewalld: https://github.com/firewalld/firewalld/issues/195
I don't know if that fix works, nor if all the description and workaround still applies because I don't use RedHat/CentOS very frequently.
But by recent comments of this year by @PMarci and @Angelinsky7 , it seems it still applies.
tiangolo
on 5 Jun 2018
@PMarci and @Angelinsky7 : can you tell us your CentOs/RedHat version at the time of the patch? The related RedHat issue here says this is fixed since 7.3.
antoinetran
on 5 Jun 2018
@antoinetran CentOS Linux release 7.4.1708 (Core)
Angelinsky7
on 5 Jun 2018
Today I updated a CentOS 7 system by using "yum update". This also updated Docker-CE. System is now "CentOS Linux release 7.5.1804 (Core)" and Docker is "18.05.0.ce-3.el7.centos". The previous system update was done a few weeks ago.
Now DNS resolution does not work anymore from inside docker containers. Maybe this is related to this issue?
EDIT: Deleting /var/lib/docker/network/files/ fixed my issue. Maybe it's unrelated to this issue after all.
ChristianCiach
on 5 Jun 2018
@Angelinsky7 our team is also using centos 7.4.1708 and using my fix above it all works fine.
kwojcicki
on 5 Jun 2018
@antoinetran Unfortunately I'm unable to check it as it's on a customer's on-premise system, and I don't have access right now.
PMarci
on 5 Jun 2018
Ok thank you all. I added that info in the related RedHat issue here . I hope they will reopen it.
antoinetran
on 5 Jun 2018
I have this issue on CentOS 7.6.1810. Docker version is 1.13.1, build b2f74b2/1.13.1.
Apr 11 09:25:31 vps-4 dockerd-current: time="2019-04-11T09:25:31.195015109+01:00" level=info msg="Firewalld running: true"
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j D
OCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory#012#012Try `iptables -h' or 'iptables --help' for more information.
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed: iptab
les v1.4.21: Couldn't load target `DOCKER':No such file or directory#012#012Try `iptables -h' or 'iptables --help' for more information.
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule
exist in that chain?).
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exi
st in that chain?).
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -F DOCKER' failed: iptables: No chain/target/match by that name
.
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -X DOCKER' failed: iptables: No chain/target/match by that name
.
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -F DOCKER' failed: iptables: No chain/target/match by that n
ame.
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -X DOCKER' failed: iptables: No chain/target/match by that n
ame.
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -F DOCKER-ISOLATION' failed: iptables: No chain/target/match
by that name.
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match
by that name.
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -n -L DOCKER' failed: iptables: No chain/target/match by that n
ame.
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -n -L DOCKER' failed: iptables: No chain/target/match by tha
t name.
Apr 11 09:25:31 vps-4 firewalld[4121]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -n -L DOCKER-ISOLATION' failed: iptables: No chain/target/ma
tch by that name.
Chaz6
on 11 Apr 2019
in a NUTSHELL,
The below commands might be needed when the firewalld is enabled on the centos 7, is that correct? unless using different interface for docker.
nmcli connection modify docker0 connection.zone trusted
systemctl stop NetworkManager.service
firewall-cmd --permanent --zone=trusted --change-interface=docker0
systemctl start NetworkManager.service
nmcli connection modify docker0 connection.zone trusted
systemctl restart docker.service
deepaklor
on 18 Apr 2019
While adding the docker0 interface to the trusted zone might be a solution for one who wants to expose the Docker ports to public, but it is not for use cases where you need to protect the port(s) and only allow named IPs.
Solution is to let firewalld create the DOCKER-USER chain and apply rules to it.
See https://roosbertl.blogspot.com/2019/06/securing-docker-ports-with-firewalld.html
robertoschwald
on 13 Jun 2019
Note that the DOCKER-USER url ( https://roosbertl.blogspot.com/2019/06/securing-docker-ports-with-firewalld.html ) only works for docker-ce, not for native docker in rhel7/centos7.
There's a RFE about DOCKER-USER support in the rhel7 native docker 1.13: https://bugzilla.redhat.com/show_bug.cgi?id=1678883
pasikarkkainen
on 13 Jun 2019
Related issues
noahwhite
·
3Comments
netoneko
·
3Comments
programmerq
·
3Comments
karellm
·
3Comments
gg7
·
3Comments
Most helpful comment
Note: How I fixed it is a section below.
I was having issues with
firewalldand Docker, specifically with Rancher v1.2 in CentOS 7.Rancher needs a container to be able to communicate to the host via it's external (possibly local) IP address.
It was a requirement to have
firewalldenabled and working.Update 2017-01-13
My previous fix would work only for the current session but not after a restart or after restarting or reconnecting interfaces in some specific order. This is due to a misbehavior of
firewalld.It is reported in the
firewalldrepo with a lot of details, here: ~https://github.com/t-woerner/firewalld/issues/195~ https://github.com/firewalld/firewalld/issues/195 If you want to know and understand all the details read that issue.If you want a quick and simple fix just for Docker (that now works after system restarts, etc), read the updated fix below.
Environment
OS:
CentOS Linux release 7.3.1611 (Core)Kernel updated manually using "elrepo" to be able to use Overlay2 (
devicemappergave us several problems), although I don't think the kernel or the back end store is related to the problem:4.9.0-1.el7.elrepo.x86_64Docker:
firewalldis enabled:...but those errors in
firewalldare expected as pointed out in: https://github.com/docker/docker/issues/16137#issuecomment-269398488The problem (updated 2017-01-13)
firewalldusesiptablesand IPtables rules under the hood, but the way it "works" is with different "zones", with different levels of access (as I understand, different sets ofiptablesrules for each "zone").The
docker0network interface is not assigned to any zone, so, some of its communications are blocked.In my case, running the Rancher "agent" (a Docker container) with
firewalldactive gives errors about not being able to connect to the Rancher server (via it's external IP, in where the Rancher server, another container, is listening to). But withfirewalldinactive, it works normally.To make it worse, in all the systems I have tested (bare metal and Linode VPS) the
docker0interface'sfirewalldzone cannot be modified consistently usingfirewalldcommands (firewall-cmd), because it is "controlled" by the "NetworkManager"....and there's more,
firewalldis currently misbehaving in its communication with "NetworkManager" right after a boot, giving as a result that the configurations are "lost" after rebooting the system.But
firewalldinteracts with zone configurations in 3 places,ifcfgfiles in/etc/sysconfig/network-scripts/,.xmlfiles in/etc/firewalld/zones/and listen to messages given by NetworkManager via "D-Bus" (a Linux inter-process message system).If NetworkManager is running,
firewalldwill try to read NetworkManager's configurations via D-Bus, and when setting zone configurations it will write them toifcfgfiles.But when NetworkManager is not running it will write (and use) the configurations in
/etc/firewalld/zones/as.xmlfiles.And for networks not controlled by NetworkManager (or if it is not running),
firewalldwill also overwrite some settings in thoseifcfgfiles, even if it is reading and writing its rules to.xmlfiles in/etc/firewalld/zones/.All this gets messed up at boot time (here's the misbehavior), as
firewalldseems to be unable to communicate with NetworkManager and assume it is not running, using only the files in/etc/firewalld/zones/and overwriting all theifcfgfiles in/etc/sysconfig/network-manager/... and those are the files that NetworkManager reads to set its configurations.And this is just the short version... to know all the details read that issue in
firewalld: https://github.com/t-woerner/firewalld/issues/195The (new) fix (updated 2017-01-13) (updated 2017-01-16)
The ultra short version of the fix
The explained version and how to check everything worked
The current workaround that seems to work ends up creating a
trusted.xmlfile AND aifcfg-docker0file. Thetrusted.xmlfile would set the zone after a reboot (read and used byfirewalld) and theifcfg-docker0would set the zone after reload or restart of services and interface or connections restarted (read and used mainly by NetworkManager).To achieve that:
nmcli:...that would set the zone in NetworkManager and FirewallD for the current session and will create the
ifcfg-docker0file for services, network or interfaces' restarts and reloads....it should output something like:
...specifically, it should have a:
trusted.xmlfile so that it uses it while booting, but for FirewallD to write that file it must think that NetworkManager is not active, so stop NetworkManager:firewall-cmd:ifcfg-docker0file, if NetworkManager was running it would try to create that same file and wouldn't work after reboot. But this time, as Networkmanager is stopped, it will create a file in the other place for configurations, we can see it with:...outputs:
...we can see that the
docker0interface was added to thistrustedzone by the:firewalldmight have "forgotten" the zone settings, it won't do any harm:docker0is in thetrustedzone. Check the zone of thedocker0interface as seen by FirewallD:...outputs:
trustedzone. Check the zone of thedocker0interface as seen by NetworkManager:...outputs something like:
We can restart the system and check that the zone will persist, for both FirewallD and NetworkManager.
If you already checked that it worked and don't want to restart the system, you still will have to restart the Docker service for it to re-create it's
ipatablesrules:(obsolete - read the updated fix section above) The fix
Note: This fix won't work after a restart. Read the new fix above.
References
http://unix.stackexchange.com/a/225845/90829
Introduction to FirewallD on CentOS - Linode
RedHat forums (in the comments is mentioned the
nmcli)RedHat docs for
nmcliNew References
FirewallD docs
FirewallD description of behavior with NetworkManager in its docs
NetworkManager docs
FirewallD source code