Moby: No DOCKER chain in iptables

[brianclements]

I've been doing this tutorial and had trouble at the step:

DJANGO_APP=$(docker run -d -i -t -p 8000:8000 ubuntu /bin/bash)
WARNING: Docker detected local DNS server on resolv.conf. Using default external servers: [8.8.8.8 8.8.4.4]
WARNING: IPv4 forwarding is disabled.
2013/09/12 12:52:02 Error: Error starting container 037aac8ca3ab: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 8000 ! -i docker0 -j DNAT --to-destination 172.17.0.3:8000

So trying it manually gave me this:

sudo iptables -t nat -A DOCKER -p tcp --dport 8000 ! -i docker0 -j DNAT --to-destination 172.17.0.3:8000
iptables: No chain/target/match by that name.

This is verified by checking (note: I deleted the rules, just kept the chain names):

sudo iptables -L -n --line-numbers
Chain INPUT (policy DROP)
...

Chain FORWARD (policy DROP)
...

Chain OUTPUT (policy DROP)
...

Chain INBOUND (4 references)
...

Chain LOG_FILTER (5 references)
...

Chain LSI (2 references)
...

Chain LSO (0 references)
...

Chain OUTBOUND (3 references)
...

Any ideas? I did have an issues just before this where I couldn't access the internet at all from within containers, but "sharing" the internet locally with device docker0 using firestarter (gufw just doesn't work on my ubuntu 12.04) fixed it.

[crosbymichael]

@Briguy2k

Can you look for the DOCKER chain using sudo iptables -L -n -t nat ??

[brianclements]

@crosbymichael

I didn't see anything helpful there:

$ sudo iptables -L -n -t nat 
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0

[matteosister]

same problem here....any solution?

[crosbymichael]

@brianclements @matteosister

I'm guessing that you have the docker0 bridge so you can try to create the chain manually.

iptables -t nat -N DOCKER
iptables -t nat -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
iptables -t nat -A PREROUTING -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER

[matteosister]

@crosbymichael I've got the docker0 bridge, but the commands you gave didn't solve the issue....
Is there a way to restart from scratch?

[crosbymichael]

You can remove the docker0 bridge and have docker recreate the network setup.

Michael Crosby

On Nov 10, 2013, at 10:00 AM, Matteo Giachino [email protected] wrote:

@crosbymichael I've got the docker0 bridge, but the commands you gave didn't solve the issue....
Is there a way to restart from scratch?

—
Reply to this email directly or view it on GitHub.

[brianclements]

@crosbymichael
Your suggestions worked for me and I was finally able to complete the tutorial in the original post exactly as directed, even down to the working django web server. However when I restarted, all the iptables changes were undone. Any tips on where I should start looking to make those permanent?

[matteosister]

@crosbymichael for me nothing worked. Even after deleting the bridge still gives me the same error...

[crosbymichael]

@matteosister What host OS are you running on?

[shykes]

Tagging as /system/networking

[jpoimboe]

@brianclements @matteosister do you still see the issue with the latest version of docker?

[brianclements]

Thanks for following up @jpoimboe, I have since installed docker on many other machines and had 0 occurrence of this issue on those machines. And I'm pretty sure that I've had other miscellaneous firewall issues on the machine that this issue was occurring on, so I'm going to go ahead and take myself out of this equation and say that it's some issue with my firewall here on this machine and not docker.

Curious though myself @matteosister, what is your host system that you have the issue on and is it a fresh install or a system upgrade? My particular system has been under probably 3 years of Ubuntu upgrades and I think my iptables has too many different versions and types of firewalls all grabbing at it and undoing each others changes.

[eliasp]

@brianclements It still fails here:

Jan 25 00:12:21 moria docker[5337]: Unable to allow incoming packets: iptables failed: iptables -I FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT: iptables: No chain/target/match by that name.

• Distribution: Gentoo
• Kernel: 3.13.0
• Docker: 0.7.6
• Go: 1.2

Running iptables -t nat -L shows:

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  172.17.0.0/16       !172.17.0.0/16  

There's no other firewall framework/daemon (Shorewall, firewalld, ufw, …) enabled/active at the time of testing this, so there's no chance this is a conflict with any of them.

[tianon]

@eliasp I wonder if you just found another kernel flag our ebuild is missing. Do you have CONFIG_NETFILTER_XT_MATCH_CONNTRACK set in your kernel config, by any chance? :)

[eliasp]

@tianon That's the culprit:

# CONFIG_NETFILTER_XT_MATCH_CONNTRACK is not set

Added it and it works now! So this should be also added to the ebuild.

[brianclements]

I would like to try this as well. Could you please give some instructions?
I don't quite follow what you did.

[eliasp]

@brianclements Check, whether your Kernel has support for netfilter's connection tracking builtin.

A reasonably well built Kernel should expose its features through /proc/config.gz, so running

zgrep NETFILTER_XT_MATCH_CONNTRACK /proc/config.gz

should tell you about the support for this feature in the currently running Kernel.
Some distributions tend to use another not so deterministic approach and just place a plain config-files for each Kernel in /boot, so you'd have to do something like:

grep NETFILTER_XT_MATCH_CONNTRACK /boot/config-$(uname -r)

If the result of this check is is not set, this feature is not provided by your Kernel and you have to customize your Kernel to get support for it.

If the result of this check is =m, the feature is built as a module, so you'll have to load the module (it is loaded automatically on demand here, I don't know what would make your Ubuntu not to load it automatically) by running:

modprobe xt_conntrack

If the result of this check is =y the feature is already built into your Kernel and your problem has another reason than missing support for NETFILTER_XT_MATCH_CONNTRACK.

[brianclements]

So the result of: grep NETFILTER_XT_MATCH_CONNTRACK /boot/config-$(uname -r) was =m, but trying to load it with modprobe or adding it to /etc/modules or even having Ubuntu's UFW load it by adding it to /etc/default/ufw at the bottom via this section # Extra connection tracking modules to load., never changed it to =y. So I just tried reseting my firewall, deleting all chains and rules, completely purged lxc-docker AND lxc from my system (I think the mistake I made before was not purging lxc, which didn't clear out the faulty bridge), made sure the bridge was deleted with brtcl show, then finally reinstalled lxc-docker and this has finally fixed it for me. The chain persists and port forwarding works as it should. Thanks all for the help.

[jpoimboe]

Ok, I think this issue can be closed now.

[crosbymichael]

Thanks for the update

[fxtentacle]

I've experienced precisely this issue on a paravirtualized Ubuntu 12.04. The fix was to switch the kernel from my hoster's precompiled binary (which doesn't include the iptables addrtype module) to ubuntu's regular paravirtualized kernel.

[jpoimboe]

Hm, this kind of seems to be a common issue. Maybe we should add a kernel check to ensure that all the needed iptables modules are available. Or at least a better iptables error message to tell the user they might be missing something.

[MrManor]

Edit: I have now traced this problem down on my OpenSUSE box and I add this info here for others google'ing for a solution. I don’t know about other distributions.

On Suse the problem occurs when dhcp is used to acquirer ip addresses. Every time a dhcp lease is renewed - then the SUSEfirewall script is rerun. This destroys the ip rule setup by docker.

[Pensu]

@MrManor I am using static IP address. Still I am getting this issue. What solution did you use?

[MrManor]

@Pensu As I am only playing around whith this on my own network I moved the hole thing to another machine and disabled SuSEfirewall2 completely.

[stieg]

I just hit this issue on CentOS 7. The root cause for me is that when firewalld gets restarted then the rules disappear from the iptables. Here you can see this effect in action:

# systemctl start docker.service
# iptables -L | grep DOCKER
DOCKER     all  --  anywhere             anywhere
Chain DOCKER (1 references)
# systemctl restart firewalld.service
# iptables -L | grep DOCKER
#

This was seen on the following docker package:

Name        : docker
Arch        : x86_64
Version     : 1.6.0
Release     : 11.0.1.el7.centos
Size        : 32 M
Repo        : installed
From repo   : extras

The package is the most recent one available from CentOS repos.

So this is most definitely a problem on RHEL 7 / CentOS 7 because any restarting of the firewalld service will cause this problem. Not saying this event will happen often, but services should be able to restart at any time without creating this kind of problem.

Proper fix is to ensure you have all kernel modules loaded and that the needed chains are present in the iptables rule set prior to running the iptables configuration rules.

[shanye1987]

@stieg then,What solution did you use?

[stieg]

@shanye1987 We killed firewalld and removed the package. Since our services are on AWS we just use their security groups anyways.

[SandeepKarnawat]

I faced this issue when my iptables were stopped. After starting iptables docker was able to create the container with exposed ports.

[lukecampbell]

On a CentOS 7 VM, I was able to fix the issue by restarting docker through systemd which recreated the appropriate iptables rules.

sudo systemctl restart docker.service

This assumes of course that you have docker managed by systemd

[jdejoode]

Or if you used

sudo update-rc.d docker defaults

to manage docker:

sudo service docker restart

[davclark]

Just to clarify - this is still broke but the issue is closed?

[drptbl]

Still broke. Happening for me all the time with "selenium/node-chrome-debug" image from here https://github.com/SeleniumHQ/docker-selenium.

[mayank941]

Just stop all the running container then
systemctl restart docker.service
after that you can start your containers

[JoseAlban]

I had the same issue after restarting firewalld on Centos7 to apply a new rule, and fixed it after @stieg and @lukecampbell 's great insights.

OS: Centos7
Kernel: 3.10.0-327.4.4.el7.x86_64
Docker: version 1.10.3, build 20f81dd
Problem: docker fails to publish ports due to no 'DOCKER' chain present in iptables
Root Cause: firewalld reload screws up iptable state: sudo firewall-cmd --reload

Is it a firewalld bug, or just expected behaviour?
Possible Solution: make firewalld aware of docker's iptable chain, so a firewalld restart behaves properly?

docker will recreate iptable's 'DOCKER' chain after a sudo systemctl restart docker, which can be confirmed via sudo iptables -L -n -t nat | grep DOCKER

[mayank941]

thanx

Regards
Mayank Chauhan
[phone]

On Thu, Apr 7, 2016 at 4:55 PM, Jose Alban [email protected] wrote:

I had the same issue after restarting firewalld on Centos7 to apply a new
rule, and fixed it after @stieg https://github.com/stieg and
@lukecampbell https://github.com/lukecampbell 's great insights.

OS: Centos7
Kernel: 3.10.0-327.4.4.el7.x86_64
Problem: docker fails to publish ports due to no 'DOCKER' chain present in
iptables
Root Cause: firewalld reload screws up iptable state (probably, since I've
done it but gathered no evidence beforehand)
Solution: fix firewalld bug, docker is actually helpfully recreating
iptable's 'DOCKER' chain after a sudo systemctl restart docker, which can
be confirmed via sudo iptables -L -n -t nat | grep DOCKER

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/1871#issuecomment-206823204

[TeraHz]

For CentOS 6 the fix is:

$ sudo service docker restart
$ sudo service iptables save

That will save the required chain in the iptables settings and when it reloads it will recreate what's needed.

[timothyparez]

I had the same problem with v1.11.1 on Ubuntu 14.04 solved it like this:

sudo service docker restart
iptables-save > /etc/iptables/rules.v4

This requires iptables-persistent to be installed:

sudo apt-get install iptables-persistent

[denysvitali]

Why was @mayank941 answer "downvoted"?

Removing docker0 with ip link remove docker0 and restarting docker afterward with systemctl restart docker fixed my issues...

[giaosudau]

@mayank941 It works.

[richardkeit]

I was getting the below when trying to expose ports on a container

"Error": "driver failed programming external connectivity on endpoint delete (7a2524372e20fe17adff94a3d327994c41219739afeed4304109cdd2c01954c9): iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.3 --dport 9200 -j ACCEPT: iptables: No chain/target/match by that name.\n (exit status 1)",

To fix this, modify the /etc/systemd/system/docker.service or /etc/systemd/system/docker.service.d/docker.conf file (depends what you're using) to include:
--iptables=false
then
systemctl daemon-reload
service docker restart

[haridsv]

@denysvitali I was able to get this working by running ip link delete docker0. To restart docker, /etc/init.d/docker restart gave me the message Docker is managed via upstart, try using service docker, so I ran sudo service docker restart but it failed with the message Unknown job: docker, so I just ended up killing the dockerd and docker-containerd processes and they got automatically restarted and the iptable entries were also recreated.

[robertorubioguardia]

For CentOS 6 the fix is:

$ sudo service docker restart
$ sudo service iptables save

That will save the required chain in the iptables settings and when it reloads it will recreate what's needed.

This worked like a charm on Alpine + Docker. Thanks!