fail2ban with docker(host mode networking) is making iptables entry but not stopping connections

[rnjailamba]

Environment:

• Fail2Ban version - 1.48
• OS - Debian 9
• [x] Fail2Ban installed via OS/distribution mechanisms - apt-get install -y fail2ban
• [x] You have not applied any additional foreign patches to the codebase
• [x] Some customizations were done to the configuration (provide details below is so)

The issue:

Fail2ban is putting entries in iptables as expected for the jail defined in jail.local as

[xyz_failure]
enabled=true
filter=xyz_failure
logpath=/var/log/xyz_failure.log
maxretry=10
findtime=1800
bantime=604800

The output of iptables -S

-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-N f2b-xyz_failure
-A INPUT -p tcp -m multiport --dports 0:65535 -j f2b-xyz_failure
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A f2b-xyz_failure -m string --string "X-Forwarded-For: 200.194.41.xxx" --algo bm --to 65535 -j REJECT --reject-with icmp-port-unreachable

Steps to reproduce

server with docker installed. load balancer in front of server. install fail2ban.

Expected behavior

The iptables entry should stop the ip from accessing the server

Observed behavior

The ip not stopped from accessing website in browser.

Any additional information

When i open the website as a webview in the ios app the ip is banned correctly. This seems like an issue of established connections not being banned.

Configuration, dump and another helpful excerpts

Any customizations done to /etc/fail2ban/ configuration

Relevant parts of /var/log/fail2ban.log file:

_preferably obtained while running fail2ban with loglevel = 4_

Here are the logs -

2018-12-06 23:45:13,895 fail2ban.actions        [9036]: NOTICE  [xyz_failure] Ban 200.194.41.xxx
2018-12-07 00:01:44,627 fail2ban.filter         [9036]: INFO    [xyz_failure] Found 200.194.41.xxx
2018-12-07 00:01:47,654 fail2ban.filter         [9036]: INFO    [xyz_failure] Found 200.194.41.xxx
2018-12-07 00:01:53,764 fail2ban.filter         [9036]: INFO    [xyz_failure] Found 200.194.41.xxx
2018-12-07 00:01:55,767 fail2ban.filter         [9036]: INFO    [xyz_failure] Found 200.194.41.xxx
2018-12-07 00:01:56,901 fail2ban.filter         [9036]: INFO    [xyz_failure] Found 200.194.41.xxx

Relevant lines from monitored log files in question:

189.188.53.xxx - - [07/Dec/2018:00:07:02 -0600]  User Agent:Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36 HTTP_REFERRER: 
189.188.53.xxx - - [07/Dec/2018:00:07:20 -0600]  User Agent:Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36 HTTP_REFERRER: 
18.208.165.xxx - - [07/Dec/2018:00:08:15 -0600]  User Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36 HTTP_REFERRER:  email: 
187.254.108.xxx - - [07/Dec/2018:00:08:20 -0600]  User Agent:Android app ver 3.3.9 Mozilla/5.0 (Linux; Android 7.0; SM-G950U Build/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/71.0.3578.83 Mobile Safari/537.36 HTTP_REFERRER: 
201.175.202.xxx - - [07/Dec/2018:00:08:20 -0600]  User Agent:Android app ver 3.3.9 Mozilla/5.0 (Linux; Android 8.1.0; Mi A2 Lite Build/OPM1.171019.019; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/68.0.3440.91 Mobile Safari/537.36 HTTP_REFERRER: 

[sebres]

Fail2ban is putting entries in iptables as expected for the jail defined in jail.local
The iptables entry should stop the ip from accessing the server

So if fail2ban works as expected, what would you like to listen from us?
The issue looks like - my iptables does not work as expected resp. does not work across docker container.
The same you would see, if you don't use fail2ban at all and just create the iptables rules manually.

• either you need to try another action working across docker;
• or to configure your docker container network to do it properly;

Anyway, I'll close it as 3rd party issue, for it seems to be.
Please, do not hesitate to provide more info or tell us when or where I'm wrong.

BTW. What is Fail2Ban version - 1.48?
I don't know such version has been ever existing.

[rnjailamba]

@sebres Yes if i put the iptable rules manually they still dont work. Is the issue due of this rule
"-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT".
Is this rule accepting before
"-A f2b-xyz_failure -m string --string "X-Forwarded-For: 200.194.41.xxx" --algo bm --to 65535 -j REJECT --reject-with icmp-port-unreachable"
can ban? If yes how can we put the fail2ban rule above the docker rule?

[sebres]

Is the issue due of this rule "-A FORWARD -o docker0 -m conntrack ..."

I don't think so, because the rule of f2b-xyz_failure placed before your conntrack rule in the main chain, take a look for -A INPUT -p tcp -m multiport --dports 0:65535 -j f2b-xyz_failure.

The last entry from your excerpt -A f2b-xyz_failure -m string ... belong to this rule and will be considered as the rule of INPUT chain above is processed.

Also you are trying here to create a match-rule -m string --string "X-Forwarded-For: <ip>", so it is not IP-related rule, then note that this packet (containing the string) is already processed, so possibly it would not reject the future packets of the remaining connection (see in the documentation of iptables).
Also I doubt that this will work with https (because the content of packets is encrypted).

Perhaps you should try another actions like nginx-block-map etc, which may work on another layer (http-proto or session-based, directly in web-server) behind proxy.

If yes how can we put the fail2ban rule above the docker rule?

Well, you can write your own action or rewrite some available fail2ban-actions like iptables-allports for fail2ban v0.9 or iptables-allports for fail2ban v0.10.
You can also write a derivative action like /etc/fail2ban/action.d/iptables4docker.conf:

[INCLUDES]
before = iptables-allports.conf
[Definition]
actionstart = ... my rewritten action start rules ...
# all other parameters are included unmodified (the same as in iptables-allports.conf).

And use it hereafter in your jail.local:

[my-jail]
banaction = iptables4docker

Anyway as it is a 3rd party issue, try to find a solution in google and/or on some forums like stackoverflow, etc or seek support from the developers/maintainers of iptables/docker.

[rnjailamba]

ok @sebres, ill update here if i get to the root cause.

[alextaujenis]

I'm having this same problem with alpline linux, docker-swarm, nginx, and fail2ban.

1. Docker manages IPTABLES internally.
2. In order to manage the host IPTABLES you need to add the cap_add: NET_ADMIN permission on your docker container.
3. The cap_add permissions cannot be applied to containers in swarm mode.
4. If you are not running in swarm mode, then you can add --cap-add NET_ADMIN to your docker run command to allow the container to control the host IPTABLES (see here).
5. If you are running in swarm mode, then the workaround would be to mount your log files (from your container) to the host OS, install Fail2Ban on the host OS, then point Fail2Ban at the mounted log files.

[hcderaad]

Although this issue is closed, the solution I believe is to add the DOCKER-USER chain to the jail. This will ensure fail2ban rules are indeed included in the docker iptables config in the right order (iptables filters on rule order, so any rule that allows something before the block/drop rule will get precedence. For this the DOCKER-USER chain is available as that is actually evaluated before the container rules).

Adding this line (chain) to the jails sections in jail.local that refer to docker containers solved this issue for me:
[FILTERNAME]
enabled = true
chain = DOCKER-USER

[cljk]

Thank you @hcderaad
That really solved my problem ! I have a sshd docker container writing logs to the host - but the port was never blocked externally.

I found other people with the same problem but only dirty hacks - like adding a special action-rule etc.
The chain parameter is far more elegant.

[cljk]

Since docker is not a so brand new technology and not only used by special nerds.... wouldn´t it be perhaps necessary to add this to the fail2ban manual ... even if the main problem is related to/caused by another software?

I think there are more people using containers faced to the internet who would like to protect the containers with fail2ban on the host ... or (like me) think that´s already done by simply watching the logs.

[juliohm1978]

Dropping by to reaffirm that @hcderaad's advice is very correct, but DOCKER-USER is very specific to Docker alone, and perhaps Docker Swarm.

I'm using Kubernetes to host a number of applications and just came across this while trying to make fail2ban work in a k8s cluster. I had to use the KUBE-FIREWALL chain to get it working.

Whatever fail2ban container you are using, make sure to include a custom action similar the following:

## image entrypoint
cat > /etc/fail2ban/action.d/iptables-common.local <<EOL
[Init]
chain = ${FAIL2BAN_IPTABLES_CHAIN}
EOL

... where FAIL2BAN_IPTABLES_CHAIN is the chain you want all fail2ban rules to fall under.

• DOCKER-USER if you are using plain Docker / Docker Swarm
• KUBE-FIREWALL if you are using Kubernetes