fail2ban network

[szepeviktor]

The same botnet is attacking different servers administered by me.
Is there a way to syncronize banned IPs?
Thus banning the other server's attackers.

[buanzo]

Check my blog. I started an article about sharing ban data. You might be
able to implement it but will require you to write a couple scripts.

http://blogs.buanzo.com.ar
On Dec 7, 2014 7:16 AM, "Viktor Szépe" [email protected] wrote:

The same botnet is attacking different servers administered by me.
Is there a way to syncronize banned IPs?
Thus banning the other server's attackers.

—
Reply to this email directly or view it on GitHub
https://github.com/fail2ban/fail2ban/issues/881.

[yarikoptic]

On Sun, 07 Dec 2014, buanzo wrote:

Check my blog. I started an article about sharing ban data. You might be
able to implement it but will require you to write a couple scripts.
http://blogs.buanzo.com.ar

This one also looks interesting
https://github.com/mmunz/fail2ban-p2p
although I do not see any recent activity and haven't tried it myself
yet

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Research Scientist, Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: [phone] Fax: [phone]
WWW: http://www.linkedin.com/in/yarik

[buanzo]

Thanks Yaroslav!

I had started writing an ad hoc solution called fail2ban-cluster but ended
up supporting the actions architecture native in f2b. Please keep us posted!
On Dec 7, 2014 12:41 PM, "Yaroslav Halchenko" [email protected]
wrote:

On Sun, 07 Dec 2014, buanzo wrote:

Check my blog. I started an article about sharing ban data. You might be
able to implement it but will require you to write a couple scripts.
http://blogs.buanzo.com.ar

This one also looks interesting
https://github.com/mmunz/fail2ban-p2p
although I do not see any recent activity and haven't tried it myself
yet

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Research Scientist, Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: [phone] Fax: [phone]
WWW: http://www.linkedin.com/in/yarik

—
Reply to this email directly or view it on GitHub
https://github.com/fail2ban/fail2ban/issues/881#issuecomment-65941210.

[szepeviktor]

@yarikoptic Do have any intent to add a feature of "sending/receiving remote ban request"?

[yarikoptic]

Unfortunately I have no intent to invest any major time of coding new features for fail2ban in near future, but I would be glad to review a PR (the smaller/the more functional the better ;))

[sebres]

I have a small f2b-extenstion doing similar, but:

• currently it works only with/over my observer module, what included in PR #716 (branch sebres:ban-time-incr); and makes sense only there, because ...
• that shares bad ips in fail2ban.db between all registered hosts, but not bans on other servers if no failure there occurred (only table "bips" will be shared - faster ban of this bad ips by first failure);
• I have no intent to make it public, as long as #716 is not accepted (have already enough problems with master merge, etc.);

[szepeviktor]

A list of pros/cons of pull and push notifications would be helpful.
No. Only the push model could be quick enough.

[szepeviktor]

@sebres Does your solution work throuhg an open TCP port on the client with public/private key signing?

[sebres]

@szepeviktor it is configurable, at the moment is possible:

• custom command to tranfer a bulk stream to f2b-client on each target server;
• ssh with public/private key (or without it, if you will use an ip white list internally);
• (not yet fully tested) tcp/udp port with ticket by connect (listener + openssl encrypted packet with signature by connect); But i'm not sure whether I want to continue supporting this method;

[szepeviktor]

The simplest way is to append to a log file over SSH. In Bash:

echo "Host to ban: 1.2.3.4" | ssh -i private.key [email protected] "cat >> from-host-xxxxx.log"

And set up a filter with retrycount = 1.

[szepeviktor]

I think it could be done without (python) coding.
Just adding filters, actions here and there.

[sebres]

I think it could be done without (python) coding. Just adding filters, actions here and there.

Yep, and have an endless cycle (as soon as each server forwards it further in cluster)...

[szepeviktor]

:smile:

[szepeviktor]

Maybe if I set action to _iptables+sendmail+distribute-ban_ for all jails except for the special jail:
_iptables_.

[buanzo]

if you can ssh then fail2ban-client banip ....
On Dec 8, 2014 7:22 PM, "Viktor Szépe" [email protected] wrote:

The simplest way is to append to a log file over SSH. In Bash:

echo "Host to ban: 1.2.3.4" | ssh -i private.key other.host "cat >> immediate-ban.log"

And set up a filter with retrycount = 1.

—
Reply to this email directly or view it on GitHub
https://github.com/fail2ban/fail2ban/issues/881#issuecomment-66199257.

[sebres]

@szepeviktor Yes, of course it will work, but it is rather a workaround as solution.
Think for example about of any registered server in cluster may be down or by possible network problems etc (your action will block the executing thread in f2b up to connect timeout).
If the server started up again, it should share all known bad ips from any other server, etc.
My solution is free from this children's diseases, more "creative", for example works asynchronous, sends a bulk packets (have a time latency) etc.
@buanzo Yes, but:

• you need a ssh with root access (compared to proposal above - ssh with user, just allowed to write log-file);
• banip only for jail, that not have an action _distribute-ban_, otherwise endless cycle;

[buanzo]

what about updating a dnsbl and having fail2ban query it?

off the top of my head.
On Dec 8, 2014 8:25 PM, "Serg G. Brester" [email protected] wrote:

@szepeviktor https://github.com/szepeviktor Yes, of course it will
work, but it is rather a workaround as solution.
Think for example about of any registered server in cluster may be down or
by possible network problems etc (your action will block the executing
thread in f2b up to connect timeout).
If the server started up again, it should share all known bad ips from any
other server, etc.
My solution is free from this children's diseases, more "creative", for
example works asynchronous, sends a bulk packets (have a time latency) etc.
@buanzo https://github.com/buanzo Yes, but:

• you need a ssh with root access (compared to proposal above - ssh
  with user, just allowed to write log-file);
• banip only for jail, that not have an action _distribute-ban_,
  otherwise endless cycle;

—
Reply to this email directly or view it on GitHub
https://github.com/fail2ban/fail2ban/issues/881#issuecomment-66207768.

[szepeviktor]

if you can ssh then fail2ban-client banip ....

Only root can issue fail2ban-client.

[szepeviktor]

DNSBL is a good idea. It is very easy to query.
How would you send the data to it? SSH?

[szepeviktor]

I have an unbound install. Look, it can be remote controlled:
https://www.unbound.net/documentation/unbound-control.html

[buanzo]

has everyone forgotten about sudo and its companion /etc/sudoers ?
On Dec 8, 2014 8:55 PM, "Viktor Szépe" [email protected] wrote:

if you can ssh then fail2ban-client banip ....

Only root can issue fail2ban-client.

—
Reply to this email directly or view it on GitHub
https://github.com/fail2ban/fail2ban/issues/881#issuecomment-66211290.

[opoplawski]

FWIW - I handle this by configuring fail2ban to insert a drop rule into our main firewall. - ssh - trusted keys.

[grinapo]

I don't want to disturb the waters but I usually prefer _secure_, _reliable_ and _flexible_ transport between servers. There are a few message passing solutions (esp. if you're already possess some clustering running) but the simplest out of the top of my head is serf (https://www.serfdom.io/) where you basically setup a mesh of serf nodes (either autodetected or manual, multicast or unicast) then use the client a a few simple "scripts" to broadcast bans and unbans. Doesn't care if servers are offline, can queue up messages or not, etc. And pretty simple to set up.
[Note to self: someday I shoulda follow my own advice and set it up. :-P]

[buanzo]

serf is quite awesome indeed.

but if you need to keep things simple, and also if you do not necessarily
have an established relationship between nodes (for instance,
self-organized attack prevention/detection human group), then
secure/reliable/flexible is https.

On Fri, Jan 9, 2015 at 12:31 PM, grinapo [email protected] wrote:

I don't want to disturb the waters but I usually prefer _secure_,
_reliable_ and _flexible_ transport between servers. There are a few
message passing solutions (esp. if you're already possess some clustering
running) but the simplest out of the top of my head is serf (
https://www.serfdom.io/) where you basically setup a mesh of serf nodes
(either autodetected or manual, multicast or unicast) then use the client a
a few simple "scripts" to broadcast bans and unbans. Doesn't care if
servers are offline, can queue up messages or not, etc. And pretty simple
to set up.
[Note to self: someday I shoulda follow my own advice and set it up. :-P]

—
Reply to this email directly or view it on GitHub
https://github.com/fail2ban/fail2ban/issues/881#issuecomment-69349223.

[leeclemens]

I'd like to add that it is not that two nodes don't trust each others' identities (https - Authentication), but that they need to determine their Authorization (to 'post' data).

[leeclemens]

The nsupdate action may meet the goals, secure (uses a tsig for auth), reliable, flexible, https://github.com/fail2ban/fail2ban/pull/925

[klepsydra]

My solution to this was simpler than a privately distributed p2p ban list.

My script simply applies the global attacker lists (sources are configurable, using a list of URLs) which my servers using fail2ban also contribute to as well via my blocklist.de account:

http://j.mp/syncFail2ban

So in that sense the whole world is my P2P fail2ban network.

If other servers around the world get attacked, and the global blacklists find out about their IP addresses, then my servers will also block that addresses, for at least a day, until they get cleared from the global blacklists.

My (re)search culminated at
http://serverfault.com/a/648087 until I just deployed my unofficial, but pretty reliable and simple solution.

[szepeviktor]

@klepsydra What had you got -5 on serverfault?

[klepsydra]

@szepeviktor Apparently the question landed in a forum I thought was for network/system admins, but not really. Did you read their comments etc. to find out why?

[szepeviktor]

After reading those I setup somethin like an RBL but text based.
A central list of IP-s and IP ranges GPG signed and an hourly cron job on every server I administer.
I'll post the script here.

[buanzo]

You could use the output from fail2ban-client status and fail2ban-client
status plus some bash scripting to get a nice list of currently
banned IPs, indeed.

On Mon, Feb 16, 2015 at 7:22 PM, Viktor Szépe [email protected]
wrote:

After reading those I setup somethin like an RBL but text based.
A central list of IP-s and IP ranges GPG signed and an hourly cron job on
every server I administer.

—
Reply to this email directly or view it on GitHub
https://github.com/fail2ban/fail2ban/issues/881#issuecomment-74581587.

[szepeviktor]

It would be nice to have parsable output.
https://github.com/fail2ban/fail2ban/issues/861

[szepeviktor]

For example after calling BT Italia, I've found out that they do not have an abuse department/don't care about abuse, so I've block their network.
So it is not only about already banned IP-s.

[sebres]

just as reference - #238

[szepeviktor]

See https://github.com/stintel/vallumd !

[sebres]

Well, this one looks nice, but would be non-native for fail2ban...
Not to mention, the usage of MQTT-broker for such purposes is IMHO a bit... strange.

[benrubson]

I just did it using badips.py :

In jail.local, to report bad IPs :

action = %(action_)s
         badips.py[category="%(badips_category)s"]

In jails, for example jail.d/sshd.conf, to set badips_category :
_(sometimes the jails' name, set by default in jail.conf, does not correspond to a badips category)_

[sshd]
badips_category = ssh

In jail.d/mybadips.conf, to get and ban IPs my servers banned :

[mybadips]
enabled = true
port =
protocol = all
filter =
action = %(action_)s
         badips.py[category="unknown", bancategory="any", banaction="iptables-allports", score="0", bankey="the_key_you_get_below_(*)"]

In jail.d/combadips.conf, to get and ban IPs community banned :

[combadips]
enabled = true
port =
protocol = all
filter =
action = %(action_)s
         badips.py[category="unknown", bancategory="any", banaction="iptables-allports", score="3"]

(*) You'll have to get your own bankey which will be assigned to your server's IP :

curl https://www.badips.com/add/ssh/1.2.3.4
curl https://www.badips.com/get/key

Then, from each of your other servers, assign your key to their IP :

curl https://www.badips.com/add/ssh/1.2.3.4
curl https://www.badips.com/set/key/the_key_you_get_above

(**) bancategory="any" requires #2056.

And it works perfectly 👍
Nice things with this solution is that this is a community one, helping to track and ban big contributors.

[sebres]

And it works perfectly

Note that this work only if all your servers have the same IP (because the key is IP related).
From the badips-docu:

... /set/key/\ This can only be done from a reporting IP.

So I assume it is not possible to share private list using multiple servers (with different IPs).

[benrubson]

You can share the same key on different servers, having different IPs.
The last 2 commands I give allow to attribute the key to a second, third... IP.

This can only be done from a reporting IP.

Means that you first have to report once from the server you will set the key for.
So : curl https://www.badips.com/add/ssh/1.2.3.4
Yes, does not really make sense, but it works...

[sebres]

Hmm.... strange.
I tried it, but I have not got the IP "published" from one server, calling it hereafter from another with the same key (badips just returns empty list).

[onlyjob]

IMHO the elegant solution to this problem might be a shared backend storage for banned IPs.

I would probably use Consul which have a simple API for key/value store.

[sebres]

Sure, just writing this as a new backend (in sense of fail2ban) would imply active-passive cluster (1/N)...
At least unless fail2ban does not support two backends per jails.
Whereas an additional notification mechanism somewhere in actions area could be used for active-active approach.

BTW, since #2351 got implemented we have even an API (client / pickle) to notify about multiple bans with a single command, so something like this could be good enough:

• new pythonic action (e. g. notify-network) can be implemented to accumulate banned tickets per jail and send it to the network if it reached some latency (e. g. 5 seconds) or count (e. g. 10 bans);
  it could look like ssh $connargs fail2ban-client set <jain> banip $ips for each host in net cluster;
• ticket can be extended with new flag (like manual or remote) or even already available flag restored can be used in order to avoid notifying a ban further in cycle (to stop periodic loop);
• action notify-network can then ignore such tickets similar norestored handling (implemented in #1669);

And a possible configuration for that would be pretty simple too:

[DEFAULT]
action_ = %(known/action_)s
          notify-network.py[hosts="servers1, ..., serverN",
                            connect="ssh -o 'IdentitiesOnly=yes' -i /path/to/key f2b@$host $cmd]

How does this roadmap sound?

But I still hope to provide it in my new observer variant (will be 0.12 or 1.0).

[sebres]

PR #2351 also introduced new command/API to inform fail2ban about a single attempt, so theoretically even every failure can be shared across the network.

[benrubson]

For now, for this purpose, I still use BadIPs as explained above, and it works perfectly.
Perhaps sometimes BadIPs times-out, but does not prevent this solution from working.

IMHO the elegant solution to this problem might be a shared backend storage for banned IPs.
I would probably use Consul which have a simple API for key/value store.

What about a simple PHP/SQL (SQlite / MySQL...) backend ?
With basic authentication.
Would rather be easy to develop, to maintain, to deploy...

As shown above for BadIPs, we would then simply have to add a default action to report the banned IP to this backend, and add an additional jail to periodically retrieve & ban the IPs from the backend.

Something like this could be good enough:
(...)
And a possible configuration for that would be pretty simple too:

[DEFAULT]
action_ = %(known/action_)s
          notify-network.py[hosts="servers1, ..., serverN",
                            connect="ssh -o 'IdentitiesOnly=yes' -i /path/to/key f2b@$host $cmd]

How does this roadmap sound?

Every time the action is triggered, so every time a server bans an IP, it would have to SSH-contact all other servers to share this ban.
It has the advantage to be sort of "real-time" solution compared to the above one.
But IMO it does not look optimized, especially if there are many servers to share the banned list, as it implies N SSH connections.
In addition, SSH seems rather a heavy solution fo this purpose.
Some servers could also not be reachable through SSH, seems easier to pull than to push.
Finally, seems hard to maintain, as every time a server is added, we would have to edit Fail2Ban configuration on every other server to add the new one to the list...
My thoughts :)

[sebres]

Every time the action is triggered, so every time a server bans an IP, it would have to SSH-contact all other servers to share this ban.

Nope, read attentive what I wrote:

new pythonic action (e. g. notify-network) can be implemented to accumulate banned tickets per jail and send it to the network if it reached some latency (e. g. 5 seconds) or count (e. g. 10 bans);

Such latency and/or min count avoid that it'd happen for every ban.
Anyway nobody hinder you from implementing that more gently, e. g. by holding the ssh connection(s) for certain time (timeout) open.

[benrubson]

Such latency and/or min count avoid that it'd happen for every ban.

Yes sorry, not for every ban, but for a pool of bans.
The server would however still have to contact every other server once the pool is "full".

Anyway nobody hinder you from implementing that more gently, e. g. by holding the ssh connection(s) for certain time (timeout) open.

Sure, I was just sharing my thoughts regarding this SSH-based solution :)

[drzraf]

@sebres , I think your comment at https://github.com/fail2ban/fail2ban/issues/881#issuecomment-572524493 is the right way to go.

One comment still: I think that no one would bring full ssh control across hosts for the sole purpose of sharing f2b lists. That means that:

• hosts, IP, aliases hostkeys, would be provisioned beforehand (eg ansible)
• restricted ssh-command for a given key would be provisioned before hand and locked to fail2ban-client set <jail> banip <ips> (with jail and <ips> being (safely) derived from $SSH_ORIGINAL_COMMAND using a minimal wrapper)

So that notifier.py only needs to know about servers hostnames/alias and IPs to ban (and possibly the name of the jail), given that the rest is up to ssh configuration.
(Eg paramiko can do SSH connection from python while considering existing SSH configuration)

About loops, isn't possible for an action script to know whether it's been called from command-line or from parsing files? Is so nested loops could be avoided without introducing another complex locking mechanism.

Do you already have an implementation?

[sebres]

I think that no one would bring full ssh control across hosts for the sole purpose of sharing f2b lists

If the emphasis on "full", then OK. Otherwise I don't know why millions people give the access to git, sftp, etc pp via ssh. :)

Additionally note that the question how the data reach the "network" (synchronized across several hosts) is secondary unless the basic functionality is not given and if the prerequirements for active-active "cluster" synchronization mechanisms are implemented, it does no matter at all.

About loops, isn't possible for an action script to know whether it's been called from command-line or from parsing files?

At the moment, no... it does not distinguish between manual and regular ban at the action level... but it can be changed (relative simple thing, similar our restored flag).

Do you already have an implementation?

Conditionally... Because my (other variant of) fail2ban is a part of my IDS system that would allow that, just it is totally incompatible with original, so we are rather speaking about a fail2ban clone (or rather forked version).
And a backporting would cost many time and effort... and not the test cases solely is the large problem hereafter, although the code coverage (with many artificial cases) is really often bothering a future development.
But my most issue (as of many of us) is always lack of time.

[apassi]

I just created POC, i will send all Ban messages with rsyslog to another servers, which reads the fail2ban.action messages to /var/log/fail2ban_from_cloud.log file, and i created fail2ban filter which reads that file. Filter has simple regex

]\ Ban\ \

and maxretry=1, Seems to work very well.

[gnanet]

Kind of cool solution. Question is, how this transport allows multiple members to send their bans, and also retrieve the bans from the "central".
I have already some ideasfor that, but i am curious what you think

[apassi]

I dont have a central. Each node sends the message for every other node. The receiver's rsyslog has filter with test the source host of the message and if its from cloud, it add it to log file A, and if its from local, it will write it to file B AND to network. That prevents the flooding of the same message withing nodes.
I have mesh network implemented with tinc vpn, so i each node is as individual as could be.
Rsyslog configurations and fail2ban configurations is built with ansible, so it's not even hard to maintain.

[apassi]

Also, there is no danger if central drops from network, the fail2ban cluster will hang.