Fail2ban: MJ12bot - try to block

[arekstuka]

Hi All

This bot MJ12bot banging my server, (i have lot of "connection's" from this bot), i want block it by fail2ban, please take a look on log below and help me with fileregex

i did already this (but not working)

http://www.howtoforge.com/forums/archive/index.php/t-61404.html

log from my server:

blog.mywebsite.com:80 83.149.126.98 - - [27/Apr/2014:08:20:45 +0200] "GET /robots.txt HTTP/1.0" 404 2091 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)"

[yarikoptic]

see http://sourceforge.net/p/fail2ban/mailman/message/30585232/ and actually create robots.txt if you want to have legit bots stop "banging" ;-)

[arekstuka]

Right :) but i want to block this bot on my firewall. Do you have any idea how to block it on fail2ban ??

[miohtama]

Please use mailing list for asking help https://lists.sourceforge.net/lists/listinfo/fail2ban-users - GitHub issue tracker is mainly for tracking software bugs.

[yarikoptic]

look into config/filter.d/apache-badbots.conf

On Thu, 01 May 2014, Arek wrote:

Right :) but i want to block this bot on my firewall. Do you have any idea
how to block it on fail2ban ??

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate, Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: [phone] Fax: [phone]
WWW: http://www.linkedin.com/in/yarik

[arekstuka]

Hi Yaroslav
i just simple add on the end of line (MJ12bot)

like this : badbotscustom = EmailCollector|WebEMailExtrac|TrackBack/1.02|sogou music spider|MJ12bot ,

but this bot still have access to my server...

[sasha2002]

You need to put entire line :
Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
this is regexp :
Mozilla/5\.0 \(compatible; MJ12bot/v1\.4\.5; http\://www\.majestic12\.co\.uk/bot\.php\?+\)

[gagomap]

No, it don't work. I still can't block MJ12bot by fail2ban

[sasha2002]

@gagomap what is your browser agent in logs?
Do you monitor logs?

[gagomap]

Yes, i monitor logs everyday. I can block them by using nginx server block, but i can't do it with fail2ban.

[sasha2002]

can you paste browser agent from logs and your fail2ban config to some site like pastebin

[gagomap]

Yes, this is my logs.
http://pastebin.com/XGyhsNmd

I use apache-badbots filter.

And this is my jail.local

[nginx-badbots]

enabled  = true
port     = http,https
filter   = apache-badbots
logpath  = /var/log/nginx/*access.log
maxretry = 1
bantime  = 86400

My site is in maintenance mode.

[sasha2002]

Can you email me at [email protected] ?

[sebres]

apache-badbots over nginx access log-file?
They have different formats, and according to a different failregex.
Atm for nginx we have only botsearch filter.
To use it with nginx-log, you can customize apache-badbots, implements your own nginx-badbots or just rewrite failregex in your nginx-badbots jail.
For example see differences between https://github.com/fail2ban/fail2ban/blob/master/config/filter.d/nginx-botsearch.conf and https://github.com/fail2ban/fail2ban/blob/master/config/filter.d/apache-botsearch.conf

[gagomap]

Thanks,

[gagomap]

Hi,
I can block MJ12bot with:

failregex = ^<HOST> .* "GET .* \(compatible\; MJ12bot\/v1\.4\.5; http:\/\/www\.majestic12\.co\.uk\/bot\.php\?\+\)"$

But i want to block alot of bots like apache-badbots,
I try many times but i still can't :D

[sasha2002]

@gagomap you can try to use this :
https://github.com/exrat/rutorrent-essential/blob/master/files/fail2ban/nginx-badbots.conf
and here you can see how to make more regexp :)

[gagomap]

Thanks, I had it in my filters.

[gagomap]

I have modified apache-badbots for nginx

# Nginx badbots v1.0
[Definition]

badbots = MJ12bot|AhrefsBot
failregex = ^<HOST> .* "(GET|POST|HEAD) .* (%(badbots)s).*"

# Option: ignoreregex
# Notes.: regex to ignore. If this regex matches, the line is ignored.
# Values: TEXT
#
ignoreregex =

Result:

Running tests
=============

Use   failregex file : test.conf
Use         log file : /var/log/nginx/access.log


Results
=======

Failregex: 314 total
|-  #) [# of hits] regular expression
|   1) [157] ^<HOST> .* "(GET|POST|HEAD) .* (MJ12bot|AhrefsBot).*"
|   2) [157] ^<HOST> - .* "-" ".*(MJ12bot|AhrefsBot).*"
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [184] Day/MONTH/Year:Hour:Minute:Second
`-

Lines: 184 lines, 0 ignored, 157 matched, 27 missed
Missed line(s): too many to print.  Use --print-all-missed to print all 27 lines

Sample log:

154.87.1.102 - - [30/Oct/2015:18:00:21 +0700] "GET / HTTP/1.1" 200 826 "-" "-"
193.164.216.238 - - [30/Oct/2015:18:01:35 +0700] "GET /robots.txt HTTP/1.0" 404 162 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)"
188.165.15.99 - - [30/Oct/2015:18:02:51 +0700] "GET /author/admin/feed HTTP/1.1" 404 134 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"
162.210.196.130 - - [30/Oct/2015:18:26:04 +0700] "GET /robots.txt HTTP/1.0" 404 162 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)"
188.165.15.99 - - [30/Oct/2015:19:13:37 +0700] "GET /tag/ca-tam-sapa HTTP/1.1" 404 134 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"
188.165.15.99 - - [30/Oct/2015:20:45:30 +0700] "GET /tag/du-li%CC%A3ch-sapa-tranh-nang-he/feed HTTP/1.1" 404 134 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"

@sebres : Can you create a new filter name "nginx-badbots" base on my regex ? Because a lot of people use apache-badbots filter for nginx, but it doesn't work.

[sebres]

@gagomap done in #1242, but it is build relative #1241

[gagomap]

Thanks,

[benyaminl]

anyway MJ12 have change ughh and it hits many server very hard, even after set the robot.txt