Nmap: "http-config-backup" and servers responding with large garbage files to any request

[jjgh]

While I was scanning an IP range with Nmap using the "http-config-backup" script (which scans a web servers for left-behind configuration files) Nmap was outputting many lines like this

NSE: [http-config-backup 123.123.123.123:80] 123.123.123.123: found but not matching: /mediawiki/LocalSettings.php.bak

Each request was taking approximately 30 seconds, so I pointed the browser to http://123.123.123.123/mediawiki/LocalSettings.php.bak and I found out that the server was answering to any request (any file name after the / ) by sending each time a 10 Mb large garbage file.
This was tremendously slowing down Nmap, considering that it was also happening for other servers in the range.

What do you guys think about this? Do you think it's a tactic that a web servers use to slowdown a scan? Is there a way I could make Nmap ignore this kind of servers?

Thank you

[dmiller-nmap]

Thanks for pointing this out. I've made a change in r36024 (4fa478c) that should help in this regard. It won't necessarily prevent the large responses, but it will ensure that http-config-backup doesn't save things that are not config files. I used the http.identify_404 function to determine if we can distinguish pages that exist from those that don't. If we can't, then the script will fail immediately. This is a robust approach that can even identify custom-404 pages returned with a 200 response code, but it does require requesting each page. So if the server wants to return 10MB per request, we can't do much about it.

I'd appreciate your thoughts on this, if you have any better ideas.

[nnposter]

In addition to Dan's fix, web scanners typically implement a configurable response size limit (typically 1-2 MB by default) to avoid this choking. They bail out from processing a request when Content-Length is too high or when the actual length exceeds the limit.

It seems to me that other NSE HTTP probing code (like httpspider) could benefit from it.

[dmiller-nmap]

@nnposter Good idea! I put in an Issue for that over at #473.

[nnposter]

Response body size limit implemented in r37627 (df26932). See #1571 for details/discussion.