nmap hangs/never finishes a multi-host scan

[blincoln-dd]

This is a persistent issue that I've run into for years, so I can't believe I'm the first person to report it, but I don't see an existing issue for it.

When I run nmap against a single host, it always completes. If I give nmap a set of multiple hosts to scan (using a network/mask, or a list of specific targets with -iL), there is a decent chance it will slow down dramatically and then effectively hang at about 99% completion. This has happened to me numerous times using the Linux and Mac OS builds. If there were a way to force it to output any results it's obtained to that point, it would be less of an issue, but I'm unaware of a way to do that.

My workaround has always been to only run nmap against a single IP, with it run in a while read loop against a list of targets if I have more than one to scan, but I feel like this problem should be addressed in the tool itself.

I have a full process memory dump from the Mac OS version of nmap 7.70 that I can submit privately if it's helpful, but this is something I've observed with literally every version of nmap I've used since I got into security work about 7 years ago. The scan in question was against a list of 87 IP addresses, using the -iL option:

nmap -sV -p1-65535 -Pn -n --open --script=nfs --script=smb-enum* --script=smb-os* --script=smb-ls* --script=smb-server* --script=smb-system* -oA nmap-local_subnets-2018-11-08-01-all_ports -iL hosts-2018-11-08-02.txt

It ran as expected for about an hour, and produced results for four of the hosts. Then it started slowing down dramatically in terms of the estimated time to completion, without producing any additional results.

In order to eliminate the possibility that it was due to a weird host in the list (e.g. newer versions of Windows responding slowly and causing full-port scans to take much longer), I literally disabled all network interfaces on the system just after 20:52 last night. At that time, it was reporting 98.70% complete. At 10:32 this morning (so almost 14 hours later), it had progressed only to 99.09%.

I'm happy to provide any additional information that I can to eliminate this persistent, long-running bug in one of the most useful pen testing tools.

[blincoln-dd]

I'm rerunning essentially the same scan using the following syntax:

while read host; do nmap -sV -p1-65535 -Pn -n --open --script=nfs --script=smb-enum* --script=smb-os* --script=smb-ls* --script=smb-server* --script=smb-system* -oA nmap-$host $host; done<../hosts-2018-11-09-01.txt

There are only 71 hosts in the list today (vs. 87 yesterday), but they're completing in 120-300 seconds each so far. With the worst-case numbers (5 minutes per scan), the original scan should have completed in 8 hours tops, most likely less.

[blincoln-dd]

Finished in < 5 hours when run that way.

[dmiller-nmap]

Thanks for bringing this up! The description of the problem leaves a lot of possibilities open as to the cause, so please help us out by providing some more information:

1. When the scan is "hanging," what scan phase is currently executing? You can determine this by adding --stats-every 30s to your scan options; the statistics line will mention what Nmap is doing and how far it has gotten through that task.
2. Is there a reason you are using -Pn to skip host discovery? Nmap uses this scan phase to initialize some host timeout information, and if any target address is not actually responsive, scanning 65K ports is going to waste a lot of effort. Does the scan finish more quickly if you remove it or use a custom host discovery like -PE -PS80,111,2049,445,137-139 ?

Until we get to the bottom of this, you can try some workarounds:

• Add a --script-timeout to set a limit on how long Nmap lets each NSE script run against a target. Scripts that timeout will not produce output, but all other target info will be preserved. If diagnostic step 1 above shows that NSE is not the problem, this won't help, but it's handy.
• Add a --host-timeout to set an upper limit on how long you want Nmap to spend scanning a particular target. All results for that target will be lost if the timeout is reached, but you will at least get all results for your other targets and can re-scan the troublesome ones individually.

[blincoln-dd]

Thank you. I'll give that a try and add an update later about how it changes the results.

I'm using -Pn because in my experience, response to ICMP traffic is not a reliable indication of whether or not a host has services listening (especially for scanning outside of the local subnet, and especially if the traffic is traversing the Internet or going into a restricted network). Obviously if something is pingable, it's worth looking at, but I don't want to not look at something just because I can't ping it. Scanning all ports will definitely take awhile if there really is no IP address there, but spending an extra 5-10 minutes scanning each IP is worth it (IMO) in order to find more systems.

Without the additional flags, this is the output leading up to my ctrl-C'ing out of the process:

SYN Stealth Scan Timing: About 98.55% done; ETC: 15:22 (0:03:02 remaining)
Stats: 3:36:11 elapsed; 6 hosts completed (64 up), 60 undergoing SYN Stealth Scan
SYN Stealth Scan Timing: About 98.56% done; ETC: 15:25 (0:03:04 remaining)
Stats: 3:45:25 elapsed; 6 hosts completed (64 up), 60 undergoing SYN Stealth Scan
SYN Stealth Scan Timing: About 98.56% done; ETC: 15:34 (0:03:12 remaining)
Stats: 4:13:57 elapsed; 6 hosts completed (64 up), 60 undergoing SYN Stealth Scan
SYN Stealth Scan Timing: About 98.57% done; ETC: 16:03 (0:03:35 remaining)
Stats: 4:15:55 elapsed; 6 hosts completed (64 up), 60 undergoing SYN Stealth Scan
SYN Stealth Scan Timing: About 98.57% done; ETC: 16:05 (0:03:36 remaining)
Stats: 6:03:16 elapsed; 6 hosts completed (64 up), 60 undergoing SYN Stealth Scan
SYN Stealth Scan Timing: About 98.63% done; ETC: 17:54 (0:04:58 remaining)
Stats: 6:41:50 elapsed; 6 hosts completed (64 up), 60 undergoing SYN Stealth Scan
SYN Stealth Scan Timing: About 98.64% done; ETC: 18:33 (0:05:26 remaining)
Stats: 7:01:34 elapsed; 6 hosts completed (64 up), 60 undergoing SYN Stealth Scan
SYN Stealth Scan Timing: About 98.65% done; ETC: 18:53 (0:05:40 remaining)
Stats: 8:12:29 elapsed; 6 hosts completed (64 up), 60 undergoing SYN Stealth Scan
SYN Stealth Scan Timing: About 98.69% done; ETC: 20:05 (0:06:27 remaining)
Stats: 8:25:51 elapsed; 6 hosts completed (64 up), 60 undergoing SYN Stealth Scan
SYN Stealth Scan Timing: About 98.69% done; ETC: 20:18 (0:06:36 remaining)
Stats: 8:37:41 elapsed; 6 hosts completed (64 up), 60 undergoing SYN Stealth Scan
SYN Stealth Scan Timing: About 98.70% done; ETC: 20:30 (0:06:44 remaining)
Stats: 8:59:00 elapsed; 6 hosts completed (64 up), 60 undergoing SYN Stealth Scan
SYN Stealth Scan Timing: About 98.70% done; ETC: 20:52 (0:07:00 remaining)
Stats: 21:51:08 elapsed; 6 hosts completed (64 up), 60 undergoing SYN Stealth Scan
SYN Stealth Scan Timing: About 99.08% done; ETC: 09:49 (0:12:11 remaining)
Stats: 22:30:33 elapsed; 6 hosts completed (64 up), 60 undergoing SYN Stealth Scan
SYN Stealth Scan Timing: About 99.09% done; ETC: 10:29 (0:12:23 remaining)
Stats: 22:34:02 elapsed; 6 hosts completed (64 up), 60 undergoing SYN Stealth Scan
SYN Stealth Scan Timing: About 99.09% done; ETC: 10:32 (0:12:24 remaining)
Stats: 22:46:02 elapsed; 6 hosts completed (64 up), 60 undergoing SYN Stealth Scan
SYN Stealth Scan Timing: About 99.09% done; ETC: 10:44 (0:12:27 remaining)
Stats: 23:15:10 elapsed; 6 hosts completed (64 up), 60 undergoing SYN Stealth Scan
SYN Stealth Scan Timing: About 99.11% done; ETC: 11:13 (0:12:31 remaining)
Stats: 25:09:43 elapsed; 6 hosts completed (64 up), 60 undergoing SYN Stealth Scan
SYN Stealth Scan Timing: About 99.16% done; ETC: 13:08 (0:12:42 remaining)

[dmiller-nmap]

Yeah, it's grinding slower and slower on the port scan. 65K ports on 60 targets is a lot to get through, and something is causing it to get slower more rapidly than it can complete ports. Disconnecting the network will actually make Nmap go slower, not faster, so that's not a good way to force the scan to finish. I'm tracking that issue separately: #1394.

-Pn turns off all host discovery, not just ICMP Echo Request. For historical reasons, this is frequently referred to as "ping scanning," but I've been trying to improve the documentation to address this confusion. The default host discovery probes are -PE (ICMP Echo Request), -PS443 (TCP SYN port 443), -PA80 (TCP ACK port 80), and -PP (ICMP Timestamp Request). As I suggested earlier, you can come up with your own assortment of host discovery probes. Even if you plan to do a -Pn scan on all "down" targets, doing so separately ought to improve overall speed.

[spencerroth3]

Was there ever a final resolution to this issue? It is 2 years later, and I am still facing the same issues as @blincoln-dd