Nmap: Unable to connect to weak-DH servers with OpenSSL: "dh key too small"

[dmiller-nmap]

Nmap has a goal of being able to talk to anything, no matter how insecurely. We're hindered by OpenSSL's goal of making only secure communications possible. Now connecting to a server with a 768-bit DH key is impossible. We should try every trick possible to connect properly. One possibility is adding !DH to the cipher preference list to avoid DH ciphersuites.

[dmiller-nmap]

Specific error message:

NSOCK INFO [8.8650s] handle_connect_result(): EID 41 error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small

[nnposter]

IMHO removing support for classic DH might turn out too tactical. I do recall running into sites that insisted on it.

This issue falls into the same general category as #511. I wonder if a more robust solution might be to attempt several different handshakes and remember which one worked.

Judging by signals coming from the OpenSSL maintainers, at some point Nmap might also require a custom build of a TLS library, foregoing a shared instance, along the lines of enable-weak-ssl-ciphers in OpenSSL Configure.

[nnposter]

This issue is becoming more prominent with Kali and Debian configuring default OpenSSL security level as "2", i.e. requiring at least 2048-bit DH keys.

Fortunately there is a much more direct method to fix this issue (for 1024-bit DH keys), something along these lines:

--- a/nsock/src/nsock_ssl.c
+++ b/nsock/src/nsock_ssl.c
@@ -101,6 +101,9 @@
   SSL_CTX_sess_set_cache_size(ctx, 1);
   SSL_CTX_set_timeout(ctx, 3600); /* pretty unnecessary */

+  /* Make sure that we are not picky about weak crypto */
+  SSL_CTX_set_security_level(ctx, 0);
+
   return ctx;
 }

To be noted, the nsock library is being used by both nmap and ncat so the appropriate way to deal with this might be to lower the security level only for nmap.

A quick workaround, not requiring recompilation, is to run nmap with custom OpenSSL configuration:

sed -re 's/@SECLEVEL=[2-9]//' /etc/ssl/openssl.cnf >~/openssl_nmap.cnf
export OPENSSL_CONF=~/openssl_nmap.cnf
nmap -sV....

[cnotin]

This looks nice. I'll test it and let you know.

[cnotin]

It works fine!

[nnposter]

Using OpenSSL security level 0 is no longer sufficient. At some point OpenSSL hard-coded a check in their default security callback function that EDH keys must be at least 1024 bits even at level 0. For those that need to be able to run scripts against old hardware, the following OpenSSL patch removes the EDH check:

--- a/ssl/ssl_cert.c    2020-12-08 06:20:59.000000000 -0700
+++ b/ssl/ssl_cert.c    2020-12-12 13:15:27.920221911 -0700
@@ -891,9 +891,9 @@
         /*
          * No EDH keys weaker than 1024-bits even at level 0, otherwise,
          * anything goes.
-         */
         if (op == SSL_SECOP_TMP_DH && bits < 80)
             return 0;
+         */
         return 1;
     }
     if (level > 5)