Icinga2: SSL errors with leading zeros in certificate serials (created < v2.4) with OpenSSL 1.1.0

[Crunsher]

Certificate serials may not start with leading 0s since openssl 1.1.0, they should(tm) not have any in 2.7.0 but we should check just to make sure.

Further reading: http://openssl.6102.n7.nabble.com/openssl-org-4301-BUG-OpenSSL-1-1-0-pre2-fails-to-parse-x509-certificate-in-DER-format-td63467.html

[widhalmt]

@Crunsher Is this just to make sure or did you see Icinga 2 nodes having problems to start with such a serial number? I do have one Testhost which doesn't start and the error messages indicate it might have something to do with the certificates serial but I'm still searching so I didn't open a new issue.

I'm just curious if you have the same problems or if this is just a precaution.

[dnsmichi]

Precaution based on the ticket we've discussed yesterday. It is not clear if this really is a bug or not.

[dnsmichi]

https://monitoring-portal.org/index.php?thread/41664-ca-crt-verification-error-with-openssl-1-1-0-illegal-zero-content-in-field-seria/

We should verify that 2.7.x does not create invalid ca certificates. @widhalmt please share your findings from your tests once you have them.

[dnsmichi]

Edit 2017-09-04: The most recent analysis and fixes are described here: https://www.icinga.com/2017/08/30/advisory-for-ssl-problems-with-leading-zeros-on-openssl-1-1-0/

How do I check that I am affected

Copy ca.crt, client.crt to that temporary container. Extract them on your box in base64 encoded strings which helps with copy paste.

openssl x509 -text -in ca.crt
openssl x509 -text -in client.crt

Verify the certificates on a node where openssl 1.1.0 is installed (Debian Stretch for example).

docker run -ti debian:stretch bash
apt-get update
apt-get -y install openssl vim

Details: https://www.icinga.com/docs/icinga2/latest/doc/15-troubleshooting/#cluster-troubleshooting-ssl-certificate-verification

openssl verify -verbose -CAfile ca.crt client.crt

openssl x509 -text -in ca.crt
openssl x509 -text -in client.crt

Versions involved

This likely affects the CA and client certificates created in ~2015.

Possibly related issues: #3292, #3194, #3369

I've tried with new certificates with 2.2.0, 2.3.0, 2.3.8 but couldn't reproduce the issue.

I think this has something to do with the OpenSSL version too (i.e. 0.9.8 or 1.0.1 or 1.0.2). Homebrew gets me OpenSSL 0.9.8zh 14 Jan 2016 which generates working certificates on Debian Stretch (even if the version is incorrect with 1 instead of 3, and the serial is 1).

Possible Solution

First off, check whether your CA is affected too. Create a new public CA certificate first, then re-generate the client certificates and deploy them at once.

Re-create public CA certificate from existing key pair

This is similar how you re-generate an expired certificate.

1.) Generate a CSR from an existing key pair (public and private key)
2.) Sign the CSR with the CA's private key

You'll need to run both steps using openssl cli commands, as we need to specify the v3 extensions including the CA:TRUE constraints.

        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE

cd /var/lib/icinga2/ca

openssl x509 -x509toreq -in ca.crt -signkey ca.key -out regenerated-ca.csr

cat > icinga2-ext.cnf <<EOF
[v3]
basicConstraints = critical,CA:true
EOF

openssl x509 -in regenerated-ca.csr -out regenerated-ca.crt -signkey ca.key -req -days 5475 -extensions v3 -extfile icinga2-ext.cnf

mv ca.crt ca.crt.borked
mv regenerated-ca.crt ca.crt

Copy the public CA certificate ca.crt to all your nodes in /etc/icinga2/pki/.

Example

openssl x509 -text -in ca.crt -noout
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number: 0 (0x0)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Icinga CA
        Validity
            Not Before: Aug 23 15:44:30 2017 GMT
            Not After : Aug 16 15:44:30 2047 GMT
        Subject: CN=Icinga CA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:c5:d0:30:1c:66:44:ac:7d:17:39:d3:0f:b5:16:
                    d0:3d:4e:e1:07:7c:0d:0f:e1:6f:2f:6e:c4:5d:41:
                    41:6c:a2:ec:0b:e0:a0:3d:3e:b3:02:b3:53:b5:da:
                    ec:df:2d:5b:cd:5c:6a:cb:6c:34:e2:35:39:08:8a:
                    b8:ae:03:c0:da:04:bd:d6:05:e5:04:f3:bf:10:f7:
                    f6:3d:9f:98:9c:6b:16:6b:67:e7:73:72:ee:fd:9c:
                    13:f5:20:88:b2:89:4a:6b:ab:58:a2:7a:55:14:64:
                    12:18:ab:60:4b:0b:2e:43:ac:9b:3a:11:d0:16:79:
                    5f:fd:cf:df:3e:76:38:7d:b5:e1:43:d7:28:58:8a:
                    12:47:3e:35:d5:d7:75:f8:06:9f:cb:a4:1a:59:27:
                    0f:96:e6:b6:1b:83:7c:dc:d9:e4:97:00:ab:38:3c:
                    5f:d2:30:79:2a:42:f8:23:d5:3c:23:06:2f:f1:56:
                    b4:c6:d9:67:0f:b1:3e:76:1e:35:2d:bd:6a:8f:c8:
                    83:af:9b:cb:25:d2:92:21:e6:ba:2b:d0:1f:67:81:
                    f2:6d:f7:9d:80:be:5e:19:e4:16:b3:f4:5a:ef:f4:
                    d1:0b:57:97:ce:0e:20:3a:28:0a:b1:f6:1c:bb:61:
                    b0:47:47:8b:1c:b2:fb:bc:bd:32:a0:0e:94:4a:78:
                    4b:33:e7:f0:a6:ae:95:aa:6e:14:f1:0f:a5:42:4b:
                    56:40:e8:96:ff:eb:dd:e8:61:6d:45:90:5d:13:ee:
                    50:86:8a:40:bc:5e:03:70:21:f0:04:c1:39:a5:24:
                    26:e1:f8:ae:d8:02:f7:e7:87:83:c9:94:1e:e4:d8:
                    87:17:cc:1a:4a:c6:e4:bf:51:40:26:db:15:0b:ee:
                    20:5f:0c:dd:62:df:31:94:84:cb:52:e3:28:b9:57:
                    6f:dc:f3:56:17:a8:6b:f6:a3:da:70:d1:0e:59:1d:
                    5d:f4:13:22:b2:56:13:e1:06:73:3c:0e:96:b2:66:
                    ff:fa:84:de:36:45:a5:26:a4:a7:51:13:1c:b5:4e:
                    b1:84:3b:71:c3:8b:9d:78:cf:a0:de:03:7c:82:b3:
                    71:8d:e7:34:3a:32:13:d8:cb:d7:f9:72:a6:01:f5:
                    42:1b:7a:99:ed:d4:36:48:19:e5:eb:aa:3a:7b:cb:
                    da:17:0d:58:f4:d6:03:c0:82:2e:8b:89:79:d3:b0:
                    56:03:e3:5f:a5:a3:66:03:b6:b1:76:11:ff:c3:cc:
                    20:60:33:c0:09:87:54:13:e8:f2:bd:ca:cb:b7:05:
                    ec:73:2c:0b:3f:07:f8:d2:34:e4:b4:ca:35:31:11:
                    72:4f:08:f0:1d:b6:c1:37:70:dd:34:0f:53:f0:19:
                    b1:2c:c3
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         80:2f:94:f0:2c:ae:9c:c9:41:b2:2d:b0:f3:55:c1:e0:13:17:
         24:ff:d4:13:76:5e:6b:dc:44:00:a1:27:1c:7b:c3:23:61:49:
         8b:42:f5:72:23:67:3d:da:7e:33:fb:c6:f3:23:41:ed:fa:d1:
         74:f7:0b:93:40:6f:f2:8c:86:f7:b8:da:03:c8:1a:4e:91:7c:
         ec:f6:f9:68:d1:8c:32:cc:1a:60:c8:19:2d:9c:c3:81:68:fb:
         ab:89:89:3e:25:f5:5a:cd:d7:82:ff:f7:b1:47:3f:bf:5e:ac:
         be:df:6b:ca:c6:5d:98:d8:18:ac:8b:e1:9d:ac:b1:c8:d8:f8:
         b6:d8:f3:be:69:1c:a2:ff:c6:17:5e:f5:43:3b:ff:3f:35:63:
         c5:14:d6:8c:b2:62:89:67:c7:d6:7d:e5:6f:dd:28:da:9f:31:
         9d:36:15:7f:3b:12:8c:9c:d0:29:8f:90:34:3e:52:06:e1:c8:
         7a:d9:a2:0d:4f:87:ef:c2:11:b6:03:4b:4a:0b:17:31:0b:95:
         34:d7:cf:c3:29:3b:24:b2:f3:7e:1c:e6:9c:3f:ca:be:de:a7:
         ff:31:38:6e:ae:af:54:5a:e4:24:c1:7a:d9:85:d2:a4:13:e5:
         03:4f:d7:97:ff:33:9c:a7:dd:ad:d0:db:45:c4:29:e8:5e:a4:
         6f:b2:58:16:e8:90:9c:aa:4a:9e:77:85:1e:78:b1:c0:19:bd:
         b6:8f:2c:f6:82:c8:6f:28:90:fc:56:90:fb:5a:91:4b:5d:41:
         46:e0:ab:9c:5c:d8:82:f9:e4:3f:f3:4d:f5:6e:92:d7:ab:c7:
         c5:ae:07:7b:5b:0d:58:b8:6c:d0:ee:df:48:bd:17:ec:d3:eb:
         e0:65:bb:da:32:8a:50:da:ad:5f:c9:2c:ae:fb:42:e1:d0:10:
         84:9a:c5:b0:69:bc:66:d1:06:70:5f:c6:17:d5:18:e8:07:89:
         bb:75:00:33:dd:dd:c2:98:aa:39:fe:57:ac:da:d9:3c:96:8a:
         02:2f:fc:1c:a5:48:e2:36:22:5d:df:35:87:21:1a:42:eb:8a:
         b5:8e:24:3b:5e:ff:25:d0:17:e4:20:c8:2b:09:f7:06:40:b5:
         f0:4a:b3:d4:f5:5f:c7:1c:be:4d:1d:5a:06:d8:1a:4c:f8:0b:
         c7:48:9f:6b:07:dc:b4:c3:da:aa:15:78:e7:fa:4c:27:8f:50:
         56:f0:00:ae:8e:d8:e4:8f:7b:47:2a:7b:01:a2:b7:42:38:6a:
         59:85:32:c9:11:3a:4d:c9:56:7c:3c:ca:9f:b4:1e:0e:f1:6e:
         79:72:2f:86:29:77:d2:78:60:b8:07:60:32:c3:6e:6b:51:ae:
         5f:be:b2:b5:da:b1:1e:aa

Re-create client certificates

This works in a similar fashion like the CA key pair. Create a CSR from the existing client key pair and sign it again with the private CA key on your master.

Note: This requires you to copy the client's public and private key to your Icinga 2 master where the private CA key is located.

On the client:

openssl x509 -x509toreq -in client.crt -signkey client.key -out client.csr

openssl req -text -in client.csr

On the master:

icinga2 pki sign-csr --csr client.csr --cert client.crt

Then roll out the updated certificates to your clients. A different method is to re-run the CSR-Autosigning wizard (move away etc/icinga2/pki and re-run the wizard).

Example

openssl x509 -text -in client.crt -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            de:cc:4a:1b:04:67:79:e3:64:ee:12:f8:26:aa:de:ab:ce:54:3f:31
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Icinga CA
        Validity
            Not Before: Aug 24 09:14:41 2017 GMT
            Not After : Aug 20 09:14:41 2032 GMT
        Subject: CN=client
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:bb:d8:e6:48:58:1f:69:fa:7f:ce:00:46:d9:8e:
                    5d:22:2e:f5:ce:47:15:f0:93:a2:43:b1:a2:33:3a:
                    25:af:e4:d5:05:e7:f5:f7:23:50:09:b3:a7:26:2c:
                    54:c9:44:50:c3:9e:24:21:71:5a:18:c0:96:86:ab:
                    3c:21:3a:f7:87:9b:bf:8a:e6:99:e4:dc:fa:c1:4b:
                    40:43:9d:b0:a6:d6:17:2c:75:3e:93:77:3f:bc:e1:
                    9d:7f:f0:87:b1:d6:7a:3d:1c:9c:5d:8c:1a:b3:b9:
                    0d:c1:2f:a0:9e:83:a5:e1:bf:d6:55:01:dd:2f:c5:
                    3f:44:ab:b1:0d:1a:15:f3:6d:81:09:3e:08:e2:ad:
                    a1:2f:6f:18:4a:6d:48:ad:f5:d8:16:00:b8:f6:11:
                    bd:cf:fc:b8:e6:e8:47:a1:c5:f5:1f:ec:a9:34:96:
                    45:2d:dd:0a:aa:7d:dc:aa:23:04:d1:9d:58:2f:ad:
                    61:55:07:48:bf:76:52:3f:fd:85:42:e4:0a:e5:64:
                    5f:1e:c2:82:8c:fb:5a:91:20:f3:52:86:5a:fa:58:
                    6a:9c:df:a2:18:70:37:cb:17:b5:8c:6c:c0:28:54:
                    53:21:07:36:0f:a7:16:86:45:8d:0b:43:92:b0:2f:
                    47:d5:96:9c:5b:76:d0:f4:46:71:90:fc:84:aa:97:
                    ec:f5:a2:9a:ba:b4:ca:e6:69:8e:84:63:f6:ef:72:
                    5d:4a:4e:d6:7c:8e:c9:7f:1a:ca:61:7a:c5:33:90:
                    19:54:67:d9:0d:e7:67:0e:90:d8:4c:74:62:7c:fc:
                    1b:ab:59:b0:9d:d4:b6:c5:46:a1:5b:bc:54:5a:8d:
                    de:35:19:ae:8c:a6:d6:f6:9f:c4:cb:67:43:67:58:
                    b3:2d:54:4f:45:61:50:3c:1e:12:56:cc:4a:c5:46:
                    46:6c:f9:3d:a8:8a:b1:37:b4:cb:59:bf:bd:76:13:
                    1f:b1:64:8c:6d:2c:e3:28:2f:ca:34:8f:57:8f:85:
                    52:b4:1e:ca:56:66:b6:ee:ad:2d:80:11:87:63:6b:
                    d5:7b:a8:f0:7a:3d:81:2b:64:73:33:f3:7e:5d:0f:
                    4c:2d:ca:ff:d5:f2:d0:56:e8:0f:d5:5a:d7:ac:b9:
                    db:df:fb:9f:b6:af:a2:90:83:2f:a8:9c:6b:1d:00:
                    0d:b5:d0:7f:d8:8e:35:11:9f:c8:b0:2b:92:3d:ef:
                    62:fb:81:ae:09:6d:13:ae:a7:23:d4:a5:7b:a0:bd:
                    01:32:a1:3d:c3:07:60:4e:4f:e5:b6:62:21:8d:ea:
                    42:53:1b:47:fc:fa:32:a3:d1:02:66:17:13:31:01:
                    37:fb:36:1d:6a:63:d5:80:74:d1:fd:a5:58:f8:41:
                    aa:e0:9b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:FALSE
    Signature Algorithm: sha256WithRSAEncryption
         93:48:aa:a3:ee:f3:c7:12:e6:06:be:aa:e3:b9:36:63:3c:98:
         3a:e0:09:48:f5:d8:a6:25:ed:ae:f6:55:0f:ec:15:5d:77:ce:
         d0:c7:88:fa:ca:23:56:8a:de:95:d9:c1:ec:f7:85:63:f8:f5:
         b5:6a:d1:48:f1:eb:12:df:15:19:7e:70:5b:a0:b5:a5:82:f5:
         ff:55:fe:0e:4c:6e:55:73:e1:42:0e:0e:78:25:83:8c:eb:76:
         d3:40:e7:9c:20:03:ab:e8:73:af:cc:8d:97:f1:0c:97:9a:ef:
         bc:36:4c:2c:72:e8:05:80:cb:0d:b3:a0:1a:a1:6e:4f:9b:01:
         b7:54:be:7b:a4:ef:ab:d0:35:60:c5:2a:0e:25:dc:1a:7f:42:
         25:f1:b7:29:7a:86:4d:99:2b:e4:29:e3:7f:c7:a7:4a:4b:a5:
         78:01:95:fb:63:e2:16:eb:90:5a:be:f9:52:ef:36:51:b5:2a:
         37:2e:0d:2e:0b:42:0d:15:1a:98:43:5a:2c:52:0a:21:b2:49:
         3f:3b:01:fb:00:d1:48:5f:02:23:e6:74:28:98:c3:2b:61:cf:
         36:42:b0:fd:82:54:9a:4e:6a:61:6e:30:91:98:b6:f1:9f:8e:
         38:ca:79:42:79:8f:fe:f5:1b:24:93:17:8b:df:e1:6f:91:24:
         92:5f:cd:08:ab:50:86:a6:76:9a:0c:a0:3d:6c:48:65:ea:7b:
         8c:07:8e:3c:ed:f0:bc:57:d9:3e:a7:06:cf:d2:7c:a2:ad:52:
         86:29:1c:25:c9:f3:30:cb:55:ba:19:06:e6:0c:53:83:1d:e5:
         09:dc:75:90:49:00:38:75:29:99:95:b0:51:20:6c:22:1c:91:
         48:f9:19:79:60:04:fc:68:c7:44:99:ee:90:1a:a9:d4:ee:8b:
         d4:10:68:f1:3a:b7:ca:a9:1e:ee:49:b9:88:ff:39:22:1c:ec:
         a7:fb:17:c4:40:2f:0a:6a:86:60:de:c9:dc:b0:28:0d:c6:20:
         a3:d7:83:c8:0c:42:4e:c1:58:c6:12:3e:7b:68:80:bc:db:db:
         9d:74:a4:d8:27:29:c8:a5:b6:98:24:bf:82:51:4e:50:17:c7:
         89:69:ee:7c:fe:3d:bf:7e:8d:8c:f5:71:39:f3:b7:3e:49:d7:
         b7:0a:6c:25:17:96:81:fd:f6:29:29:e7:a7:78:d6:9e:59:63:
         b5:ec:45:41:df:43:8c:13:0c:b6:ab:fa:cc:5c:86:45:fa:21:
         6c:b6:8d:a1:14:b1:5f:c9:76:93:84:9e:65:f3:10:f5:41:90:
         3f:b3:3f:d1:33:76:16:2b:49:6c:1c:94:30:ee:f9:14:c9:40:
         f1:06:e6:a7:ec:dd:c6:6a

[dnsmichi]

@widhalmt it seems this can only be reproduced reliably with Debian from 2015 (Jessie I guess), OpenSSL 1.0.1f and Icinga 2 v2.3.0 on the master for generating the CA and client certificates. Please take that into account for your research.

https://monitoring-portal.org/index.php?thread/41664-ca-crt-verification-error-with-openssl-1-1-0-illegal-zero-content-in-field-seria/&postID=254852#post254852

[dnsmichi]

Updated the CA regeneration, this contained faulty signing. We need to ensure that the v3 extension sets the critical constraint to CA:TRUE when signing the CA certificate.

See https://github.com/Icinga/icinga2/issues/5511#issuecomment-324384804

[dnsmichi]

I couldn't reproduce the issue in a Debian Jessie container with 2.3.8, 2.3.0 and OpenSSL 1.0.1f (copying the certificates to a Debian Stretch container and validating them - works like normal).

[dnsmichi]

Note: Existing Icinga 2 setups with 2.4+ will generate proper certificates valid with 1.1.0. This issue is solely for keeping track of affected users and possible workarounds.

[dnsmichi]

Re-iterating the error message from what we do during ApiListener::OnConfigLoaded inside MakeSSLContext(), this only affects the ca_path and in turn the ca.crt file distributed to all clients. IT does not seem to be necessary to look into new client certificates at all.

Please add your feedback on this.

[edpstiffel]

FYI
Ran into this problem while deploying icinga2-client to the first Debian Stretch machine.
Our ca.crt was from 2014/12/04.
Followed DNSMichis instructions to recreate the ca.crt based on the same key. Worked like a charm. With the new ca.crt, the icinga-client on the Stretch-machine came up.
Other clients didn't have a problem with the new cert, I didn't have to replace the ca.crt on these machines neither did I have to create new client certs.
One addition: After recreating the ca.crt, you HAVE TO copy the new ca.crt on your icinga-server from /var/lib/icinga2/ca to /etc/icinga2/pki, otherwise you get Client TLS handshake failed errors with your new client.
Thanks for clarifying the issue and the clear instructions.

[dnsmichi]

Updated my comment. The up-to-date analysis and fix procedure is documented here: https://www.icinga.com/2017/08/30/advisory-for-ssl-problems-with-leading-zeros-on-openssl-1-1-0/

[widhalmt]

If this is only reproducible with Debian it can't be the problem I hit. This was a Windows node connected to SLES.

[dnsmichi]

Whereas the Windows package uses OpenSSL 1.1.0 which breaks on the ca.crt file during startup.

[dnsmichi]

Please follow the linked advisory and ensure to update the ca.crt on all clients (all of them, sooner or later they will receive a dist-upgrade and OpenSSL 1.1.0).

This one is informational only, therefore assigning for 2.8 to reflect it in the Changelog.