Nixpkgs: cacert blacklisting & UTF-8 encoding

[evrim]

Issue description

$ cat /etc/nixos/configuration.nix

security.pki.caCertificateBlacklist = [
"Redacted CACert with UTF-8 Chars in its name"
];

$ nixos-rebuild switch
building
certdata2pem.py:103: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
if obj['CKA_LABEL'] in blacklist:

So basically, if the provider has UTF-8 chars, you cannot blacklist it.

Steps to reproduce

$ cd && mkdir cert && cd cert
$ wget "https://hg.mozilla.org/releases/mozilla-release/raw-file/default/security/nss/lib/ckfw/builtins/certdata.txt"
$ wget "https://anonscm.debian.org/cgit/collab-maint/ca-certificates.git/plain/mozilla/certdata2pem.py?h=debian/20160104"
$ echo "Redacted CACert with UTF-8 Chars in its name" > ./blacklist.txt
$ nix-shell -p python
$ python -V
Python 2.7.13
$ python ./certdata2pem.py |grep -vE '(!|UNTRUSTED)'
./certdata2pem.py:103: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
if obj['CKA_LABEL'] in blacklist:
Ignoring certificate "UTN USERFirst Object Root CA". SAUTH=CKT_NSS_MUST_VERIFY_TRUST, EPROT=CKT_NSS_MUST_VERIFY_TRUST
Ignoring certificate "ComSign Secured CA". SAUTH=CKT_NSS_MUST_VERIFY_TRUST, EPROT=CKT_NSS_MUST_VERIFY_TRUST
Ignoring certificate "VeriSign-C3SSA-G2-temporary-intermediate-after-1024bit-removal". SAUTH=CKT_NSS_MUST_VERIFY_TRUST, EPROT=CKT_NSS_MUST_VERIFY_TRUST
Found duplicate certificate name StartCom_Certification_Authority, renaming.
$ ^D
$ nix-shell -p python3
$ python -V
Python 3.5.3
$ python ./certdata2pem.py |grep -vE '(!|UNTRUSTED)'
Ignoring certificate "UTN USERFirst Object Root CA". SAUTH=CKT_NSS_MUST_VERIFY_TRUST, EPROT=CKT_NSS_MUST_VERIFY_TRUST
Ignoring certificate "ComSign Secured CA". SAUTH=CKT_NSS_MUST_VERIFY_TRUST, EPROT=CKT_NSS_MUST_VERIFY_TRUST
Ignoring certificate "VeriSign-C3SSA-G2-temporary-intermediate-after-1024bit-removal". SAUTH=CKT_NSS_MUST_VERIFY_TRUST, EPROT=CKT_NSS_MUST_VERIFY_TRUST
Certificate "Redacted CACert with UTF-8 Chars in its name" blacklisted, ignoring.
Found duplicate certificate name b'StartCom_Certification_Authority', renaming.

Solution

Alter /nixpkgs.git/pkgs/data/misc/cacert/default.nix as:

-- nativeBuildInputs = [ python ];
++ nativeBuildInputs = [ python3 ];

Technical details

• System:
  $ nixos-version
  17.03.1506.b6f05ca54d (Gorilla)

PS. May I have it also backported to 17.03?

[evrim]

ping @wkennington @fpletz

[fpletz]

Backported to 17.03: 3c6e969bde049d1b9897b65a474cbf7d959b2bbc

Thanks for the report and providing the solution!

[evrim]

I am afraid the patch didn't cut it :*

# nixos-version
17.03.1623.138313fe01 (Gorilla)
# nixos-rebuild test
building Nix...
building the system configuration...
these derivations will be built:
  /nix/store/7vhrnv1pcfxvcgizzq71nnjywvsfg8mz-nss-cacert-3.31.drv
  /nix/store/k8zah7rqdhqwgngqfqbskp71nj1612wm-ca-certificates.crt.drv
  /nix/store/6wnx6hr8lvbhsxr3iwk83fdcnr7h5wzm-etc.drv
  /nix/store/pkiyvngldjwycdc0afr89b70hnfmz2hd-nixos-system-node1-17.03.1638.2d80003beb.drv
building path(s) ‘/nix/store/6n1m30h0xnhk3mxsalfpd55pfrzcjkdi-nss-cacert-3.31’
unpacking sources
unpacking source archive /nix/store/d7h29gyz2dlvvfjnsrg7gcd64fykq41k-nss-3.31.tar.gz
source root is nss-3.31
setting SOURCE_DATE_EPOCH to timestamp 1496919181 of file nss-3.31/nss/trademarks.txt
patching sources
configuring
building
Traceback (most recent call last):
  File "certdata2pem.py", line 91, in 
    for line in open('blacklist.txt', 'r'):
  File "/nix/store/58kwrs1w21lxxk1d2v25k7wprmlrr6jj-python3-3.5.3/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 26: ordinal not in range(128)
builder for ‘/nix/store/7vhrnv1pcfxvcgizzq71nnjywvsfg8mz-nss-cacert-3.31.drv’ failed with exit code 1
cannot build derivation ‘/nix/store/k8zah7rqdhqwgngqfqbskp71nj1612wm-ca-certificates.crt.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/6wnx6hr8lvbhsxr3iwk83fdcnr7h5wzm-etc.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/pkiyvngldjwycdc0afr89b70hnfmz2hd-nixos-system-node1-17.03.1638.2d80003beb.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/pkiyvngldjwycdc0afr89b70hnfmz2hd-nixos-system-node1-17.03.1638.2d80003beb.drv’ failed

Same in the unstable.

# nixos-version 
17.09pre111970.c6be8c75fe (Hummingbird)
# nixos-rebuild test
building Nix...
building the system configuration...
these derivations will be built:
  /nix/store/zimpiznkj2npmz6sj8lqgfp4g3fq3lwp-nss-cacert-3.31.drv
  /nix/store/mvqwgyqx0c42288q0h12yv4w9rdpd51a-ca-certificates.crt.drv
  /nix/store/6jmm24896k0qjqgi4dgx05sagzm3f664-etc.drv
  /nix/store/c3sr5l7207s53k9p8342s6229y5v4wml-nixos-system-node1-17.09pre111970.c6be8c75fe.drv
these paths will be fetched (9.08 MiB download, 52.90 MiB unpacked):
  /nix/store/pdx46najk62akv5bw7yidnwxmw586aql-python3-3.6.2
fetching path ‘/nix/store/pdx46najk62akv5bw7yidnwxmw586aql-python3-3.6.2’...

*** Downloading ‘https://cache.nixos.org/nar/0a7xb2inizv1vi7l15hjhh2cyavzp795g5i8a8fa8ri8f3z6w94i.nar.xz’ (signed by ‘cache.nixos.org-1’) to ‘/nix/store/pdx46najk62akv5bw7yidnwxmw586aql-python3-3.6.2’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 9298k  100 9298k    0     0  5433k      0  0:00:01  0:00:01 --:--:-- 5437k

building path(s) ‘/nix/store/skr6nccz5lp9l4n8mljrkzy4vn977mvb-nss-cacert-3.31’
unpacking sources
unpacking source archive /nix/store/d7h29gyz2dlvvfjnsrg7gcd64fykq41k-nss-3.31.tar.gz
source root is nss-3.31
setting SOURCE_DATE_EPOCH to timestamp 1496919181 of file nss-3.31/nss/trademarks.txt
patching sources
configuring
building
Traceback (most recent call last):
  File "certdata2pem.py", line 91, in 
    for line in open('blacklist.txt', 'r'):
  File "/nix/store/pdx46najk62akv5bw7yidnwxmw586aql-python3-3.6.2/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 26: ordinal not in range(128)
builder for ‘/nix/store/zimpiznkj2npmz6sj8lqgfp4g3fq3lwp-nss-cacert-3.31.drv’ failed with exit code 1
cannot build derivation ‘/nix/store/mvqwgyqx0c42288q0h12yv4w9rdpd51a-ca-certificates.crt.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/6jmm24896k0qjqgi4dgx05sagzm3f664-etc.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/c3sr5l7207s53k9p8342s6229y5v4wml-nixos-system-node1-17.09pre111970.c6be8c75fe.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/c3sr5l7207s53k9p8342s6229y5v4wml-nixos-system-node1-17.09pre111970.c6be8c75fe.drv’ failed

[evrim]

I have the following, hope it helps. This works with nativeBuildInputs = [python];

# diff -Naur ./certdata2pem.py\?h\=debian%2F20160104 ./certdata2pem.py 
--- ./certdata2pem.py?h=debian%2F20160104   2017-08-01 23:10:00.000000000 +0300
+++ ./certdata2pem.py   2017-08-01 23:08:21.131297636 +0300
@@ -88,7 +88,7 @@
 \# Read blacklist.
 blacklist = []
 if os.path.exists('blacklist.txt'):
-    for line in open('blacklist.txt', 'r'):
+    for line in io.open('blacklist.txt', 'r', encoding='utf-8'):
         line = line.strip()
         if line.startswith('#') or len(line) == 0:
             continue
@@ -101,7 +101,7 @@
     if obj['CKA_CLASS'] != 'CKO_NSS_TRUST':
         continue
     if obj['CKA_LABEL'] in blacklist:
-        print("Certificate %s blacklisted, ignoring." % obj['CKA_LABEL'])
+        print("Certificate %s blacklisted, ignoring." % unicode(obj['CKA_LABEL']).encode('utf-8'))
     elif obj['CKA_TRUST_SERVER_AUTH'] == 'CKT_NSS_TRUSTED_DELEGATOR':
         trust[obj['CKA_LABEL']] = True
     elif obj['CKA_TRUST_EMAIL_PROTECTION'] == 'CKT_NSS_TRUSTED_DELEGATOR':

[evrim]

ping @fpletz

[fpletz]

Hmm, weird, I tested it with a CA that had special characters its name. Well, probably wasn't UTF-8 encoded after all. :)

I will test and apply your patch. Thanks!

[fpletz]

Works fine! Pushed and backported to 17.03. Thanks! :beers: