Nix: Nix 1.11.2 - nix-channel fails to update due to nix-prefetch-url 'Problem with the SSL CA cert'

[cmpitg]

Issue description

Nix 1.11.2 - nix-channel --update failed as nix-prefetch-url reported 'Problem with the SSL CA cert'.

Steps to reproduce

• Upgrading Nix 1.10 to 1.11.2 from source (./configure && make && sudo make install).

• Running nix-channel --update failed with the following message:

  ✗ nix-channel --update
  downloading Nix expressions from ‘https://nixos.org/releases/nixpkgs/nixpkgs-16.09pre80183.551296a//nixexprs.tar.xz’...
  downloading ‘https://nixos.org/releases/nixpkgs/nixpkgs-16.09pre80183.551296a//nixexprs.tar.xz’... [0/0 KiB, 0.0 KiB/s]
  error: unable to download ‘https://nixos.org/releases/nixpkgs/nixpkgs-16.09pre80183.551296a//nixexprs.tar.xz’: Problem with the SSL CA cert (path? access rights?) (77)
  cannot fetch ‘https://nixos.org/releases/nixpkgs/nixpkgs-16.09pre80183.551296a//nixexprs.tar.xz’
  

• Looking at nix-channel script, I found the failing line:

          (my $hash, $path) = `PRINT_PATH=1 QUIET=1 $Nix::Config::binDir/nix-prefetch-url '$fullURL'`;
  

  ✗ nix-prefetch-url https://nixos.org/releases/nixpkgs/nixpkgs-16.09pre80183.551296a//nixexprs.tar.xz
  downloading ‘https://nixos.org/releases/nixpkgs/nixpkgs-16.09pre80183.551296a//nixexprs.tar.xz’... [0/0 KiB, 0.0 KiB/s]
  error: unable to download ‘https://nixos.org/releases/nixpkgs/nixpkgs-16.09pre80183.551296a//nixexprs.tar.xz’: Problem with the SSL CA cert (path? access rights?) (77)
  

  Yup, that's the one.

• At first, I thought somehow nix-prefetch-url couldn't find the CA Certificates bundle. I tried setting the paths manually with environment variables. Didn't work:

  export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
  export CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
  # Just to be sure
  export GIT_SSL_CAINFO=/etc/ssl/certs/ca-certificates.crt
  

• According to some previous issues, installing cacert might solve the problem. As cacert package doesn't exist anymore, I installed nss-cacert, thus having the ${HOME}/.nix-profile/etc/ssl/certs/ca-bundle.crt. Pointing the above 3 environment variables to ${HOME}/.nix-profile/etc/ssl/certs/ca-bundle.crt didn't work either.

  Technical details

• System: Debian 9 (Stretch/testing)

• Nix version: nix-env (Nix) 1.11.2
• Nixpkgs version: error: file ‘nixpkgs’ was not found in the Nix search path (add it using $NIX_PATH or -I)
• Channel (from ${HOME}/.nix-channels): https://nixos.org/channels/nixpkgs-unstable nixpkgs

Ha-Duong

[dtzWill]

I ran into similar issue and it turned out I built a newer version of curl for nix but forgot to add it to my LD_LIBRARY_PATH.

If you haven't, try upgrading curl and ensure you're using the corresponding libcurl from that version by inspecting the output of lddwhich nix-prefetch-url``.

Hope this helps!

[gwillen]

I just ran into this trying to install systemwide Nix for the first time, on a fresh Debian 8.6 install, following the instructions at https://nixos.org/wiki/Installing_Nix_on_Debian .

The rabbit-hole seems pretty deep and I've been crawling down it for about an hour now; but from what I'm seeing on my system, I think this is a consequence of replacing nix-prefetch-url, previously a shellscript in 1.10, with a binary in 1.11, which links against libcurl-nss. On my Debian system, the ca-certificates package provides a certificate bundle in pem format, but apparently libcurl-nss cannot read pem format without an auxiliary library called libnsspem, which everyone seems to point fingers at each other about, but nobody seems to provide on Debian. (I can observe nix-prefetch-url trying to load libnsspem by using strace, which is what led me to find the bug I link below.)

There's a Debian bug filed about this, dating back to 2013; it's hard for me to figure out why my brand new Debian system doesn't have this library, which was allegedly (circa 2014) going to be provided as part of the nss package "in the future".

See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=726073 .

If I'm right it seems like more people should be seeing this brokenness, though... but I guess systemwide Nix installs aren't that common on non-NixOS systems, and perhaps this doesn't show up in a single-user install for some other reason? Unclear. Or perhaps I'm wrong and the cause is something else.

[domenkozar]

This might get fixed in Nix 1.11.7, to be released very soon.

[ulysses4ever]

@domenkozar wrote:

This might get fixed in Nix 1.11.7, to be released very soon.

Just installed 1.11.7 to vanilla Ubuntu: got the same error :(

Are there any workaround accessible for a novice?

[domenkozar]

Can you paste your error?

[ulysses4ever]

@domenkozar

$ nix-channel --update
downloading Nix expressions from ‘http://nixos.org/releases/nixpkgs/channels/nixpkgs-unstable/nixexprs.tar.bz2’...
downloading ‘http://nixos.org/releases/nixpkgs/channels/nixpkgs-unstable/nixexprs.tar.bz2’... [0/0 KiB, 0.0 KiB/s]
error: unable to download ‘http://nixos.org/releases/nixpkgs/channels/nixpkgs-unstable/nixexprs.tar.bz2’: HTTP response code said error (22)
cannot fetch ‘http://nixos.org/releases/nixpkgs/channels/nixpkgs-unstable/nixexprs.tar.bz2’

[domenkozar]

Can you try:

SSL_CERT_FILE=$NIX_SSL_CERT_FILE nix-channel --update

[ulysses4ever]

Unfortunately, it gives the same error.

$ SSL_CERT_FILE=$NIX_SSL_CERT_FILE nix-channel --update
downloading Nix expressions from ‘http://nixos.org/releases/nixpkgs/channels/nixpkgs-unstable/nixexprs.tar.bz2’...
downloading ‘http://nixos.org/releases/nixpkgs/channels/nixpkgs-unstable/nixexprs.tar.bz2’... [0/0 KiB, 0.0 KiB/s]
error: unable to download ‘http://nixos.org/releases/nixpkgs/channels/nixpkgs-unstable/nixexprs.tar.bz2’: HTTP response code said error (22)
cannot fetch ‘http://nixos.org/releases/nixpkgs/channels/nixpkgs-unstable/nixexprs.tar.bz2’
``````
Though

ubuntu@ubuntu-VirtualBox:~$ echo $NIX_SSL_CERT_FILE
/etc/ssl/certs/ca-certificates.crt
```

[domenkozar]

Does /etc/ssl/certs/ca-certificates.crt exist?

[ulysses4ever]

Seems so:

$ ll `echo $NIX_SSL_CERT_FILE `
-rw-r--r-- 1 root root 273790 Jun 14  2015 /etc/ssl/certs/ca-certificates.crt

I also tried to execute it under root -- did not help.

[domenkozar]

Can you paste:

ldd $(which curl)

[ulysses4ever]

$ ldd $(which curl)
    linux-vdso.so.1 =>  (0x00007ffe399ce000)
    libcurl.so.4 => /usr/lib/x86_64-linux-gnu/libcurl.so.4 (0x00007f7dc9a4f000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f7dc9834000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7dc9616000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7dc924c000)
    libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007f7dc9019000)
    librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f7dc8dfd000)
    libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f7dc8b9c000)
    libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f7dc87b8000)
    libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f7dc8570000)
    liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f7dc8361000)
    libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f7dc810f000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f7dc9ee8000)
    libgnutls-deb0.so.28 => /usr/lib/x86_64-linux-gnu/libgnutls-deb0.so.28 (0x00007f7dc7df4000)
    libhogweed.so.2 => /usr/lib/x86_64-linux-gnu/libhogweed.so.2 (0x00007f7dc7bc6000)
    libnettle.so.4 => /usr/lib/x86_64-linux-gnu/libnettle.so.4 (0x00007f7dc7995000)
    libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f7dc7715000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7dc7511000)
    libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f7dc7242000)
    libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f7dc7011000)
    libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f7dc6e0d000)
    libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f7dc6c02000)
    libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f7dc69e6000)
    libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f7dc67cb000)
    libgssapi.so.3 => /usr/lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007f7dc658b000)
    libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f7dc62a9000)
    libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f7dc6063000)
    libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f7dc5e4f000)
    libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f7dc5c4b000)
    libheimntlm.so.0 => /usr/lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007f7dc5a42000)
    libkrb5.so.26 => /usr/lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007f7dc57b8000)
    libasn1.so.8 => /usr/lib/x86_64-linux-gnu/libasn1.so.8 (0x00007f7dc5513000)
    libhcrypto.so.4 => /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007f7dc52df000)
    libroken.so.18 => /usr/lib/x86_64-linux-gnu/libroken.so.18 (0x00007f7dc50ca000)
    libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f7dc4eb8000)
    libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f7dc4cb0000)
    libwind.so.0 => /usr/lib/x86_64-linux-gnu/libwind.so.0 (0x00007f7dc4a87000)
    libheimbase.so.1 => /usr/lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007f7dc4878000)
    libhx509.so.5 => /usr/lib/x86_64-linux-gnu/libhx509.so.5 (0x00007f7dc462d000)
    libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f7dc4366000)
    libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f7dc412e000)

[domenkozar]

Ah @gwillen already tracked this one. I'm going to look if we can fix it.

[domenkozar]

@ulysses4ever how did you install Nix?

And also: $ ldd $(which nix-prefetch-url)

[ulysses4ever]

The way pointed out in the official site: curl https://nixos.org/nix/install | sh

$ ldd $(which nix-prefetch-url)
    linux-vdso.so.1 =>  (0x00007ffeba572000)
    libnixmain.so => /nix/store/4ssykr786d0wp7y6m4xd4qwqs4nrry1z-nix-1.11.7/lib/libnixmain.so (0x00007efca0bb5000)
    libnixexpr.so => /nix/store/4ssykr786d0wp7y6m4xd4qwqs4nrry1z-nix-1.11.7/lib/libnixexpr.so (0x00007efca092c000)
    libgc.so.1 => /nix/store/3iclzxr6d7ykr6ph5kgxsjk3a7hjrl8b-boehm-gc-7.2g/lib/libgc.so.1 (0x00007efca06c2000)
    libnixstore.so => /nix/store/4ssykr786d0wp7y6m4xd4qwqs4nrry1z-nix-1.11.7/lib/libnixstore.so (0x00007efca0407000)
    libnixutil.so => /nix/store/4ssykr786d0wp7y6m4xd4qwqs4nrry1z-nix-1.11.7/lib/libnixutil.so (0x00007efca01d6000)
    libnixformat.so => /nix/store/4ssykr786d0wp7y6m4xd4qwqs4nrry1z-nix-1.11.7/lib/libnixformat.so (0x00007efc9ffcc000)
    libstdc++.so.6 => /nix/store/gs36amglhgii6i9nb7wrlps2lnvcq4f6-gcc-5.4.0-lib/lib/libstdc++.so.6 (0x00007efc9fc54000)
    libm.so.6 => /nix/store/q3wx1gab2ysnk5nyvyyg56ana2v4r2ar-glibc-2.24/lib/libm.so.6 (0x00007efc9f94f000)
    libgcc_s.so.1 => /nix/store/q3wx1gab2ysnk5nyvyyg56ana2v4r2ar-glibc-2.24/lib/libgcc_s.so.1 (0x00007efc9f739000)
    libc.so.6 => /nix/store/q3wx1gab2ysnk5nyvyyg56ana2v4r2ar-glibc-2.24/lib/libc.so.6 (0x00007efc9f39b000)
    libdl.so.2 => /nix/store/q3wx1gab2ysnk5nyvyyg56ana2v4r2ar-glibc-2.24/lib/libdl.so.2 (0x00007efc9f197000)
    libpthread.so.0 => /nix/store/q3wx1gab2ysnk5nyvyyg56ana2v4r2ar-glibc-2.24/lib/libpthread.so.0 (0x00007efc9ef7a000)
    /nix/store/q3wx1gab2ysnk5nyvyyg56ana2v4r2ar-glibc-2.24/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007efca0dc6000)
    libsqlite3.so.0 => /nix/store/y8fwydq5y1y866cz6p8j1044ys0rpxxi-sqlite-3.14.2/lib/libsqlite3.so.0 (0x00007efc9eca1000)
    libbz2.so.1 => /nix/store/dris5iaziqna4l3kafiyhr0wyx34gjmz-bzip2-1.0.6.0.1/lib/libbz2.so.1 (0x00007efc9ea91000)
    libcurl.so.4 => /nix/store/fa9yzmm44n6py8zn42ib3fv9jwgfqsgc-curl-7.53.0/lib/libcurl.so.4 (0x00007efc9e81a000)
    liblzma.so.5 => /nix/store/8rlyzn7cyq23a0gbg824nhddgb6m4207-xz-5.2.2/lib/liblzma.so.5 (0x00007efc9e5f4000)
    libcrypto.so.1.0.0 => /nix/store/rxahqlnfsl01gzx4k3b67l78hi4n6wdi-openssl-1.0.2k/lib/libcrypto.so.1.0.0 (0x00007efc9e1b8000)
    libnghttp2.so.14 => /nix/store/dim0r1da3rw62wnrcsw8w0cdnrwqlwxk-nghttp2-1.14.1-lib/lib/libnghttp2.so.14 (0x00007efc9df93000)
    libssh2.so.1 => /nix/store/l0908346dhsqcjsybazgnvvqaj8vq1df-libssh2-1.7.0/lib/libssh2.so.1 (0x00007efc9dd65000)
    libssl.so.1.0.0 => /nix/store/rxahqlnfsl01gzx4k3b67l78hi4n6wdi-openssl-1.0.2k/lib/libssl.so.1.0.0 (0x00007efc9daf7000)
    libz.so.1 => /nix/store/6rhawr786hmpg0lw910rna03v0gbw93j-zlib-1.2.8/lib/libz.so.1 (0x00007efc9d8e1000)

[domenkozar]

Wait, that link doesn't exist. Can you do nix-channel --list

[ulysses4ever]

$ nix-channel --list
nixpkgs http://nixos.org/releases/nixpkgs/channels/nixpkgs-unstable

[domenkozar]

How did you end up with that channel? That link doesn't exist.

nix-channel --add https://nixos.org/channels/nixpkgs-unstable

[ulysses4ever]

I'm very sorry for this mess. I just followed an installation guide for an outdated project which happened to use Nix.

[pmarreck]

I'm having this same exact issue while trying to set up Nix on the latest version of OS X and getting to the unstable branch in order to install Elixir 1.5 on OTP20

[masaeedu]

strace-ing on Cygwin, I identified what might be the same issue. When nix-channel --update calls nix-prefetch-url, this happens:

   67  472304 [main] nix-prefetch-url 5100 fhandler_base::open: (\??\D:\tools\chocolatey\cygwin\usr\ssl\certs\ca-certificates.crt, 0x108000)
   95  472399 [main] nix-prefetch-url 5100 seterrno_from_nt_status: /home/corinna/src/cygwin/cygwin-2.8.1/cygwin-2.8.1-1.x86_64/src/newlib-cygwin/winsup/cygwin/fhandler.cc:671 status 0xC0000034 -> windows err
   61  472460 [main] nix-prefetch-url 5100 geterrno_from_win_error: windows error 2 == errno 2
   69  472529 [main] nix-prefetch-url 5100 fhandler_base::open: 0xC0000034 = NtCreateFile (0x1802D0520, 0x80100000, \??\D:\tools\chocolatey\cygwin\usr\ssl\certs\ca-certificates.crt, io, NULL, 0x0, 0x7, 0x1, 0
   30  472559 [main] nix-prefetch-url 5100 fhandler_base::open: 0 = fhandler_base::open(\??\D:\tools\chocolatey\cygwin\usr\ssl\certs\ca-certificates.crt, 0x108000)
   47  472606 [main] nix-prefetch-url 5100 fhandler_base::open_fs: 0 = fhandler_disk_file::open(\??\D:\tools\chocolatey\cygwin\usr\ssl\certs\ca-certificates.crt, 0x8000)
   85  472691 [main] nix-prefetch-url 5100 open: -1 = open(/etc/ssl/certs/ca-certificates.crt, 0x8000), errno 2

Note that D:\tools\chocolatey\cygwin\usr\ssl\certs\ca-certificates.crt doesn't actually exist: the only .crt files in there are ca-bundle.crt and ca-bundle.trust.crt.

Not 100% sure if this is the same problem. Output is basically the same:

$ nix-channel --update
downloading Nix expressions from ‘https://d3g5gsiof5omrk.cloudfront.net/nixpkgs/nixpkgs-18.03pre119245.5cfd049a03/nixexprs.tar.xz’...
downloading ‘https://d3g5gsiof5omrk.cloudfront.net/nixpkgs/nixpkgs-18.03pre119245.5cfd049a03/nixexprs.tar.xz’... [0/0 KiB, 0.0 KiB/s]
error: unable to download ‘https://d3g5gsiof5omrk.cloudfront.net/nixpkgs/nixpkgs-18.03pre119245.5cfd049a03/nixexprs.tar.xz’: Problem with the SSL CA cert (path? access rights?) (77)
cannot fetch ‘https://d3g5gsiof5omrk.cloudfront.net/nixpkgs/nixpkgs-18.03pre119245.5cfd049a03/nixexprs.tar.xz’

Pre-emptively:

$ nix-channel --list
nixpkgs https://nixos.org/channels/nixpkgs-unstable

[masaeedu]

As a workaround, I can do:

SSL_CERT_FILE=/usr/ssl/certs/ca-bundle.crt nix-channel --update

Which successfully downloads and unpacks the tarball.

[s50600822]

I am getting this error too, not sure if it wasn't able to download or wasn't able to unpack with this msg though
SSL_CERT_FILE=/usr/ssl/certs/ca-bundle.crt nix-channel --update

unpacking channels...
warning: unable to download 'https://cache.nixos.org/s8r0ir2anvg1n8i082zly4qq9j035hyc.narinfo': Problem with the SSL CA cert (path? access rights?) (77); retrying in 260 ms
warning: unable to download 'https://cache.nixos.org/s8r0ir2anvg1n8i082zly4qq9j035hyc.narinfo': Problem with the SSL CA cert (path? access rights?) (77); retrying in 624 ms
warning: unable to download 'https://cache.nixos.org/s8r0ir2anvg1n8i082zly4qq9j035hyc.narinfo': Problem with the SSL CA cert (path? access rights?) (77); retrying in 1190 ms
warning: unable to download 'https://cache.nixos.org/s8r0ir2anvg1n8i082zly4qq9j035hyc.narinfo': Problem with the SSL CA cert (path? access rights?) (77); retrying in 2437 ms
warning: unable to download 'https://cache.nixos.org/s8r0ir2anvg1n8i082zly4qq9j035hyc.narinfo': Problem with the SSL CA cert (path? access rights?) (77); retrying in 4849 ms
warning: unable to download 'https://cache.nixos.org/s8r0ir2anvg1n8i082zly4qq9j035hyc.narinfo': Problem with the SSL CA cert (path? access rights?) (77); retrying in 10743 ms

[psoberoi]

This just happened to me on Fedora 32. The problem was that the /etc/ssl/certs/ca-certificates.crt existed on my system, even though it shouldn't have (it was three years old; not sure how it got there). Deleting the file fixed the problem, but either of these also worked:

export NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt
export NIX_SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt

There is a list of locations that .nix-profile/etc/profile.d/nix.sh checks, and it starts with /etc/ssl/certs/ca-certificates.crt - which is the value on Debian/Ubuntu. So if this file exists accidentally, as in my case, the results can be bad.

The only solutions I can think of are:

1. set the path using the output of lsb_release -i
2. instead of just checking for the presence of the file, try to use it (the install script would need to do this check and pick the right path)

[ethancedwards8]

I have this issue on MacOS Big Sur. Nothing has worked so far. Surprisingly my ~/nix-profile/etc/profile.d/nix.sh is gone. Nix is installed as multi-user mode and on an "unencrypted /nix" volume. Nothing suggested in this thread has worked so far.