Nixpkgs: Get rid of sha1 for fixed-output derivations in nixpkgs

[grahamc]

Issue description

We're in 2020 and:

• chosen-prefix sha1 collisions can now be found for as little as $45,000
• a classic collision can be found for as little as $11k

Biggest usage of sha1 is files generated for Node packages:

2372    pkgs/development/node-packages/node-packages-v10.nix
1741    pkgs/servers/web-apps/codimd/yarn.nix
1644    pkgs/applications/version-management/gitlab/yarnPkgs.nix
1485    pkgs/servers/monitoring/prometheus/webui-yarndeps.nix
1481    pkgs/servers/gotify/yarndeps.nix
476 pkgs/development/tools/yarn2nix-moretea/yarn2nix/yarn.nix
421 pkgs/applications/networking/cluster/spacegun/node-packages.nix
290 pkgs/servers/rippled/package.nix
213 pkgs/development/compilers/elm/packages/node-packages.nix
189 pkgs/development/web/remarkjs/node-packages.nix
169 pkgs/development/mobile/androidenv/generated/packages.nix
141 pkgs/development/node-packages/node-packages-v12.nix
127 pkgs/applications/networking/instant-messengers/riot/riot-desktop-yarndeps.nix
116 pkgs/servers/matrix-synapse/matrix-appservice-slack/node-packages.nix
88  pkgs/misc/base16-builder/node-packages-generated.nix
74  pkgs/development/mobile/androidenv/generated/addons.nix
73  pkgs/development/node-packages/node-packages-v13.nix
70  pkgs/development/compilers/graalvm/default.nix
66  pkgs/applications/networking/instant-messengers/matrix-recorder/node-packages.nix
48  pkgs/tools/networking/airfield/node-packages.nix
48  pkgs/development/misc/google-clasp/node-packages.nix
36  pkgs/development/mobile/androidenv/generated/system-images-android.nix
33  pkgs/servers/web-apps/cryptpad/node-packages-generated.nix
33  pkgs/development/mobile/androidenv/generated/system-images-google_apis.nix
10  pkgs/development/mobile/androidenv/generated/system-images-android-tv.nix
6   pkgs/tools/package-management/nixui/node-packages.nix
6   pkgs/development/mobile/androidenv/generated/system-images-google_apis_playstore.nix
6   pkgs/development/mobile/androidenv/generated/system-images-android-wear.nix
6   pkgs/development/mobile/androidenv/convertaddons.xsl
4   pkgs/development/mobile/androidenv/generated/system-images-android-wear-cn.nix
2   pkgs/development/mobile/androidenv/convertpackages.xsl
2   pkgs/applications/networking/browsers/mozilla-plugins/google-talk-plugin/default.nix
1   pkgs/tools/typesetting/tex/texlive/default.nix
1   pkgs/development/tools/yarn2nix-moretea/yarn2nix/lib/generateNix.js
1   pkgs/development/tools/unity3d/default.nix
1   pkgs/development/mobile/androidenv/convertsystemimages.xsl
1   pkgs/development/libraries/wxsqliteplus/default.nix
1   pkgs/development/libraries/wxsqlite3/default.nix
1   pkgs/development/haskell-modules/configuration-hackage2nix.yaml
1   pkgs/development/haskell-modules/configuration-common.nix
1   pkgs/build-support/vm/rpm/rpm-closure.pl
1   pkgs/applications/office/grisbi/default.nix
1   pkgs/applications/graphics/gcolor2/default.nix
1   doc/release-notes.xml

Steps to resolve, higher level

• [ ] generator tools should be updated to use better hashes:
  
  
  
  • [ ] node2nix
  
  
  • [ ] yarn2nix
  
  
• [ ] generator tools should be updated to print out how they were generated:
  
  
  
  • [ ] yarn2nix-moretea.yarn2nix
  
  
• [ ] update all the packages which use generated dependency files
• [ ] update all the packages which use sha1 without a generator

Files to address

• [x] pkgs/applications/graphics/gcolor2/default.nix
• [x] pkgs/applications/networking/browsers/mozilla-plugins/google-talk-plugin/default.nix
• [ ] pkgs/applications/networking/cluster/spacegun/node-packages.nix
• [ ] pkgs/applications/networking/instant-messengers/matrix-recorder/node-packages.nix
• [ ] pkgs/applications/networking/instant-messengers/riot/riot-desktop-yarndeps.nix
• [x] pkgs/applications/office/grisbi/default.nix
• [ ] pkgs/applications/version-management/gitlab/yarnPkgs.nix
• [x] pkgs/build-support/vm/rpm/rpm-closure.pl
• [ ] pkgs/development/compilers/elm/packages/node-packages.nix
• [ ] pkgs/development/compilers/graalvm/default.nix
• [x] pkgs/development/haskell-modules/configuration-common.nix (false positive)
• [x] pkgs/development/haskell-modules/configuration-hackage2nix.yaml (false positive)
• [x] pkgs/development/libraries/wxsqlite3/default.nix
• [x] pkgs/development/libraries/wxsqliteplus/default.nix
• [ ] pkgs/development/misc/google-clasp/node-packages.nix
• [ ] pkgs/development/mobile/androidenv/convertaddons.xsl
• [ ] pkgs/development/mobile/androidenv/convertpackages.xsl
• [ ] pkgs/development/mobile/androidenv/convertsystemimages.xsl
• [ ] pkgs/development/mobile/androidenv/generated/addons.nix
• [ ] pkgs/development/mobile/androidenv/generated/packages.nix
• [ ] pkgs/development/mobile/androidenv/generated/system-images-android.nix
• [ ] pkgs/development/mobile/androidenv/generated/system-images-android-tv.nix
• [ ] pkgs/development/mobile/androidenv/generated/system-images-android-wear-cn.nix
• [ ] pkgs/development/mobile/androidenv/generated/system-images-android-wear.nix
• [ ] pkgs/development/mobile/androidenv/generated/system-images-google_apis.nix
• [ ] pkgs/development/mobile/androidenv/generated/system-images-google_apis_playstore.nix
• [ ] pkgs/development/node-packages/node-packages-v10.nix
• [ ] pkgs/development/node-packages/node-packages-v12.nix
• [ ] pkgs/development/node-packages/node-packages-v13.nix
• [ ] pkgs/development/tools/unity3d/default.nix
• [ ] pkgs/development/tools/yarn2nix-moretea/yarn2nix/lib/generateNix.js
• [ ] pkgs/development/tools/yarn2nix-moretea/yarn2nix/yarn.nix
• [ ] pkgs/development/web/remarkjs/node-packages.nix
• [ ] pkgs/misc/base16-builder/node-packages-generated.nix
• [ ] pkgs/servers/gotify/yarndeps.nix
• [ ] pkgs/servers/matrix-synapse/matrix-appservice-slack/node-packages.nix
• [ ] pkgs/servers/monitoring/prometheus/webui-yarndeps.nix
• [ ] pkgs/servers/rippled/package.nix
• [ ] pkgs/servers/web-apps/codimd/yarn.nix
• [ ] pkgs/servers/web-apps/cryptpad/node-packages-generated.nix
• [ ] pkgs/tools/networking/airfield/node-packages.nix
• [ ] pkgs/tools/package-management/nixui/node-packages.nix
• [ ] pkgs/tools/typesetting/tex/texlive/default.nix

[edolstra]

We can script most of this by using maintainers/scripts/find-tarballs.nix to get all the URLs/hashes, fetching them and then doing s/$sha1/$sha256 on the Nix expression. Or use tarballs.nixos.org to map sha1 hashes to sha512:

$ curl -v https://tarballs.nixos.org/sha1/e410a52dcff3d5c6c3d448b68a026d04ccd744be
location: /sha512/90ae204071cdab3b5e0da28709afa8ac0a5ee13737254cded76a6b8cf323ed2b42390aaa10dbb7935b2aaa57504ceba21306a7eaeea088ab85ee9bdb0a05b530

[cdepillabout]

@grahamc Thanks for looking into this. I think these types of security related fixes are important!

I was wondering how you determined the list of files using sha1?

I wanted to fix the haskell-related files (pkgs/development/haskell-modules/configuration-common.nix and pkgs/development/haskell-modules/configuration-hackage2nix.yaml), so I grepped through the files looking for the strings sha1 and hash, but I couldn't find anything suspicious. Also configuration-hackage2nix.yaml doesn't appear to have any hashes at all.

[grahamc]

I did a simple search of sha1 =. It has a few false positives, for sure :) Please tick them as done if you find them!

[7c6f434c]

git grep -l -E '(^| )sha1\> *= *' should catch less false positives, I believe.

Once we clean up all the generators, we can rerun with \<sha1\> *= as a safety check.

[MetaDark]

All the packages in node-packages-v10.nix that use sha1, only do so because they don't list a sha512 hash in the npm registry. Regenerating this file is already slow enough, if we wanted to use sha256 or sha512, node2nix (and yarn2nix) would also have to download the files.

For example, acorn 4.0.13 doesn't have a sha512 hash:

nix-shell -p curl jq --run 'curl http://registry.npmjs.org/acorn/4.0.13 | jq .dist'

```json
{
"shasum": "105495ae5361d697bd195c825192e1ad7f253787",
"tarball": "https://registry.npmjs.org/acorn/-/acorn-4.0.13.tgz"
}

but version 7.1.0 does:

nix-shell -p curl jq --run 'curl http://registry.npmjs.org/acorn/7.1.0 | jq .dist'

```json
{
  "integrity": "sha512-kL5CuoXA/dgxlBbVrflsflzQ3PAas7RYZB52NOm/6839iVYJgKMJ3cQJD+t2i5+qFa8h3MDpEOJiS64E8JLnSQ==",
  "shasum": "949d36f2c292535da602283586c2477c57eb2d6c",
  "tarball": "https://registry.npmjs.org/acorn/-/acorn-7.1.0.tgz",
  "fileCount": 11,
  "unpackedSize": 1104477,
  "npm-signature": "-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJdibjuCRA9TVsSAnZWagAAmQgQAIkJPo9i+wdBLLZY8yjT\nHI82RsWU5OvIQxhE6H9qLiEfF57ZSn3LfKVugsEocc84loCiJkkd7vsGsidJ\nRMbkEIK2ENjH1VcUHdHq8PnA/HgnMgHNq6nGg+h6aXLsN/H5SFdyvWedoLXw\n5bwhLbfURV/00bx2himrt/RBNL8SxfowdpQ/ps5+mAWUzLTf7D2+PchYktve\nRJ1lSwpshrgu4AxUdbRmqDEH7+Wm5SutB/EMkQPxQV/sJviFzB8qqFlQopJS\n83RuvIhQb7ZNUQeh5A1+pQocFU+yd227m9tAr69UeES8vinI+0FYVh5GNh0y\nk5MoiQiG3LROtO1RNea0uYGv8W7cU9CK/6MnHbDMUgr+fpEeU/VGItZfOLhB\nTg10PpYCKtQCsF+Z36YLTXgxeWEoRvvKWmJET/MDQzBM/pRzMRpPHiWn6E9T\nl9wt1WLix2qM1jymwaq+0H6hcCtu0ioY7qn4GGeCAfVlBxWYcydNR11Rb++n\nO3oXEqhc8oxKy+hIDHIMDDz/YfCrJ9q7ZzMO9Ie5dzOqeLxty4fbuDeLcD9U\nkbuA+I8dfEFmO2UtRwyx3k9yjX2Tsf2oAC8VOTINN8TXiauyTjWyU//lSKDX\nmUHBDXXZhKWgygi3TMNW7gvmzYuzmDcbkienQeoADvONpCuJNmraVaC33n1h\ncQmj\r\n=blZm\r\n-----END PGP SIGNATURE-----\r\n"
}

[vcunat]

texlive: it has one "sha1 =" string but the code is generic – pkgs/tools/typesetting/tex/texlive/fixedHashes.nix has over 8k sha1 hashes. I haven't looked into complications when generating these, but if I look right, the file's size would increase roughly 460k -> 624k. (Seems acceptable, I suppose, given the circumstances.)

[worldofpeace]

@grahamc I removed this as a blocker because I'm not convinced it's something we should block release on. Thoughts?

[prusnak]

• [x] pkgs/applications/networking/browsers/mozilla-plugins/google-talk-plugin/default.nix is not an issue since c415d67284ad3b621a42c88e0a9bb5838708b14e has been merged

[prusnak]

• [x] pkgs/build-support/vm/rpm/rpm-closure.pl is a false positive

[prusnak]

• [x] pkgs/applications/office/grisbi/default.nix -> https://github.com/NixOS/nixpkgs/pull/86966

[prusnak]

• [x] pkgs/development/libraries/wxsqliteplus/default.nix -> https://github.com/NixOS/nixpkgs/pull/86969
• [x] pkgs/development/libraries/wxsqlite3/default.nix -> https://github.com/NixOS/nixpkgs/pull/86969