Nix: hash mismatch in downloaded path

[domenkozar]

TLDR; keep all of your binary cache history or don't use binary-cache-v3.sqlite:

hash mismatch in downloaded path '/nix/store/nmhvs4pihm56qnhpb6j09ajcwxjx9h07-digest-0.0.1.2': expected 8afe6105f80feb8f8b9e176bace053da40714598125b1201f91775fc9b1c22f4, got 772e334daa6beff675bee87adaaa2d38e45c0ccd15f065dad32ac88c22cd6637

Turns out what happened:

1. Hydra built /nix/store/nmhvs4pihm56qnhpb6j09ajcwxjx9h07-digest-0.0.1.2
2. Build slave substituted it and wrote the hash to binary-cache-v3.sqlite
3. Hydra GC-ed /nix/store/nmhvs4pihm56qnhpb6j09ajcwxjx9h07-digest-0.0.1.2
4. Months later someone wanted the same old package and Hydra rebuilt /nix/store/nmhvs4pihm56qnhpb6j09ajcwxjx9h07-digest-0.0.1.2
5. Build slave downloaded the new narinfo, but binary-cache-v3.sqlite had the old hash

I presume this is due to binary non-determinism?

The fix is: rm /nix/var/nix/binary-cache-v3.sqlite*

[cleverca22]

sqlite> .mode line
sqlite> select * from NARs where url = 'nar/0p7x4l2zab62bghdhc5ssbqi7kpwnblj-deriving-compat-0.3.6';
      cache = 2
  storePath = 0p7x4l2zab62bghdhc5ssbqi7kpwnblj-deriving-compat-0.3.6
        url = nar/0p7x4l2zab62bghdhc5ssbqi7kpwnblj-deriving-compat-0.3.6
compression = xz
   fileHash = 
   fileSize = 
    narHash = sha256:0yifcx0975amb366q6nshb5qm2rrppy2f7h7k4q66hlblxr8m3rp
    narSize = 4655744
       refs = 0p7x4l2zab62bghdhc5ssbqi7kpwnblj-deriving-compat-0.3.6 6bw3rq0xhayyj9dl158rqwi8ap395aa0-transformers-compat-0.5.1.4 73n08rwpzjqwvyizwphybcjfd85pkvqh-ghc-8.0.2 fjn0g0q5wrphysg3ijydijcpi274nwi8-gmp-6.1.2 l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25
    deriver = 8f35pf501k765mdz6vjwg1pibmgxckwp-deriving-compat-0.3.6.drv
   signedBy = hydra.iohk.io
  timestamp = 1514632193

$ curl https://hydra.iohk.io/0p7x4l2zab62bghdhc5ssbqi7kpwnblj.narinfo
StorePath: /nix/store/0p7x4l2zab62bghdhc5ssbqi7kpwnblj-deriving-compat-0.3.6
URL: nar/0p7x4l2zab62bghdhc5ssbqi7kpwnblj-deriving-compat-0.3.6
Compression: xz
NarHash: sha256:0qsf2vki8g6p3lhn1jqn1vpq8s4lqanjmnfqmg88m9pc7ycgbjmm
NarSize: 4655736
References: 0p7x4l2zab62bghdhc5ssbqi7kpwnblj-deriving-compat-0.3.6 6bw3rq0xhayyj9dl158rqwi8ap395aa0-transformers-compat-0.5.1.4 73n08rwpzjqwvyizwphybcjfd85pkvqh-ghc-8.0.2 fjn0g0q5wrphysg3ijydijcpi274nwi8-gmp-6.1.2 l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25
Deriver: 8f35pf501k765mdz6vjwg1pibmgxckwp-deriving-compat-0.3.6.drv
System: x86_64-linux
Sig: hydra.iohk.io:fNL1+eP/1CVem0xispVMChbktbEjujeAI0Eu4Te8ka21GRg2zUu+lKnp30Eb6lQjiqC83YVc7DEkEJ5M26B9BQ==

[domenkozar]

lesson learned: don't GC your binary cache

[AmineChikhaoui]

Or probably introduce a config for reducing the disk cache ttl for both positive/negative lookups. Which should be useful for binary caches that are GCed.

[cleverca22]

https://github.com/NixOS/nix/blob/1.11.15/scripts/download-from-binary-cache.pl.in

i think it can only expire negative hits, so positives stick around forever

nix 2.0 will expire things better

[domenkozar]

So https://github.com/NixOS/nix/blob/c0015e87af70f539f24d2aa2bc224a9d8b84276b/src/libstore/nar-info-disk-cache.cc#L52 needs a config setting to tweak it.

[Gabriel439]

There is another fix which doesn't require rm /nix/var/nix/binary-cache-v3.sqlite. You can instead add --fallback to your nix-build command or (better) add --option build-fallback true. The latter is more robust because it works for import-from-derivation, too.

See this other issue, which is related to this one: https://github.com/NixOS/nix/issues/1148

[blitz]

@domenkozar We just hit this in our system as well. Was there some activity on this issue since last year? Or is the solution still to not expire things on the cache server?

[domenkozar]

See narinfo-cache-positive-ttl in man nix.conf

[roberth]

Looks like a duplicate of https://github.com/NixOS/nix/issues/2035 - or the other way around I guess

[domenkozar]

I really think the correct fix is for binary cache to advertise this setting and user min(user_setting, cache_setting).

[roberth]

Can't we just invalidate the entry when it turns out to be unusable and retry the substitution?

That's what @volth suggested in https://github.com/NixOS/nix/issues/2035#issuecomment-37788632 and it appears to solve this issue too.

[domenkozar]

Sure, that would be most reliable. Note that in IFD we'd have to make sure it works too.

[roberth]

Why do you think IFD is different? Afaik, it builds and substitutes normally.

[domenkozar]

I don't know the code base, I want to be sure IFD is tested.

[domenkozar]

It seems like this would be easily added at https://github.com/NixOS/nix/blob/master/src/libstore/binary-cache-store.cc#L89 to fetch narinfo without using the cache and getting the nar file again.

[coderfromhere]

I've stumbled upon the same error message today. The original tar I depend on is https://github.com/NixOS/nixpkgs-channels/archive/f9eba87bf03318587df8356a933f20cfbc81c6ee.tar.gz, when I run:

nix-prefetch-url --unpack https://github.com/NixOS/nixpkgs-channels/archive/6e1393654240cca71efe1c7712c870aad1ab0792.tar.gz
unpacking...
[18.5 MiB DL]
path is '/nix/store/wcmdxyqq4m2761aqyrhz1v28mj5a0wdj-6e1393654240cca71efe1c7712c870aad1ab0792.tar.gz'
0vq5z3pmz5vf4637fhyg4ina3nhg29l94pnf962nw0b5lvwn9mxr

But when instantiating a nix-shell, I get the error:

% nix-shell             
unpacking 'https://github.com/NixOS/nixpkgs-channels/archive/f9eba87bf03318587df8356a933f20cfbc81c6ee.tar.gz'...
error: hash mismatch in file downloaded from 'https://github.com/NixOS/nixpkgs-channels/archive/f9eba87bf03318587df8356a933f20cfbc81c6ee.tar.gz':
  wanted: sha256:0vq5z3pmz5vf4637fhyg4ina3nhg29l94pnf962nw0b5lvwn9mxr
  got:    sha256:15cfqd3hargafkhfq5p9hw75dibnbajh06ha69k18scg9l1p1hnl
(use '--show-trace' to show detailed location information)

how can I keep the same references without breaking checksums in the future?

[domenkozar]

I've tried to reproduce this in https://github.com/NixOS/nix/pull/3969 but it seems to just work? Maybe my test is not testing the right thing?

[domenkozar]

I've found the problem, Nix will say it's copying it, but it will actually fallback to building it.

[avanov]

not sure if it helps, but I've just discovered this one:

building '/nix/store/j8z3r8nzig402vwdqww3f2a6bd05zc3q-source.drv'...

trying https://github.com/hashicorp/packer/archive/v1.6.2.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   124  100   124    0     0    558      0 --:--:-- --:--:-- --:--:--   558
100 16.5M    0 16.5M    0     0  3803k      0 --:--:--  0:00:04 --:--:-- 4288k
unpacking source archive /private/var/folders/7s/_k9r0mwd15b4pr4h78v5x2km0000gn/T/nix-build-source.drv-0/v1.6.2.tar.gz
hash mismatch in fixed-output derivation '/nix/store/1x9w8wf4qfgf2xnmafjmqd2mshzk47hr-source':
  wanted: sha256:104jw2jcshzds74d7m4yqn6mbk7lgps6qnqmp6h395b1mdyjhink
  got:    sha256:0kr9whv3s0f8866yjmwg311j3kcj29bp5xwpnv43ama4m1mq3bm7
cannot build derivation '/nix/store/f8idr16gzbh6nbh546ym1d2qyiyi3907-packer-1.6.2.drv': 1 dependencies couldn't be built
error: build of '/nix/store/f8idr16gzbh6nbh546ym1d2qyiyi3907-packer-1.6.2.drv', '/nix/store/wp749d94mr8zx1cpjm388k9ggnbc2aay-ghc-8.10.2-with-packages.drv' failed

MacOS, the pinned nixpgs are:

builtins.fetchTarball {
        name = "nixpkgs-unstable-2020-09-01";
        url = "https://github.com/NixOS/nixpkgs-channels/archive/780da9407e6da081b505f299681c78833357ebe0.tar.gz";
        sha256 = "0l78bgsil1h5i390d34x5qh3raji0wdxqhsqbqs3ihyi6i6c2403";
    };

but then it works well with a more recent https://github.com/NixOS/nixpkgs-channels/commit/a0b3103d81bcd5049dc51598314fc437f6f317d1

builtins.fetchTarball {
        name = "nixpkgs-unstable-2020-09-01";
        url = "https://github.com/NixOS/nixpkgs-channels/archive/a0b3103d81bcd5049dc51598314fc437f6f317d1.tar.gz";
        # Hash obtained using `nix-prefetch-url --unpack <url>`
        sha256 = "0fcsc4qkqd3m4m24wyypppqiv5z06b6wsvsfa90x4qfzbadgdc0x";
    }

Both checkouts were marked as "verifed" on github.

Maybe my reasoning is not right and it is possible to have hash mismatches for some packages in nixpkgs-unstable? Does CI make sure that the entire package set is consistent with their checksums?

[roberth]

@coderfromhere @avanov The issues you're seeing seem to be distinct from the original issue discussed here, because your mismatches occur in fetchers and fixed-output derivations, not during substitution.

These are usually caused by changes to the contents backing those URLs you're fetching from. For example, maintainers can change git tags or re-upload an "improved" release tarball and cause a hash mismatch that way. The reason it works on the recent commit, is because it was fixed four days ago.

@coderfromhere In your case, a prefetching command should be able to produce the correct hash. Please make sure that you're using the same parameters in your command and expression and that you're using a fixed-output derivation, not an evaluation-time fetcher like fetchurl. If the issue remains, please open a separate issue.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/hash-mismatch-in-downloaded-path/11056/1

[domenkozar]

The minimum actionable here is to improve the error message.