Nixpkgs: How are you supposed to use fetchpatch?

[qolii]

Issue description

It seems unclear exactly how you are supposed to use fetchpatch. I can't find anything by searching.

Specifically, how are you supposed to determine the sha256 value in advance, as you use nix-prefetch-url to determine it for fetchurl?

Also, what really is purpose, above fetchurl? Git's commit hashes never change, so, any URL pointing to a GitHub commit will be immutable. Or at least, as immutable as any URL you might point something to. Right?

Finally, if you provide the wrong hash to fetchpatch inside a package derivation, that package fails to install with an error message – but only the first time you try it. The second time it will silently work.

Is all this documented somewhere? The contributor guide mentions fetchpatch but doesn't answer these questions I think.

[Anton-Latukha]

You can use nix-prefetch-url to determine sha256.

Or, when building, Nix errors with wrong sha256, it throws error:

fixed-output derivation produced path '/nix/store/8x6gjhdp9yww8cb589q0cqlg9wz1amaj-at-spi2-atk-2.26.2.tar.xz' with sha256 hash '0vkan52ab9vrkknnv8y4f1cspk8x7xd10qx92xk9ys71p851z2b1' instead of the expected hash '00akn52ab9vrkknnv8y4f1cspk8x7xd10qx92xk9ys71p851z200'
cannot build derivation '/nix/store/fnrv8b63wqy3p60yrdwgvg9hcrcy2giw-at-spi2-atk-2.26.2.drv': 1 dependencies couldn't be built
error: build of '/nix/store/fnrv8b63wqy3p60yrdwgvg9hcrcy2giw-at-spi2-atk-2.26.2.drv' failed

So it says that he got: '0vkan52ab9vrkknnv8y4f1cspk8x7xd10qx92xk9ys71p851z2b1
But I provided: 00akn52ab9vrkknnv8y4f1cspk8x7xd10qx92xk9ys71p851z200
So you can change it.

[Anton-Latukha]

You can revive https://github.com/NixOS/nixpkgs/issues/39392 discussion with fresh look perspective.

[jtojnar]

While the content of a patch provided by GitHub api should not change, details like the patch header are not guaranteed to stay the same. For that reason fetchpath tries to normalize the patch using filterdiff after downloading it:

https://github.com/NixOS/nixpkgs/blob/646767e9615bd3fff5d0765f011393008d31bc4d/pkgs/build-support/fetchpatch/default.nix#L10-L30

[orivej]

@qolii To get a hash for fetchpatch, you can not use nix-prefetch-url because it does not postprocess the patch before computing the hash. You have to use the second method described by @Anton-Latukha: write a wrong hash, then copy the right hash from the error that is printed when you attempt to build the package.

We should document this in the manual near https://nixos.org/nixpkgs/manual/#sec-patches

[qolii]

I'll take a look at #39392, thanks.

It seems like an oversight to have to construct a failing build in order to get at the hashes. And, as I found out in my recent PRs, if you've already downloaded patches with fetchurl, fetchpatch will find those and not even check the hashes you give it, making it very easy to PR incorrect hashes.

Would it be possible to make a fetchpatch equivalent of nix-prefetch-url? Or, since it doesn't exist already, nobody must want it very much. Am I missing something?

[jtojnar]

if you've already downloaded patches with fetchurl, fetchpatch will find those and not even check the hashes you give it

It is the other way around – it will check the hash and see that a content with such hash was already downloaded, skipping the second download. It worked for you because the uncleaned patch still applies. It failed on othersʼ computers since they did not have any content with such hash in their store and after they downloaded it, the hashes did not match.

[orivej]

I think that @qolii got it right: he meant that Nix will search for the patches in the /nix/store under the same names in either case, and will not validate the found patch against the sha256 hash of its contents.

[qolii]

@orivej, indeed! Does nobody else find this to be a problem?

[Ekleog]

(triage) Ugh what? @qolii or @orivej , can you provide a reproducer? That seems very weird.

[jtojnar]

@qolii well, that is how fixed-output derivations are supposed to work – the hash of the derivation is based solely on the value of its sha256 attribute. The only way we could ensure the hash matches the expression would be to build (download) the expression during every evaluation, which would defeat the purpose of fixed-output derivations.

[timokau]

For the sake of persistence: I suggested a change to the fixed-output mechanics on IRC which would make them more intuitive and fix the fetchpatch problem. My proposition would be to handle FO derivations just the same as regular derivations (i.e. hashing based on inputs), and then just check the output against the provided hash (but not using that hash for the $out name, or maybe only using it as an alias).

That way the usual nix semantics apply: If you change an input (curl, patchutils, url, excludes etc.) the derivation is rebuilt. The hash is still checked though, containing the impurity. A simple nix-review could check if a change invalidated any hashes.

The counter arguments were

1) this would blow up the nix cache, since every change to e.g. git would generate a whole bunch of now distinct fixed output derivations. That should be fixable with deduplication.
2) it may get hydra ratelimited/banned on upstream platforms, since it would refetch all sources every now and then

(2) probably makes this infeasible :/

[timokau]

Sorry, meant to post that to https://github.com/NixOS/nixpkgs/issues/48567

[ckauhaus]

I think this discussion should be continued on https://discourse.nixos.org/