Nixpkgs: [Discussion] On updating Leksah — and packages that define nontrivial default.nix in their repo

[michalrus]

Issue description

Nixpkgs’ Leksah definition is broken.

In Leksah itself, we’ve got a non-trivial default.nix, which you can see here:

• https://github.com/leksah/leksah/blob/8ed4ea9ff78218c9fad5c3fd27d6632a125e5379/default.nix
• https://github.com/leksah/leksah/blob/8ed4ea9ff78218c9fad5c3fd27d6632a125e5379/shell.nix

There, nix-build run from the root directory works well, providing a working bin/leksah.

Now, for example, in _my_ configuration.nix I do:

{
  environment.systemPackages = with pkgs; [
    # Leksah
    (import (fetchFromGitHub {
      owner = "leksah";
      repo = "leksah";
      rev = "8ed4ea9ff78218c9fad5c3fd27d6632a125e5379";
      sha256 = "0wh7gx3gqjbvj15zs5n9v033zd33zl466w787pyfrzzz16jaxy8k";
      fetchSubmodules = true;
    }) {})
  ];
}

And this works well — for _my_ desktop machine.

Nixpkgs don’t want to allow “import-from-derivation”, cf.

• @peti in https://github.com/NixOS/nixpkgs/pull/24853#issuecomment-293992238,
• @shlevy in https://github.com/shlevy/rfcs/blob/6d3bc9f9f960079e2120030376f50c35cf783e88/rfcs/0008-readonly-recursive-nix.md#alternatives.

So my question is: what to do here?

_Copy-pasting_ Leksah’s default.nix to Nixpkgs each time it changes seems a rather… bad idea?

[michalrus]

/cc @hamishmack

[michalrus]

Note: such situations should be happening more and more commonly, as NixOS gets more mainstream… So a solution seems to be needed.

[lheckemann]

I think the best solution for this is to maintain a non-trivial default.nix upstream only if it's not included in nixpkgs, and to prefer overriding the expression from nixpkgs in the upstream repo otherwise — much like nox already does.

This has the benefit of resulting in a single source of truth on how to build the software (either nixpkgs or upstream, but not both). However, it requires coordination between upstream and nixpkgs; this may be unpleasant when larger changes are introduced. This could be somewhat mitigated through pinning nixpkgs in upstream expressions (as leksah already does) but I'm not sure this is an ideal solution.

Not sure about other ways to solve this problem, but I definitely think IFD is not the way to go here — we don't want to fetch a package's source just to get its metadata! :)

[peti]

Why is Nixpkgs's auto-generated definition for leksah broken?

[michalrus]

@lheckemann, uh, rrright, it seems unpleasant. =(

@peti:

• many GI packages need https://github.com/leksah/leksah/blob/8ed4ea9ff78218c9fad5c3fd27d6632a125e5379/default.nix#L32
• I’m not sure about auto-generation of the stdenv.isDarwin parts…
• Leksah needs version 0.0 of haskell-gi-overloading https://github.com/leksah/leksah/blob/8ed4ea9ff78218c9fad5c3fd27d6632a125e5379/default.nix#L83
• @hamishmack, I probably forgot something, could you verify?

[lheckemann]

@michalrus what exactly is the problem with it, given that pinning nixpkgs in the upstream expression can AFAIK solve the coordination problems?

[michalrus]

@lheckemann, hmm, so if I wanted to change something, I’d have to do a PR to Nixpkgs… and then I could temporarily pin upstream to my PR’s SHA1, even before it was merged, hmm. Well, maybe it’s not _that_ bad? ¯\_(ツ)_/¯

[michalrus]

But, on the other hand, default.nix in upstream can have src = ./. w/o the need to update r:sha256, hmmm. It’s convenient for iteratively changing that default.nix in parallel with code in src. Hmm.

[lheckemann]

Exactly — it's a fairly flexible approach and I think the complexity of coordinating versions that it involves is inherent to the problem as opposed to being an artifact of the particular solution — unless anyone can come up with a better approach, which I would highly appreciate 😄

[lheckemann]

And optimally I'd say upstream would always be pinned to some nixpkgs revision that is known to work, which allows that particular upstream revision to be built no matter how nixpkgs has changed since then.

[shlevy]

@michalrus The only long term solution here is recursive nix or allowing import-from-derivation in nixpkgs. There's no real alternative in the mean time besides copying the relevant changes in.

[lheckemann]

@shlevy wouldn't recursive nix have the same problem as IFD in this case, i.e. needing to download more stuff before a package's meta attribute can be evaluated? I don't see how that problem is solved in this case.

[michalrus]

@shlevy, ++! =)

@lheckemann, when is _meta_ needed, but _building_ not so much? You have to download it on a (Hydra?) machine to build it anyway, right?

[lheckemann]

meta is needed for nix-env -q --json, which feeds into search tools like nox, for instance. The setup you describe doesn't even allow evaluating the name of the package so even nix-env -qa would require downloading it.

[michalrus]

:sob: So, I guess, for now, I’d move leksah.git/default.nix to nixpkgs.git/…/leksah/default.nix and override src of the Nixpkgs version in our new leksah.git/default.nix…

[FRidh]

fetchTarball with a hash works well for this. I use that for some of my own packages, where I keep a default.nix in the source tree, and then fetchTarball that tree.

[lheckemann]

@FRidh I don't see how that solves this problem, which is specifically how to maintain a package that's supposed to be both in nixpkgs (= must be possible to evaluate without downloads) and in an upstream repo with minimal duplication.

[FRidh]

nixpkgs (= must be possible to evaluate without downloads)

Why this? There is no need for that. fetchTarball exactly solves this issue, its just not in Nix stable. It fetches and unpacks an archive during evaluation, but it is not import from derivation.

[lheckemann]

Because that makes evaluation (performed by nix-env -qa for instance) require downloading a tarball.

[FRidh]

And that is a problem? After that the archive will be cached. By the time we get to such an approach we'll likely create a separate set for such packages, and not recurse into them. Then there won't be a problem either.

[lheckemann]

Still seems like a problem to me — nix-env -qa won't work after gc if you don't have an internet connection, and with the separate set it just won't find the packages in question any more…

[FRidh]

with the separate set it just won't find the packages in question any more…

Exactly. In the end its a design choice. Do you want Nixpkgs to have everything that's out there, or do you accept other sets as well? Either entirely separate sets, or sets/items linked from Nixpkgs.

[shlevy]

There are middle grounds as well. We can include just name and metadata info in nixpkgs and fetch the rest, or we can teach querying tools to distinguish between packages with metadata fully available vs not and let the user decide which behavior they want. See also https://github.com/NixOS/rfcs/pull/14

[puffnfresh]

Nix 2.0 will have support for specifying a hash for fetchTarball (and fetchGit) - which I think changes from caching to using fixed-output derivations.

@lheckemann does this fix the problems with importing from a fetch?

[lheckemann]

@puffnfresh not really:

nix-env -qa won't work after gc if you don't have an internet connection

still applies AFAIUI. Unless we have some sort of "evaluation gc-root" thing which seems like a horrible idea to me.

[Ekleog]

(triage) I think the discussion here is more or less complete with the solutions currently available. Further comments about import-from-derivation or recursive-nix should likely be made over at the RFC repository.

Closing for now, feel free to reopen if you think the discussion wasn't actually over.