Nixpkgs: A function to compose overlays?

[michalrus]

Wouldn’t it be useful to have a function in lib to compose overlays? It’s short to define, but perhaps non-trivial (given we have no types in Nix — nb., I’m not sure if my definition is really correct):

{
  composeOverlays = overlays: self: super:
    lib.foldl' (lib.flip lib.extends) (lib.const super) overlays self;
}

It would also be more in line with what is promised in these slides: https://nbp.github.io/slides/NixCon/2017.NixpkgsOverlays/ (→ slide 25).

This way the user can define expression files that themselves are valid overlays (not lists of overlays) and which themselves comprise of other valid overlays.

I.e. the user can compose the overlays…

[michalrus]

/cc @edolstra @nbp @peti @copumpkin @ttuegel

[michalrus]

I’m not sure if my definition is really correct

It seems to be fine.

With it, you can do even nested overlays that work as expected (and definition sites need not—and should not—know about the nesting):

https://github.com/michalrus/dotfiles/blob/c5c352b0dccb70a0515277ab79f591080cd03e6c/nixos-config/overlays/default.nix#L25-L49.

It’s not super convenient, but at least it’s something. ¯\_(ツ)_/¯

[ttuegel]

About the name: this is not only useful to compose overlays, but also Haskell, Qt, and Emacs package sets. (All three use the self: super: pattern.)

[volth]

lib.composeExtensions (self: super: {}) overlays

[michalrus]

@volth, the comment says of two only? And to my eye, the implementation confirms that:

https://github.com/NixOS/nixpkgs/blob/0ea0e3cf719e2a238ecd5e9918cc30e2b642cce4/lib/fixed-points.nix#L46-L53

[volth]

oops, sorry.
lib.foldl' lib.composeExtensions (self: super: {}) overlays

[FRidh]

Reopening because I think such function should be added to lib.

[michalrus]

@FRidh thank you!

[volth]

@FRidh, the 2-argument function for that is already in the library: lib.composeExtensions
lib.foldl' just extends it to use with arbitrary number of arguments supplied as a list.

---

composeOverlays = lib.foldl' lib.composeExtensions (self: super: {})

is like

sum = lib.foldl' (a: b: a + b) 0
product = lib.foldl' (a: b: a * b) 1
...

It is a bit confusing and hard to read the zero-overlay which is a function (self: super: {}) but there is no magic

[michalrus]

@volth, would it be very terrible to add your definition under the name of composeNExtensions? It’d probably be better to name that one compose2Extensions and this one composeExtensions but… too late.

[copumpkin]

Eh, from the standpoint of "provide the simplest building blocks to be composed as needed by users" there's a lot of precedent (in other functional languages, math, etc.) for just providing the two-argument version and letting people fold the two together.

Here's a concept I like in this space:

The Fairbairn threshold is the point at which the effort of looking up or
keeping track of the definition is outweighed by the effort of rederiving
it or inlining it.
[...]
The primary use of the Fairbairn threshold is as a litmus test to avoid
giving names to trivial compositions, as there are a potentially explosive
number of them. In particular any method whose definition isn't much longer
than its name (e.g. fooBar = foo . bar) falls below the threshold.
[...]
The effect is to encourage simple combinators that can be used in multiple
situations, while avoiding naming the explosive number of combinations of
those combinators.

[michalrus]

@volth, emm, is the sum/product example an argument for or against? Other fine libraries do define them:

https://hackage.haskell.org/package/base-4.10.1.0/docs/Data-Foldable.html#v:sum

@copumpkin, if this is not needed in general, fine. I can define it myself, no problem. However, intuitively (but my intuition in Nixpkgs is very lacking!), I would say that from an OS declaration standpoint, I’d be much more likely to use the list version rather than the pair version. If you could quickly, in 1-2 sentences, discuss why the pair version is more useful in general, I’d love to read that!

[peti]

👎 on adding a function that's just a trivial fold over composeExtensions. I'd be in favor of extending the documentation of composeExtensions to point out this use-case, but adding yet another function to accomplish nothing that can't be accomplished already is not a good idea IMHO.

[volth]

@michalrus

@volth, emm, is the sum/product example an argument for or against?

just an implementation comment highlighting what we already have. I have no opinion on which of the versions should be a base building block. You might want to git blame the author of the original function to discuss

[michalrus]

@peti @copumpkin @volth, why is there lib.map that’s also trivial to define using fold? (ﾉ^_^)ﾉ

Adding documentation is probably even more work than adding the function.

But, I guess, the real question is why that lib.composeExtensions function takes a pair and not a list. :thinking: Why is that more useful in declaring an operating system? If I want to compose just two, there are… 2-element lists available out there?

[michalrus]

Or maybe!

I’d like to propose that we add my trivial fold and remove this silliness, in line with your argumentation, if I get it right:

https://github.com/NixOS/nixpkgs/blob/bdb2985949714edb43e63f6ec9df5a9218369917/lib/lists.nix#L80-L94

Net, we’d all win. :grinning:

[FRidh]

Eh, from the standpoint of "provide the simplest building blocks to be composed as needed by users" there's a lot of precedent (in other functional languages, math, etc.) for just providing the two-argument version and letting people fold the two together.

I agree, although I would like to note that this is Nixpkgs, not Nix. Here we have a wider audience to consider.

emm, is the sum/product example an argument for or against? Other fine libraries do define them:

In my opinion overlays is a fundamental concept of how we (want to) do package sets in Nixpkgs, and therefore I think it should become trivial to merge extensions using a single function, just as most languages offer a sum or product function for their fundamental building blocks: integers and floats.

Now that we recommend using overlays in various places throughout Nixpkgs, I don't think users should need to know or learn how to use a higher-order function like fold. Sure, this new function would still be a higher-order function but thinking in overlays it becomes trivial to understand how it should be used.

[Andrei-Pozolotin]

+1

[michalrus]

Is containers a fine library? =0

http://hackage.haskell.org/package/containers-0.5.11.0/docs/Data-Set.html#g:5

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/infinite-recursion-when-composing-overlays/7594/1

[deliciouslytyped]

Common idioms form a domain specific language. If people keep independently rediscovering and implementing their own helper functions (as linked above), that should be obvious and definitive evidence of the need for a library function.

I think someone comfortable with it just needs to make the PR and get it merged. :)