Nix: Lazy attribute names

Created on 29 Sep 2020  ·  3Comments  ·  Source: NixOS/nix

Nix could support evaluating attribute sets more lazily, such that e.g. the following works:

(throw "" // { x = 0; }).x
-> 0

Currently this doesn't work because attribute sets are evaluated strictly in all names, meaning all // operations and co. will have to be applied to figure out all names, making the above code throw in the current version.

Making attribute names lazy in that way has the potential to solve many existing problems:

  • .extend attributes of package scopes won't necessarily have to evaluate all package attributes. Similarly .override, etc. of packages won't have to call the derivation primop multiple times anymore
  • types.lazyAttrsOf might not be needed anymore
  • In overlays you'll be able to use self.lib for the attribute definitions if lib is added in a later overlay
  • lib.mkIf might not be necessary for the module system anymore
  • Potentially more

However this also changes semantics, so this should be thought out well.

I have implemented an initial prototype that supports above evaluation in https://github.com/Infinisil/nix/commit/afeda4fda1214f285083acad64e1d5425e8b42cb. I'll continue with this if I have time.

Ping @edolstra @Profpatsch

improvement

Most helpful comment

Turns out there was no infinite recursion in the last version, but instead it was so slow that I didn't think it would ever finish!

I worked more on this, with the latest version being in https://github.com/Infinisil/nix/tree/lazy-attr-names-v3. It's still about 40% slower than stable (tested on a firefox instantiation), but it's usable now! There are some other things to do though. I still have high hopes that it can improve the speed of Nix in general, though it might be pretty hard to get there.

All 3 comments

Gave this another implementation try in https://github.com/Infinisil/nix/commit/5dbf67cab037b3bb8302052fdda6534e5cf23d9b, which works much better than the previous one. For example:

let
  lib = import ./lib;

  set1 = lib.makeExtensible (self: builtins.trace "set1" {
    foo = 1;
  });

  set2 = set1.extend (self: super: builtins.trace "set 2" {
    bar = 2;
  });

  set3 = set2.extend (self: super: builtins.trace "set  3" {
    baz = 3;
  });

  set4 = set3.extend (self: super: builtins.trace "set   4" {
    qux = 4;
  });
in set4

Evaluating this in a current Nix version would print

trace: set1
trace: set1
trace: set 2
trace: set1
trace: set 2
trace: set  3
trace: set1
trace: set 2
trace: set  3
trace: set   4
{ __unfix__ = <LAMBDA>; bar = 2; baz = 3; extend = <CODE>; foo = 1; qux = 4; }

Whereas with above commit it just prints:

trace: set1
trace: set 2
trace: set  3
trace: set   4
{ __unfix__ = <LAMBDA>; bar = 2; baz = 3; extend = <CODE>; foo = 1; qux = 4; }

Therefore not evaluating all the overlays multiple times! Unfortunately there is a bug with this implementation which causes it to go into an infinite loop with nixpkgs derivations. Also, this implementation copy-pasted a bunch of code around, and probably leaks some memory and does additional unnecessary allocations here and there. Definitely not ready, but it's getting there :)

Turns out there was no infinite recursion in the last version, but instead it was so slow that I didn't think it would ever finish!

I worked more on this, with the latest version being in https://github.com/Infinisil/nix/tree/lazy-attr-names-v3. It's still about 40% slower than stable (tested on a firefox instantiation), but it's usable now! There are some other things to do though. I still have high hopes that it can improve the speed of Nix in general, though it might be pretty hard to get there.

Opened a draft PR to track further development: https://github.com/NixOS/nix/pull/4154

Was this page helpful?
0 / 5 - 0 ratings

Related issues

chexxor picture chexxor  ·  4Comments

matthewbauer picture matthewbauer  ·  3Comments

bflyblue picture bflyblue  ·  3Comments

fare picture fare  ·  4Comments

ericsagnes picture ericsagnes  ·  4Comments