Nixpkgs: Very slow evaluation and huge RAM usage with many nixos-containers

[davidak]

Describe the bug

Evaluation is very slow and uses huge amounts of RAM when having many nixos-containers.

| Number of containers | GB RAM used | time needed for build |
|----------------------|-------------|-----------------------|
| 10 | 7 | 5m19.569s |
| 20 | 9 | 4m42.692s |
| 50 | 15 | 6m10.281s |
| 60 | 17 | 5m55.762s |
| 80 | 20 | 7m3.104s |
| 100 | 24 | 7m30.186s |
| 100 | 60 | 38m8.577s |
| 200 | 85 | 218m30.125s |

This is real world data, not very scientific since i did not changed anything of the config in the first runs, so when 10 container configs are build, they don't have to be build again for 20. The last 2 values are with master and probably changes to the config.

You get the idea. If it helps to get pure evaluation time, i can do that.

I think i read somewhere that nixpkgs is evaluated for every single container even tho it's the same and actually can't be changed to a different channel. That problem might be related.

What can be done to speed it up and use less RAM? Are there related issues open?

To Reproduce

Here is the configuration: https://gist.github.com/davidak/7d099b7ad4b23f144e4e8fed07e0d4f6

I can simplify it to a single file if it helps!

Expected behavior

Build should be faster, especially when the containers are idendical except the name!

Metadata

• system: "x86_64-linux"
• host os: Linux 4.19.62, NixOS, 19.03.173202.31d476b8797 (Koi)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.2.2
• channels(root): "nixos-19.03.173202.31d476b8797, nixos-hardware, nixos-unstable-19.09pre186857.239fffc90d7"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

For the last 100 and 200, i used master with this patch: https://github.com/NixOS/nixpkgs/pull/65661

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module: containers

cc @edolstra @arianvp @grahamc

[danbst]

It is known issue. Each container is a separate NixOS evaluation.

There are two ways to move forward: 1) make NixOS evaluation less costly or 2) allow "multiple-configurations" evaluations.

1. This should be achieved by rewriting module system in a lower-level language, reducing number of modules in modules-list.nix or making most modules "lazy" (until we have services.service.enable = true; we should ignore these options and modules).

2. Multiple-configuration thing is to rewrite module system in a way, so that it can produce several configurations from one evaluation. This requires substantial changes to module system.

The most realistic way to fix your issue is to manage containers imperatively, not declaratively. If you know the differences between containers, you can apply those ad-hoc in imperative way, by copy-paste-apply, without NixOS evaluations for subsequent containers. This breaks good NixOS features, unfortunately.

[arianvp]

Wouldn't switching nixos containers to types.submodule help here a bit too? we would at least not pay the price of evaluating pkgs over and over again I think?

[danbst]

@arianvp maybe a bit, but then the ability to specify custom imports is lost.

As for multiple pkgs evaluation, there already exists https://github.com/NixOS/nix/commit/0395b9b94af56bb814810a32d680c606614b29e0

But your idea about not evaluating pkgs multiple times is nice:

$ time env N=10 nix-instantiate ./nixos  -A system --arg configuration '{lib, pkgs, ...}: { 
    boot.loader.grub.devices = ["nodev"]; 
    fileSystems."/".device = "nodev"; 
    containers = lib.genAttrs (map (i: "target${toString i}") (lib.range 1 (lib.toInt (builtins.getEnv "N")))) (_: { 
        /*config.nixpkgs.pkgs = lib.mkOverride 0 pkgs;*/ 
        config.services.postgresql.enable = true; 
    }); }'
real    0m27,960s
user    0m34,359s
sys     0m1,252s

$ time env N=10 nix-instantiate ./nixos  -A system --arg configuration '{lib, pkgs, ...}: { 
    boot.loader.grub.devices = ["nodev"]; 
    fileSystems."/".device = "nodev"; 
    containers = lib.genAttrs (map (i: "target${toString i}") (lib.range 1 (lib.toInt (builtins.getEnv "N")))) (_: { 
        config.nixpkgs.pkgs = lib.mkOverride 0 pkgs;
        config.services.postgresql.enable = true; 
    }); }'

real    0m24,623s
user    0m33,506s
sys     0m0,965s

[danbst]

@davidak oh, and btw :smile: :

$ time env N=10 nix-instantiate ./nixos  -A system --arg configuration '{lib, pkgs, ...}: { 
    boot.loader.grub.devices = ["nodev"]; 
    fileSystems."/".device = "nodev"; 
    containers = lib.genAttrs (map (i: "target${toString i}") (lib.range 1 (lib.toInt (builtins.getEnv "N")))) (_: { 
        config.nixpkgs.pkgs = lib.mkOverride 0 pkgs;
        config.services.postgresql.enable = true; 
        config.documentation.nixos.enable = false;                                                                 
    }); }'
real    0m8,067s
user    0m9,671s
sys     0m0,527s

[danbst]

Alright, I have a new record for evaluating 100 containers. It took around 7Gb RAM:

$ time env N=100 NIX_PATH= nix show-derivation -f ./nixos system --arg configuration '{lib, pkgs, ...}: { 
    boot.loader.grub.devices = ["nodev"]; 
    fileSystems."/".device = "nodev"; 
    documentation.nixos.enable = false; 

    containers = lib.genAttrs (map (i: "target${toString i}") (lib.range 1 (lib.toInt (builtins.getEnv "N")))) (_: { 
        config.nixpkgs.pkgs = lib.mkOverride 0 pkgs;
        config.services.postgresql.enable = true; 
        config.documentation.nixos.enable = false;
    }); }' > /dev/null

real    0m52,887s
user    1m15,342s
sys     0m3,144s

[davidak]

There are two ways to move forward:
1) make NixOS evaluation less costly

that would be great, but probably very hard

citing edolstra here: "KDE evaluation has gone from 1.2s to 6.5s in the last 3 years"

https://hydra.nixos.org/job/nixpkgs/trunk/metrics/metric/nixos.kde.time

Not because the system got slower, but because we have more modules and packages!

References: https://github.com/NixOS/nixpkgs/issues/57477

or 2) allow "multiple-configurations" evaluations.

I think this is expected when the same nixpkgs is used. Everything else is bad design.

But also hard to change. Not sure if it is a priority since only causing problems with containers, VMs and NixOPS.

But in the long term, this should be improved for the Nix ecosystem to be able to grow.

[davidak]

nixpkgs.pkgs = lib.mkOverride 0 pkgs; don't work in my container configuration.

I get: error: infinite recursion encountered, at /home/davidak/code/nixpkgs/lib/modules.nix:338:9

@danbst any idea why this is?

log: https://gist.github.com/davidak/330d9baf772581408709ca8c53bca958

[danbst]

@davidak the pkgs should from host config lexical scope. In your case it is probably from scope of container config. Can you show code?

[danbst]

target = {
autoStart = true;
config = {
imports = [ targetConfig ];
nixpkgs.pkgs = lib.mkForce pkgs;
};

[davidak]

Can you show code?

i just added it here

https://gist.github.com/davidak/7d099b7ad4b23f144e4e8fed07e0d4f6#file-minimal-target-container-configuration-nix-L18

[danbst]

@davidak you should add it in targets-image-configuration, as I showed above. Sorry for bad formatting, on the phone

[davidak]

Ah, it still uses the imported config. Now i understand what you mean. Thanks!

Build takes now only 1m42.618s and uses less than 20 GB RAM!

[danbst]

@arianvp if you still care, I've tried to rewrite containers as proper submodules. See https://github.com/NixOS/nixpkgs/pull/75031#issuecomment-563167497 for details and discussion. It had no performance gain compared to current solution.

@davidak I think things should be much better with default containers after https://github.com/NixOS/nixpkgs/pull/75659 is merged. Instantiation on my machine is 1.2s per container.

(The build of container definitely takes more time, but if nothing had changed then you only spend time in instantiation)