Nix: Large files within source tree causes build to run out of memory

[gulrotkake]

If the size of a file or a sufficient number of files in a nix-build exceeds the amount of memory available, Nix runs out of memory and gives up. For example, given this simple build:

{ stdenv, pkgs, ... } :

stdenv.mkDerivation rec {
  name = "huge";
  src = ./.;
  buildInputs = [ ];
  buildPhase =
    ''
        md5sum huge.bin > md5.result
    '';

  installPhase =
   ''
     mkdir $out
     mv md5.result $out
  '';
}

where huge.bin is a very huge file, nix will first warn:

warning: dumping very large path (> 256 MiB); this may run out of memory

before eventually running out of memory.

The above was tested with

nix-build (Nix) 1.8pre3834_9b146a5

[edolstra]

As a workaround, you could use fetchurl to get the file, e.g. fetchurl { src = file:///bla/huge; sha256 = ...; }.

[shlevy]

Bonus feature: hole detection to avoid spurious reads.

[edolstra]

I don't think there are many realistic use cases where a Nix source file contains holes...

[domenkozar]

It would be nice if it printed out the file name

[sjmackenzie]

Agreed, the file name would be most handy.

[sjmackenzie]

We have 50 instances of src = ./. each time the ./. points to the root of a rust project which consists of 4 files, one of which is in a folder called src. We have no instances of a large file/binary called with ./., just many (and growing quickly) instances of (./.).

[Ericson2314]

Why does it even read the files into memory at once? I assume the hash can be computed with constant space.

[sjmackenzie]

For the record, we found the cause: it was a .target folder generated by the rustc compiler. Deleting that folder straightened nix again.

[domenkozar]

@Ericson2314 https://github.com/NixOS/nix/pull/619

[domenkozar]

Until this is fixed in Nix see https://nixos.org/wiki/How_to_add_files_to_the_nix-store#Large_files

[ip1981]

As a workaround, you could use fetchurl to get the file, e.g. fetchurl { src = file:///bla/huge; sha256 = ...; }.

There might be permission problem.

[fisx]

Until this is fixed in Nix see https://nixos.org/wiki/How_to_add_files_to_the_nix-store#Large_files

dead link, and it looks like this page does not exist on https://nixos.wiki/?

[dpc]

Is there any way to make src = ./. except for ./target;?

[sjmackenzie]

take a look at cleanSource and filterSource

[dpc]

src = builtins.filterSource (p: t: lib.cleanSourceFilter p t && baseNameOf p != "target") ./.;

I think this is it, in case anyone lands here from google search.

[langston-barrett]

I am also experiencing this problem while trying to install Mathematica.

Here's the old Wiki page on archive.org in case anyone else is searching for it: https://web.archive.org/web/20160829175307/https://nixos.org/wiki/How_to_add_files_to_the_nix-store

[bgamari]

This is addressed by #619.

[Pitometsu]

Looks as a solution for me: https://stackoverflow.com/a/39493330/3195266