Nixpkgs: haskellPackages: failed to create OS thread: Cannot allocate memory
Describe the bug
I have a system where normal user environment comes with a following limitations:
ulimit -a
-t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) unlimited
-c: core file size (blocks) 0
-m: resident set size (kbytes) unlimited
-u: processes 1024
-n: file descriptors 4096
-l: locked-in-memory size (kbytes) unlimited
-v: address space (kbytes) 8388608
-x: file locks unlimited
-i: pending signals 1028239
-q: bytes in POSIX msg queues 819200
-e: max nice 0
-r: max rt priority 0
-N 15: unlimited
The builds of several haskell packages (e.g. hedgehog) fails with a failure during the checkPhase:
failed to create OS thread: Cannot allocate memory
To Reproduce
This can be also reproduced on NixOS 20.03 with following expressions:
with import <nixpkgs> {};
{
# test/Test.hs:137:16: error:
# • Couldn't match expected type ‘Double’ with actual type ‘Bool’
# • In the first argument of ‘cover’, namely ‘(haskDiff == utilDiff)’
# In the expression: cover (haskDiff == utilDiff) 90 "exact match"
# In the expression:
# cover (haskDiff == utilDiff) 90 "exact match"
# $ classify
# (haskDiff == utilDiff)
# "exact match"
# (div ((length haskDiff) * 100) (length utilDiff) < 110)
# |
# 137 | in cover (haskDiff == utilDiff) 90 "exact match" $
# | ^^^^^^^^^^^^^^^^^^^^
#
# builder for '/nix/store/r27mnlkwh4wi36rdvpi1jg1jp35z5pgd-Diff-0.3.4.drv' failed with exit code 1
cachix_ghc844 = haskell.packages.ghc844.cachix;
# Test suite test: RUNNING...
# ━━━ Test.Hedgehog.Applicative ━━━
# ✓ prop_StateT_outside passed 100 tests.
# ✓ 1 succeeded.
# ━━━ Test.Hedgehog.Confidence ━━━
# ✓ prop_with_confidence passed 300 tests.
# number == 1 33% ██████▋············· ✓ 20%
# test: failed to create OS thread: Cannot allocate memory
# Test suite test: FAIL
# Test suite logged to: dist/test/hedgehog-1.0.2-test.log
# 0 of 1 test suites (0 of 1 test cases) passed.
# builder for '/nix/store/z5sn9pxvbasmzk1is39aka55jks2nsnq-hedgehog-1.0.2.drv' failed with exit code 1
hedgehog_ghc865 = haskell.packages.ghc865.hedgehog.overrideAttrs (_: {
preCheck = "ulimit -v 8388608"; # limit vram to 8gb
});
hedgehog_ghc865_disableLargeAddressSpace = (haskell.packages.ghc865.override {
overrides = self: super: {
ghc = super.ghc.override { disableLargeAddressSpace = true; };
hedgehog = super.hedgehog.overrideAttrs (_: {
preCheck = "ulimit -v 8388608"; # limit vram to 8gb
});
};
}).hedgehog;
hedgehog_ghc883 = haskell.packages.ghc883.hedgehog.overrideAttrs (_: {
preCheck = "ulimit -v 8388608"; # limit vram to 8gb
});
# builder for '/nix/store/d7lj01v5pw0nwwwp53v05rm4djmkc622-constraints-0.10.1.drv' failed with exit code 1
hedgehog_ghc8101 = haskell.packages.ghc8101.hedgehog.overrideAttrs (_: {
preCheck = "ulimit -v 8388608"; # limit vram to 8gb
});
# builds, but cachix will fail to run beyond "cachix --help" (when starting several threads?)
hedgehog_ghc865_limitstack = haskell.packages.ghc865.hedgehog.overrideAttrs (_: {
preCheck = ''
ulimit -v 8388608 # limit vram to 8gb
ulimit -s 1024 # limit stack to 1mb
'';
});
}
Expected behavior
cachix and other packages build and work.
Metadata
% git rev-parse HEAD
10bfa0159c2f5941057420283adac2ed0ddc274e
- system: `"x86_64-linux"`
- host os: `Linux 5.4.25, NixOS, 20.09.git.cab6b019b17 (Nightingale)`
- multi-user?: `yes`
- sandbox: `yes`
- version: `nix-env (Nix) 2.3.3`
Maintainer information:
# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:
veprbl
All 11 comments
Some relevant issues:
veprbl
on 1 May 2020
Works in 19.09:
nix-repl> :b (import (builtins.fetchTarball "https://nixos.org/channels/nixos-19.09/nixexprs.tar.xz") {}).haskell.packages.ghc865.hedgehog.overrideAttrs (_: { preCheck = "ulimit -v 8388608"; })
this derivation produced the following outputs:
doc -> /nix/store/6xshn1jf5pjs1d4q01fyw0rbgcikmcmh-hedgehog-1.0-doc
out -> /nix/store/2spiac2pv2z3l4xlhjp8px5q96wav8jd-hedgehog-1.0
Broken in 20.03:
nix-repl> :b (import (builtins.fetchTarball "https://nixos.org/channels/nixos-20.03/nixexprs.tar.xz") {}).haskell.packages.ghc865.hedgehog.overrideAttrs (_: { preCheck = "ulimit -v 8388608"; })
builder for '/nix/store/z5sn9pxvbasmzk1is39aka55jks2nsnq-hedgehog-1.0.2.drv' failed with exit code 1; last 10 log lines:
━━━ Test.Hedgehog.Applicative ━━━
✓ prop_StateT_outside passed 100 tests.
✓ 1 succeeded.
━━━ Test.Hedgehog.Confidence ━━━
✓ prop_with_confidence passed 600 tests.
number == 1 30% █████▉·············· ✓ 20%
test: failed to create OS thread: Cannot allocate memory
Test suite test: FAIL
Test suite logged to: dist/test/hedgehog-1.0.2-test.log
0 of 1 test suites (0 of 1 test cases) passed.
[0 built (1 failed)]
error: build of '/nix/store/z5sn9pxvbasmzk1is39aka55jks2nsnq-hedgehog-1.0.2.drv' failed
@veprbl I was able to reproduce the error you're seeing:
nix-repl> :b (import (builtins.fetchTarball "https://nixos.org/channels/nixos-20.03/nixexprs.tar.xz") {}).haskell.packages.ghc865.hedgehog.overrideAttrs (_: { preCheck = "ulimit -v 8388608"; })
builder for '/nix/store/z5sn9pxvbasmzk1is39aka55jks2nsnq-hedgehog-1.0.2.drv' failed with exit code 1; last 10 log lines:
[7 of 8] Compiling Test.Hedgehog.Zip ( test/Test/Hedgehog/Zip.hs, dist/build/test/test-tmp/Test/Hedgehog/Zip.o )
[8 of 8] Compiling Main ( test/test.hs, dist/build/test/test-tmp/Main.o )
Linking dist/build/test/test ...
running tests
Running 1 test suites...
Test suite test: RUNNING...
test: failed to create OS thread: Cannot allocate memory
Test suite test: FAIL
Test suite logged to: dist/test/hedgehog-1.0.2-test.log
0 of 1 test suites (0 of 1 test cases) passed.
[0 built (1 failed), 54 copied (1767.3 MiB), 142.5 MiB DL]
error: build of '/nix/store/z5sn9pxvbasmzk1is39aka55jks2nsnq-hedgehog-1.0.2.drv' failed
I have one question and one suggestion for you:
What do other packages do in this scenario? I imagine is it a relatively(?) common case for packages to require a lot of memory either to be built or to run the tests? How is this handled in other places in nixpkgs?
In this instance, it seems like we could just disable the tests for
hedgehog? That would probably solve the problem for you, and make sure you could compile cachix? (You could probably also just do this locally.)Here is how to disable tests:
nix-repl> :b with import (builtins.fetchTarball "https://nixos.org/channels/nixos-20.03/nixexprs.tar.xz") {}; haskell.lib.dontCheck haskell.packages.ghc865.hedgehog
cdepillabout
on 2 May 2020
@cdepillabout Yes, I have compiled cachix successfully by disabling the tests (for 3 packages in total IIRC), and it simply doesn't work:
which cachix | xargs realpath | xargs dirname | xargs dirname | cachix push bla
cachix: failed to create OS threadcachix: failed to create OS thread: Cannot allocate memory
: Cannot allocate memory
cachix: failed to create OS thread: Cannot allocate memory
cachix: failed to create OS thread: Cannot allocate memory
cachix: failed to create OS thread: Cannot allocate memory
zsh: done which cachix | xargs realpath | xargs dirname | xargs dirname |
zsh: segmentation fault cachix push bla
The problem is not with the resident memory used, but with the virtual memory required to run an application.
veprbl
on 2 May 2020
@veprbl Ah, I see the problem.
GHC does something where it allocates like 1TB of virtual memory in order to speed up garbage collection. It looks like you've already found some links talking about it.
Here are a few more:
- https://gitlab.haskell.org/ghc/ghc/issues/14192
- https://downloads.haskell.org/~ghc/8.0.1/docs/html/users_guide/8.0.1-notes.html#runtime-system
- https://gitlab.haskell.org/ghc/ghc/issues/14193
It doesn't look like there is a runtime flag to turn off this behavior, but it is supposed to go away with disableLargeAddressSpace. Does that end up not working for you?
Above, you have the following expression:
hedgehog_ghc865_disableLargeAddressSpace = (haskell.packages.ghc865.override {
overrides = self: super: {
ghc = super.ghc.override { disableLargeAddressSpace = true; };
hedgehog = super.hedgehog.overrideAttrs (_: {
preCheck = "ulimit -v 8388608"; # limit vram to 8gb
});
};
}).hedgehog;
However, I'm not sure something like this should work.
I think you might have to override something like haskell.compiler.ghc865 instead of haskell.packages.ghc865.ghc. If you can't figure this out, let me know and I will take a look at it.
cdepillabout
on 2 May 2020
@cdepillabout
I'm not sure what kind of override you had in mind. I've tried:
with import <nixpkgs> {};
{
hedgehog_ghc865_disableLargeAddressSpace = (haskell.packages.ghc865.override {
overrides = self: super: {
ghc = super.ghc.override { disableLargeAddressSpace = true; };
hedgehog = super.hedgehog.overrideAttrs (_: {
preCheck = "ulimit -v 8388608"; # limit vram to 8gb
});
};
}).hedgehog;
hedgehog_ghc865_disableLargeAddressSpace_alt = (haskell.packages.ghc865.override {
ghc = haskell.compiler.ghc865.override {
disableLargeAddressSpace = true;
};
overrides = self: super: {
hedgehog = super.hedgehog.overrideAttrs (_: {
preCheck = "ulimit -v 8388608"; # limit vram to 8gb
});
};
}).hedgehog;
}
but these two seem to evaluate to the same build:
% nix-instantiate test.nix -A hedgehog_ghc865_disableLargeAddressSpace -I nixpkgs=`pwd`
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/x90k7qfvl3rwvrwx5rji5brjdcpdpgic-hedgehog-1.0.2.drv
% nix-instantiate test.nix -A hedgehog_ghc865_disableLargeAddressSpace_alt -I nixpkgs=`pwd`
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/x90k7qfvl3rwvrwx5rji5brjdcpdpgic-hedgehog-1.0.2.drv
@cdepillabout While browsing your links, I've discovered that it is possible to set RTS options using GHCRTS environment variable. I've tried limiting number of threads used and cachix doesn't crash anymore. I run into some more problems, so can't confirm that it works yet.
veprbl
on 3 May 2020
@veprbl I've created an overlay that causes the hedgehog tests not to fail with ulimit -v 8388608:
let
nixpkgs-src = builtins.fetchTarball {
# nixos-unstable as of 2020-05-03.
url = "https://github.com/NixOS/nixpkgs/archive/ab3adfe1c769c22b6629e59ea0ef88ec8ee4563f.tar.gz";
sha256 = "sha256:1m4wvrrcvif198ssqbdw897c8h84l0cy7q75lyfzdsz9khm1y2n1";
};
my-overlay = self: super: {
haskell = super.haskell // {
compiler = super.haskell.compiler // {
ghc865 = super.haskell.compiler.ghc865.override {
disableLargeAddressSpace = true;
};
};
};
myHaskellPackages = super.haskellPackages.override {
overrides = hask-self: hask-super: {
hedgehog = hask-super.hedgehog.overrideAttrs (_: {
preCheck = "ulimit -v 8388608"; # limit vram to 8gb
});
};
};
};
nixpkgs = import nixpkgs-src { overlays = [ my-overlay ]; };
in
nixpkgs.myHaskellPackages.hedgehog
I had to override haskell.compiler.ghc865 directly.
Note that the following did not work:
let
nixpkgs-src = builtins.fetchTarball {
# nixos-unstable as of 2020-05-03.
url = "https://github.com/NixOS/nixpkgs/archive/ab3adfe1c769c22b6629e59ea0ef88ec8ee4563f.tar.gz";
sha256 = "sha256:1m4wvrrcvif198ssqbdw897c8h84l0cy7q75lyfzdsz9khm1y2n1";
};
my-overlay = self: super: {
myHaskellPackages = super.haskellPackages.override {
overrides = hask-self: hask-super: {
ghc = hask-super.ghc.override {
disableLargeAddressSpace = true;
};
hedgehog = hask-super.hedgehog.overrideAttrs (_: {
preCheck = "ulimit -v 8388608"; # limit vram to 8gb
});
};
};
};
nixpkgs = import nixpkgs-src { overlays = [ my-overlay ]; };
in
nixpkgs.myHaskellPackages.hedgehog
Although I'm not sure why your hedgehog_ghc865_disableLargeAddressSpace_alt from https://github.com/NixOS/nixpkgs/issues/86496#issuecomment-623058124 doesn't work. I had expected it would work.
cdepillabout
on 3 May 2020
@cdepillabout Your override is working, this resolves my issue. Thanks a lot! I think, I overlooked these lines:
https://github.com/NixOS/nixpkgs/blob/ef0f57ff8ace1040fe9bc983bfe384352cbedcec/pkgs/top-level/haskell-packages.nix#L36
https://github.com/NixOS/nixpkgs/blob/694ac1b127ce34fb3bf5890232e1375b3911867c/pkgs/top-level/haskell-packages.nix#L107
Pehaps I also needed to override buildHaskellPackages argument:
https://github.com/NixOS/nixpkgs/blob/ef0f57ff8ace1040fe9bc983bfe384352cbedcec/pkgs/top-level/haskell-packages.nix#L127-L128
and that is why hedgehog_ghc865_disableLargeAddressSpace_alt did't work.
veprbl
on 5 May 2020
@veprbl That's great it is working for you.
Should we close this issue? It seems like this workaround is fine for you, but I guess the underlying issue is still there? Unfortunately, it seems like a decision in GHC upstream to allocate a bunch of virtual memory upon program startup.
cdepillabout
on 6 May 2020
Forgot to close :)
veprbl
on 6 May 2020
Related issues
ayyess
·
3Comments
ob7
·
3Comments
lverns
·
3Comments
matthiasbeyer
·
3Comments
domenkozar
·
3Comments