Nix: segfault with nix repl

Created on 29 Oct 2019  ·  15Comments  ·  Source: NixOS/nix

It happens only sometimes:

nix-repl> (import ./default.nix {}).devTools       
«derivation /nix/store/2b1jssy0z6v3mj0ap8bfd95yqf6z200p-niv-0.0.0.drv»; recurseForDerivations = true; }

nix-repl> (import ./default.nix {}).devTools.arion
querying info about missing paths«derivation /nix/store/8y4djxl5n792rdxyz22jmhwzznqsll4w-arion-0.1.0.0.drv»

nix-repl> (import ./default.nix {}).devTools       
«derivation /nix/store/2b1jssy0z6v3mj0ap8bfd95yqf6z200p-niv-0.0.0.drv»; recurseForDerivations = true; }

nix-repl> (import ./default.nix {}).devTools.arion
querying info about missing paths«derivation /nix/store/8y4djxl5n792rdxyz22jmhwzznqsll4w-arion-0.1.0.0.drv»

nix-repl> (import ./default.nix {}).devTools       
Segmentation fault (core dumped)

bt on coredump

(gdb) bt
#0  0x00007fd4e14f59e4 in nix::ExprVar::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/bnrk3fsrwxfalix265sx4y3r2909dc6m-nix-2.3/lib/libnixexpr.so
#1  0x00007fd4e14f74f8 in nix::ExprApp::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/bnrk3fsrwxfalix265sx4y3r2909dc6m-nix-2.3/lib/libnixexpr.so
#2  0x00007fd4e14f74f8 in nix::ExprApp::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/bnrk3fsrwxfalix265sx4y3r2909dc6m-nix-2.3/lib/libnixexpr.so
#3  0x00007fd4e14f706e in nix::ExprSelect::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/bnrk3fsrwxfalix265sx4y3r2909dc6m-nix-2.3/lib/libnixexpr.so
#4  0x00000000004e3fdb in nix::NixRepl::evalString(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, nix::Value&) ()
#5  0x00000000004e6d20 in nix::NixRepl::processLine(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) ()
#6  0x00000000004e72a2 in nix::NixRepl::mainLoop(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) ()
#7  0x00000000004e8c1b in nix::CmdRepl::run(nix::ref<nix::Store>) ()
#8  0x00000000004b6b09 in nix::StoreCommand::run() ()
#9  0x00000000004d4b18 in nix::mainWrapped(int, char**) ()
#10 0x00007fd4e11ef09c in nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) ()
   from /nix/store/bnrk3fsrwxfalix265sx4y3r2909dc6m-nix-2.3/lib/libnixmain.so
#11 0x0000000000441c9b in main ()
bug
Source
picture domenkozar

Most helpful comment

@puckipedia Thanks, that's exactly the issue. I've pushed a fix.

picture edolstra on 19 Mar 2020
🎉7

All 15 comments

@domenkozar can you trigger it with disabled http2? (might be a dup of #2733)

flokli picture flokli on 22 Nov 2019

I have a similar issue; but with a completely different backtrace...

(gdb) bt
#0  0x00007ff249635be0 in raise () from /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libc.so.6
#1  0x00007ff249636dc1 in abort () from /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libc.so.6
#2  0x00007ff24a2f24a3 in nix::showType[abi:cxx11](nix::Value const&) [clone .cold.794] ()
   from /nix/store/dn73nrnc7sk44d5r1wk1lav9jm1djsyw-nix-2.3.2/lib/libnixexpr.so
#3  0x00007ff24a318508 in nix::throwTypeError(char const*, nix::Value const&, nix::Pos const&) ()
   from /nix/store/dn73nrnc7sk44d5r1wk1lav9jm1djsyw-nix-2.3.2/lib/libnixexpr.so
#4  0x00007ff24a31c2db in nix::ExprSelect::eval(nix::EvalState&, nix::Env&, nix::Value&) ()
   from /nix/store/dn73nrnc7sk44d5r1wk1lav9jm1djsyw-nix-2.3.2/lib/libnixexpr.so
#5  0x00000000004e5b20 in nix::NixRepl::completePrefix(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) ()
#6  0x00000000004e61a9 in nix::completionCallback(char*, int*) ()
#7  0x00007ff24a4214b9 in c_complete () from /nix/store/874b5h0w4cmaddlw7s35y5x0lcc78rqw-editline-1.16.1/lib/libeditline.so.1
#8  0x00007ff24a420692 in editinput.part ()
   from /nix/store/874b5h0w4cmaddlw7s35y5x0lcc78rqw-editline-1.16.1/lib/libeditline.so.1
#9  0x00007ff24a420f97 in readline () from /nix/store/874b5h0w4cmaddlw7s35y5x0lcc78rqw-editline-1.16.1/lib/libeditline.so.1
#10 0x00000000004e42da in nix::NixRepl::getLine(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#11 0x00000000004e7b96 in nix::NixRepl::mainLoop(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) ()
#12 0x00000000004e962b in nix::CmdRepl::run(nix::ref<nix::Store>) ()
#13 0x00000000004b7409 in nix::StoreCommand::run() ()
#14 0x00000000004d5768 in nix::mainWrapped(int, char**) ()
#15 0x00007ff24a01401c in nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) () from /nix/store/dn73nrnc7sk44d5r1wk1lav9jm1djsyw-nix-2.3.2/lib/libnixmain.so
#16 0x0000000000441f6b in main ()
arianvp picture arianvp on 20 Jan 2020

I have some more input that reproduces this.

:l <nixpkgs>
(nixos {}).config.system
lib
(nixos {}).config.system.stateVersion

Or, for short, printf ":l <nixpkgs>\n(nixos {}).config.system\nlib\n(nixos {}).config.system.stateVersion\n" | nix repl. Seems to segfault pretty consistently on my machine and a friend's machine.

Backtrace:

(gdb) bt
#0  0x000000000049e33b in nix::EvalState::forceValue(nix::Value&, nix::Pos const&) ()
#1  0x00007f4356da0744 in nix::ExprVar::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/cs47wjxwiqgyl1nkjnksyf3s2rb93piq-nix-2.3.2/lib/libnixexpr.so
#2  0x00000000004e49eb in nix::NixRepl::evalString(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, nix::Value&) ()
#3  0x00000000004e7730 in nix::NixRepl::processLine(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) ()
#4  0x00000000004e7cb2 in nix::NixRepl::mainLoop(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) ()
#5  0x00000000004e962b in nix::CmdRepl::run(nix::ref<nix::Store>) ()
#6  0x00000000004b7409 in nix::StoreCommand::run() ()
#7  0x00000000004d5768 in nix::mainWrapped(int, char**) ()
#8  0x00007f4356a9a01c in nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) ()
   from /nix/store/cs47wjxwiqgyl1nkjnksyf3s2rb93piq-nix-2.3.2/lib/libnixmain.so
#9  0x0000000000441f6b in main ()
  • system: "x86_64-linux"
  • host os: Linux 5.4.16, NixOS, 19.09.1993.d3d2de8b99b (Loris)
  • multi-user?: yes
  • sandbox: yes
  • version: nix-env (Nix) 2.3.2
  • channels(root): "nixos-19.09.1993.d3d2de8b99b"
  • channels(becca): "nixpkgs-20.03pre209270.8da81465c19"
  • nixpkgs: /home/becca/.nix-defexpr/channels/nixpkgs
  • nix binary: /nix/store/cs47wjxwiqgyl1nkjnksyf3s2rb93piq-nix-2.3.2/bin/nix
9999years picture 9999years on 4 Feb 2020

These all seem like completely different backtrackes to me. I wonder if they're related at all or whether there is more than one issue with nix repl

arianvp picture arianvp on 5 Feb 2020

I suspect that at least @domenkozar's issue and mine are related -- the fact that they both are caused by accessing attributes in various orders and are somewhat unpredictable (the example I gave sometimes segfaults on lib and sometimes segfaults on the last line) made it seem like they had something in common.

It's certainly possible that they're an unrelated collection of segfaults, though, not that that's any less concerning.

9999years picture 9999years on 5 Feb 2020

Ok interesting. I run into this daily now and can reproduce reliably in an internal repo using nix module system.

Really strange things happen before the segfault


nix-repl> :l . 
Added 6 variables.

nix-repl> lol
{ config = { ... }; options = { ... }; }


nix-repl> lol.config.hsPkgs.galley.components.exes.galley
trace: [alex-plan-to-nix-pkgs] cabal new-configure --with-ghc=ghc --with-ghc-pkg=ghc-pkg
trace: [happy-plan-to-nix-pkgs] cabal new-configure --with-ghc=ghc --with-ghc-pkg=ghc-pkg
trace: [hscolour-plan-to-nix-pkgs] cabal new-configure --with-ghc=ghc --with-ghc-pkg=ghc-pkg
trace: Cleaning component source not supported for hpack package: imports-0.1.0
trace: Cleaning component source not supported for hpack package: cassandra-util-0.16.5
trace: Cleaning component source not supported for hpack package: types-common-0.16.0
trace: Cleaning component source not supported for hpack package: gundeck-types-1.45.0
trace: Cleaning component source not supported for hpack package: galley-types-0.81.0
trace: Cleaning component source not supported for hpack package: metrics-core-0.3.2
trace: Cleaning component source not supported for hpack package: metrics-wai-0.5.7
trace: Cleaning component source not supported for hpack package: sodium-crypto-sign-0.1.2
trace: Cleaning component source not supported for hpack package: bilge-0.22.0
trace: Cleaning component source not supported for hpack package: brig-types-1.35.0
trace: Cleaning component source not supported for hpack package: extended-0.1.0
trace: Cleaning component source not supported for hpack package: ssl-util-0.1.0
trace: Cleaning component source not supported for hpack package: types-common-journal-0.1.0
trace: Cleaning component source not supported for hpack package: wai-utilities-0.16.1
trace: Cleaning component source not supported for hpack package: zauth-0.10.3
trace: Cleaning component source not supported for hpack package: galley-0.83.0
warning: dumping very large path (> 256 MiB); this may run out of memory
trace: Cleaning component source not supported for hpack package: galley-0.83.0
«derivation /nix/store/p3496ryby2crnzk3scjfj40fgpxfgyqk-galley-0.83.0-exe-galley.drv»

nix-repl> lol
«unknown»

nix-repl> lol.Aborted (core dumped)

first time lol will be a a attrset generated by the moduleSystem. I then try to evaluate a component that is a derivation.

Then the second time lol will be an arbitrary nixpkgs expression. (e.g. it self-corrupts). Sometimes htis is an <>, but sometimes this is a random derivation!. If it's an unknown, the whole hting Segfaults the third time it evaluates. If it's not. everything is fine, but lol now refers to a different value than it originally did

nix-repl> :l .                                            
Added 6 variables.

nix-repl> lol
{ config = { ... }; options = { ... }; }

nix-repl> lol.config.hsPkgs.galley.components.exes.galley 
trace: [alex-plan-to-nix-pkgs] cabal new-configure --with-ghc=ghc --with-ghc-pkg=ghc-pkg
trace: [happy-plan-to-nix-pkgs] cabal new-configure --with-ghc=ghc --with-ghc-pkg=ghc-pkg
trace: [hscolour-plan-to-nix-pkgs] cabal new-configure --with-ghc=ghc --with-ghc-pkg=ghc-pkg
trace: Cleaning component source not supported for hpack package: imports-0.1.0
trace: Cleaning component source not supported for hpack package: cassandra-util-0.16.5
trace: Cleaning component source not supported for hpack package: types-common-0.16.0
trace: Cleaning component source not supported for hpack package: gundeck-types-1.45.0
trace: Cleaning component source not supported for hpack package: galley-types-0.81.0
trace: Cleaning component source not supported for hpack package: metrics-core-0.3.2
trace: Cleaning component source not supported for hpack package: metrics-wai-0.5.7
trace: Cleaning component source not supported for hpack package: sodium-crypto-sign-0.1.2
trace: Cleaning component source not supported for hpack package: bilge-0.22.0
trace: Cleaning component source not supported for hpack package: brig-types-1.35.0
trace: Cleaning component source not supported for hpack package: extended-0.1.0
trace: Cleaning component source not supported for hpack package: ssl-util-0.1.0
trace: Cleaning component source not supported for hpack package: types-common-journal-0.1.0
trace: Cleaning component source not supported for hpack package: wai-utilities-0.16.1
trace: Cleaning component source not supported for hpack package: zauth-0.10.3
trace: Cleaning component source not supported for hpack package: galley-0.83.0
warning: dumping very large path (> 256 MiB); this may run out of memory
trace: Cleaning component source not supported for hpack package: galley-0.83.0
«derivation /nix/store/p3496ryby2crnzk3scjfj40fgpxfgyqk-galley-0.83.0-exe-galley.drv»

nix-repl> lol
{ revisions = { ... }; sha256 = "1ae101fbab953817a8595999ee64a38ef366529f25497c224fa70f9a6b49a1ac"; }
# ^ lol was a totall different attrset before!!!
arianvp picture arianvp on 18 Feb 2020

HEre is another run. lol now got self-replaced to a string

nix-repl> lol  
{ config = { ... }; options = { ... }; }

nix-repl> lol.config.hsPkgs.galley.components.exes.galley 
trace: [alex-plan-to-nix-pkgs] cabal new-configure --with-ghc=ghc --with-ghc-pkg=ghc-pkg
trace: [happy-plan-to-nix-pkgs] cabal new-configure --with-ghc=ghc --with-ghc-pkg=ghc-pkg
trace: [hscolour-plan-to-nix-pkgs] cabal new-configure --with-ghc=ghc --with-ghc-pkg=ghc-pkg
trace: Cleaning component source not supported for hpack package: imports-0.1.0
trace: Cleaning component source not supported for hpack package: cassandra-util-0.16.5
trace: Cleaning component source not supported for hpack package: types-common-0.16.0
trace: Cleaning component source not supported for hpack package: gundeck-types-1.45.0
trace: Cleaning component source not supported for hpack package: galley-types-0.81.0
trace: Cleaning component source not supported for hpack package: metrics-core-0.3.2
trace: Cleaning component source not supported for hpack package: metrics-wai-0.5.7
trace: Cleaning component source not supported for hpack package: sodium-crypto-sign-0.1.2
trace: Cleaning component source not supported for hpack package: bilge-0.22.0
trace: Cleaning component source not supported for hpack package: brig-types-1.35.0
trace: Cleaning component source not supported for hpack package: extended-0.1.0
trace: Cleaning component source not supported for hpack package: ssl-util-0.1.0
trace: Cleaning component source not supported for hpack package: types-common-journal-0.1.0
trace: Cleaning component source not supported for hpack package: wai-utilities-0.16.1
trace: Cleaning component source not supported for hpack package: zauth-0.10.3
trace: Cleaning component source not supported for hpack package: galley-0.83.0
warning: dumping very large path (> 256 MiB); this may run out of memory
trace: Cleaning component source not supported for hpack package: galley-0.83.0
«derivation /nix/store/p3496ryby2crnzk3scjfj40fgpxfgyqk-galley-0.83.0-exe-galley.drv»

nix-repl> lol
"nullOr"

I'll see if I can get a minimal reproducer. At least I can reliably trigger the issue now (Though it doesn't always segfault; but nix repl is defenitely returning random chunks of memory and sometimes gets unlucky)

There seems to be some serious memory corruption when nix-repl evaluates nixos modules

arianvp picture arianvp on 18 Feb 2020

I suspect I found the issue: The Nix Command structs are stored on the heap (using ref<T>, a wrapper around std::shared_ptr<T>), which means that any pointers that the NixRepl struct contains are eligible to be reaped by the GC. This includes, but is not limited to, the Env pointer, which seems to cause most of the random segfaults, or random other values in the environment, which seems to be what @arianvp experienced too.

puckipedia picture puckipedia on 19 Mar 2020
👀7

@puckipedia Thanks, that's exactly the issue. I've pushed a fix.

edolstra picture edolstra on 19 Mar 2020
🎉7

Is this in any release yet? On 2.3.6 I still get dialy segfaults

arianvp picture arianvp on 18 Jun 2020

Crap, we forgot to backport this.

domenkozar picture domenkozar on 18 Jun 2020

Done.

domenkozar picture domenkozar on 18 Jun 2020

@domenkozar is there a PR/commit for nixpkgs already?

flokli picture flokli on 18 Jun 2020

First Eelco will need to release 2.3.7

domenkozar picture domenkozar on 18 Jun 2020

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-repl-segmentation-fault/4424/3

nixos-discourse picture nixos-discourse on 9 Aug 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

chexxor picture chexxor  ·  4Comments
vaibhavsagar picture vaibhavsagar  ·  3Comments
dasJ picture dasJ  ·  3Comments
ericsagnes picture ericsagnes  ·  4Comments
bryanhuntesl picture bryanhuntesl  ·  3Comments