Nixpkgs: stack overflow (possible infinite recursion) depending on type of git checkout

[erictapen]

Issue description

I'm able to provoke an

error: stack overflow (possible infinite recursion)

based on wether I checked out my branch or checked out the current commit of that branch in my Nixpkgs fork. The Nix expressions do not change, but the error is magically very stable.

Steps to reproduce

My setup is, that I manage my patches ontop of nixos-18.03 in a branch called custom-18.03. Git staging area is empty. As this is my working system, it is sort of complex, but not fancy enough to explain any behaviour like this one. The patches I maintain are basically version bumps and some backports of packages only available on nixos-unstable.

I can reliably cycle (like 10 times in a row) between failure and success with this sequence:

[justin@maschine:~/nixpkgs]$ git checkout custom-18.03
Already on 'custom-18.03'
Your branch is up to date with 'ele/custom-18.03'.

[justin@maschine:~/nixpkgs]$ nixos-rebuild build -I nixos-config=/home/justin/git/infra/maschine/configuration.nix -I nixpkgs=/home/justin/nixpkgs/ --show-trace
building Nix...
building the system configuration...
error: stack overflow (possible infinite recursion)

[justin@maschine:~/nixpkgs]$ git checkout $(git rev-parse HEAD)
Note: checking out '0939751d7ee721ed9fd1e6fd4ae45abb1841f82b'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 0939751d7ee youtube-dl: use toPythonApplication (#40831)

[justin@maschine:~/nixpkgs]$ nixos-rebuild build -I nixos-config=/home/justin/git/infra/maschine/configuration.nix -I nixpkgs=/home/justin/nixpkgs/ --show-trace
building Nix...
building the system configuration...

[justin@maschine:~/nixpkgs]$
... start from the beginning

As you can see, my system evaluates without error, as long I don't checkout my branch, but its commit.

I understand that this is not a good starting point for debugging and I would be glad to provide an instruction being better reproducible. But as this error appears to be absolutely magically to me, I have no idea where to start symplifying the error. I do not even know, wether this is Nix or Nixpkgs related.

Would be glad if anyone could provide me with a hint where to start.

Technical details

• system: "x86_64-linux"
• host os: Linux 4.14.51, NixOS, 18.03 (Impala)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.0.4
• nixpkgs: /home/justin/nixpkgs

[jtojnar]

Hmm, I also occasionally encountered this but did not connect it with git.

Maybe it has something to do with how nixos-version is generated:

https://github.com/NixOS/nixpkgs/blob/5ac6ab091a4883385e68571425fb7fef4d74c207/nixos/modules/installer/tools/nixos-rebuild.sh#L314-L323

https://github.com/NixOS/nixpkgs/blob/5ac6ab091a4883385e68571425fb7fef4d74c207/nixos/modules/installer/tools/get-version-suffix#L17

[erictapen]

Thanks for the hint. After rebasing my custom-18.03 branch ontop of the newest nixos-18.03, the error disappeared, so it seems to be even more fragile. I'd consider this bug too difficult to track down. Maybe someone wants to reopen this sometimes in the future.

[jerith666]

I've run into this on occasion as well. Today I figured out that mv -iv .git .gitout makes the rebuild work; then mv -iv .gitout .git to put it back after. No real insight on why it sometimes fails when it's a git repo though. :-/

[jerith666]

using nix-build -v I found that the last nix file it loads is lib/sources.nix. using strace, I found that the last file of any kind that it loads is .git/packed-refs. and using gdb I found that the overflowed stack is chock full of a bunch of regex function calls.

This leads me to believe that the problem occurs when commitIdFromGitRepo in sources.nix ends up in the clause described by this comment:

           # Sometimes, the file isn't there at all and has been packed away in the
           # packed-refs file, so we have to grep through it:

... which is running match (".*\n([^\n ]*) " + file + "\n.*") on the .git/packed-refs file. In my current repo, that file is 2387 lines long (since I have several forks of nixpkgs as remotes).

This would also explain why the build works fine in a git worktree, since there .git is a file not a directory, which commitIdFromGitRepo clearly can't handle.

[jtojnar]

Yeah, that looks like https://github.com/NixOS/nix/issues/2147

[jerith666]

Yes, thank you! The ulimit -s 100000 work-around given there lets nixos-rebuild work again!

[jb55]

I've run into this on occasion as well. Today I figured out that mv -iv .git .gitout makes the rebuild work; then mv -iv .gitout .git to put it back after. No real insight on _why_ it sometimes fails when it's a git repo though. :-/

wow I just hit this after adding a bunch of remotes and typing git remote update. It looks like it happens if there are too many refs in nixpkgs/.git/packed-refs? This is a very weird bug.

[jb55]

here is my packed-refs file if anyone wants to reproduce: https://jb55.com/s/7669361c5a10dc31.txt

[jtojnar]

@jb55 We are pretty certain this is a duplicate of https://github.com/NixOS/nix/issues/2147, see Matt’s analysis above.

[jb55]

correct, just wanted to clarify for sure where it was happening for other people googling the problem. Setting ulimit didn't seem to work. Moving .git seemed to work for now, but I guess I will have to remove my remotes so I don't keep hitting it until its fixed.

[jerith666]

Maybe you just need a larger limit? If you can't find a larger ulimit -s value that works, that makes it seem like maybe this isn't always due to https://github.com/NixOS/nix/issues/2147, which would be important to learn.

[jb55]

larger ulimits don't seem to work... and I tried some pretty large numbers. perhaps I've reached a max value or something.

[erictapen]

One and a half years ago I decided to stop messing with this issue and rebased this commit ontop of nixos-xx.xx since then:

diff --git a/lib/sources.nix b/lib/sources.nix
index 51bcf5559e3..9fd09f33139 100644
--- a/lib/sources.nix
+++ b/lib/sources.nix
@@ -105,31 +105,31 @@ rec {

   # Get the commit id of a git repo
   # Example: commitIdFromGitRepo <nixpkgs/.git>
-  commitIdFromGitRepo =
-    let readCommitFromFile = file: path:
-      with builtins;
-        let fileName       = toString path + "/" + file;
-            packedRefsName = toString path + "/packed-refs";
-        in if lib.pathExists fileName
-           then
-             let fileContent = lib.fileContents fileName;
-                 # Sometimes git stores the commitId directly in the file but
-                 # sometimes it stores something like: «ref: refs/heads/branch-name»
-                 matchRef    = match "^ref: (.*)$" fileContent;
-             in if   matchRef == null
-                then fileContent
-                else readCommitFromFile (lib.head matchRef) path
-           # Sometimes, the file isn't there at all and has been packed away in the
-           # packed-refs file, so we have to grep through it:
-           else if lib.pathExists packedRefsName
-           then
-             let fileContent = readFile packedRefsName;
-                 matchRef    = match (".*\n([^\n ]*) " + file + "\n.*") fileContent;
-             in if   matchRef == null
-                then throw ("Could not find " + file + " in " + packedRefsName)
-                else lib.head matchRef
-           else throw ("Not a .git directory: " + path);
-    in readCommitFromFile "HEAD";
+  commitIdFromGitRepo = x: "notarealversion";
+    # let readCommitFromFile = path: file:
+    #   with builtins;
+    #     let fileName       = toString path + "/" + file;
+    #         packedRefsName = toString path + "/packed-refs";
+    #     in if lib.pathExists fileName
+    #        then
+    #          let fileContent = lib.fileContents fileName;
+    #              # Sometimes git stores the commitId directly in the file but
+    #              # sometimes it stores something like: «ref: refs/heads/branch-name»
+    #              matchRef    = match "^ref: (.*)$" fileContent;
+    #          in if   isNull matchRef
+    #             then fileContent
+    #             else readCommitFromFile path (lib.head matchRef)
+    #        # Sometimes, the file isn't there at all and has been packed away in the
+    #        # packed-refs file, so we have to grep through it:
+    #        else if lib.pathExists packedRefsName
+    #        then
+    #          let fileContent = readFile packedRefsName;
+    #              matchRef    = match (".*\n([^\n ]*) " + file + "\n.*") fileContent;
+    #          in if   isNull matchRef
+    #             then throw ("Could not find " + file + " in " + packedRefsName)
+    #             else lib.head matchRef
+    #        else throw ("Not a .git directory: " + path);
+    # in lib.flip readCommitFromFile "HEAD";

   pathHasContext = builtins.hasContext or (lib.hasPrefix builtins.storeDir);

As I recently started building my systems using the experimental flakes feature, I don't see this error anymore. So maybe we can have peace from this pesky failure mode someday?

Edit: Rereading this I'd like to clarify, that I don't want to sound demanding here. Just wanted to note that with dropping impure evaluation in flakes, we won't have this instance of https://github.com/NixOS/nix/issues/2147 anymore.

[lheckemann]

I'll reopen this, even though it's technically an issue in nix, since it's still a problem and the closed status suggests (e.g. in issue searches) that it's not, i.e. for discoverability.

If anyone disagrees, do say so :)

[lheckemann]

Current workarounds:

• Patch commitIdFromGitRepo functionality out completely, like @erictapen does
• Reduce the number of refs in your nixpkgs checkout, e.g. by removing no-longer-needed remotes, running git remote prune $(git remote) and deleting no-longer-needed local branches

[lheckemann]

Worked around in #93337.