Nix: builtins.fetchGit doesn't fetch submodules

[ElvishJerricco]

[ElvishJerricco]

This is a pretty big blocker for me. I can't reasonably use builtins.fetchGit without submodule support.

[langston-barrett]

At least when building with nixpkgs, one can use

  src = fetchgit {
    url = "https://github.com/GaloisInc/some-repo.git";
    deepClone = true;
    sha256 = ...;
    rev = ...;
  };

[NorfairKing]

This is also a blocker for me. Specifically when it comes to private git repositories that have submodules.
I looked around in the code to see what would have to be changed to allow for deepClone = true in builtins.fetchGit:
https://github.com/NixOS/nix/blob/18b4c53f71dfc626f5f5ffa0282afd1b9faad6a4/src/libexpr/primops/fetchGit.cc#L125

It looks like the current approach with a bare repository and git fetch instead of git clone will not work because submodules require a working tree and cannot work with a bare repository.
Either we fix up git, or we do a full git clone --recursive and remove the .git directories recursively inside that.

cc @nh2, Thanks for the help!

[tbenst]

This issue causes difficult to debug error messages when trying to package PyToch 1.1 (https://github.com/NixOS/nixpkgs/issues/63073), and probably elsewhere, too. I took a clumsy stab at getting nested level of 2 to work with no luck: https://github.com/NixOS/nixpkgs/compare/master...tbenst:pytorch11#diff-8c857b370607b23d073067f7df717593

[qolii]

Any progress/interest on this one?

[tbenst]

@qolii might want to check out https://github.com/NixOS/nixpkgs/issues/61820#issuecomment-515553846. I'm no longer sure if this issue is valid?

[evanjs]

@qolii might want to check out NixOS/nixpkgs#61820 (comment). I'm no longer sure if this issue is valid?

I think fetchGit is still somewhat preferred if private repos are used. Somebody please correct me if I’m wrong.

[qolii]

I think fetchGit is still somewhat preferred if private repos are used.

Exactly. From my reading, https://github.com/NixOS/nixpkgs/issues/61820 is using pkgs.fetchgit. But that seems to be very difficult to use with private repos (my use case), whereas builtins.fetchGit is much more amenable.

[blitz]

I also ended up here, because I wanted to clone a private repository with submodules.

@tfc

[qolii]

~Looking quickly at fetchgit.cc, this appears like it would be trivial?~

• ~Add another input argument to builtins.fetchGit, along the lines of a boolean fetchSubmodules or so)~
• ~Then based on it, add a conditional extra call to runProgram, to perform a git submodule init.~

~I am sure I am missing something... Please let me know. If it is this easy, I can put a patch together myself.~

Ok, so, I should have actually paid attention to @NorfairKing's comment earlier. Sorry!

Did you make any progress with your git clone --recursive approach?

[NorfairKing]

@qolii I made no attempt to 'fix' this issue. I'm working around it by making my repos public :)

[blitz]

I've given this a shot after a discussion with @edolstra and @tfc at NixCon. Comments welcome.

[knedlsepp]

Finally it's happening. Can't wait for the next release. :heart:

[jakubgs]

Any news on when this will be included in a release?