Nix: Shared fetchGit git repo is unusably slow

[shlevy]

Sharing repos like fetchGit currently does is extremely slow on first fetch when you have multiple repos, as git tries to find common root commits, fails, and only then starts actually fetching. The likelihood of significant savings from shared objects seems low IMO, but if we really want to there is a better way https://git.wiki.kernel.org/index.php/Git_FAQ#How_to_share_objects_between_existing_repositories.3F

[dtzWill]

Alternates are useful, but not sure they'd be any better?

TIL there's also an environment variable you can set which might be convenient:

https://git-scm.com/docs/gitrepository-layout

I imagine the best benefit will be when hitting the same repo repeatedly, especially over time. Or branches of a repo.

Maybe we can indicate to git that it shouldn't try "too hard" to find common ancestors, limiting it from massive walks? Idea is that when savings are large it won't have to look far. Dunno. :).

[shlevy]

My point is we shouldn't share repos so completely. We can have, say, a different cache dir for each repo basename we see.

[edolstra]

Well, we don't want (say) github.com/NixOS/nixpkgs and github.com/NixOS/nixpkgs-channels to require twice the disk space, especially since a clone of Nixpkgs now requires ~900 MB. This would be less of an issue with shallow clones, though.

[ryantrinkle]

This issue definitely has a big impact in my case - builtins.fetchGit takes almost 8 minutes, whereas a clone with git on the command line takes 5 seconds, for this:

nix-build -E 'builtins.fetchGit { url = "ssh://[email protected]/reflex-frp/reflex"; rev = "5d9c8a00f2eb832f109c870963182149b988062a"; ref = "develop"; }'

Note: both measurements were taken on a mediocre wifi connection in Australia, where my roundtrip time to github was 200 ms.

[ryantrinkle]

Note: deleting ~/.cache/nix/git results in the nix-build taking 5 seconds, just like a fresh git clone.

[shlevy]

@ryantrinkle Did you see https://github.com/target/nix-fetchers/tree/master/fetch-git#fetch-git ? If we get more users there and have confidence we've smoothed the rough edges and can demonstrate real improvement, we can try upstreaming it.

[3noch]

@shlevy Does that support private repos?

[ElvishJerricco]

Yea I've been bitten by this too. I think we're making the wrong trade here. Unless we have a way to get the best of both worlds, increasing clone time 100x as in @ryantrinkle's case is unacceptable, even if it halves the disk space for some cases.

[shlevy]

@3noch yep!

[ElvishJerricco]

@3noch Note, seems to require the allow-unsafe-native-code-during-evaluation setting. Though if it were upstreamed I imagine this would no longer be the case.

[shlevy]

@ElvishJerricco @3noch we're actually using it with nix-plugins + extra-builtins https://github.com/target/nix-fetchers#prerequisites https://github.com/shlevy/nix-plugins#extra-builtins

[edolstra]

Upstreamed in Nix? I don't see how that can happen given that it depends on Python.

[ElvishJerricco]

@edolstra Didn't mean that exact code; just a feature like it.

[erikarvstedt]

This issue has been fixed in https://github.com/NixOS/nix/commit/3f4de91d80da01f5612d839e610b113b8d159065.