Cargo: Allow specifying shallow clones in .config

[Manishearth]

Servo has a dep on Skia, which takes a _very_ long time to download. Aside from appearing to hang (since Cargo doesn't show a progress message), this is a rather unnecessary delay for getting a new build running.

It would be nice if we could specify a --depth argument in the .config. (most of the size is due to it's extensive commit history)

Alternatively, a default --depth 50 would work too, though I'm not sure how well that would interact with cargo update.

Status

Blocked on https://github.com/libgit2/libgit2/issues/3058

[alexcrichton]

Note that this is blocked on https://github.com/libgit2/libgit2/issues/2782.

[alexcrichton]

And to clarify some more, I would love to implement this! The backing libgit2 support is a prerequisite, however (see https://github.com/libgit2/libgit2/issues/1430 as well)

[larsbergstrom]

Both of those blocked issues in libgit are now fixed :-)

[alexcrichton]

Ah unfortunately they were closed in favor of a new metabug in libgit2: https://github.com/libgit2/libgit2/issues/3058 I'll update the description accordingly

[larsbergstrom]

Darn, got my hopes up :-)

[larsbergstrom]

This issue is less important to Servo now that crates.io supports larger crate sizes and we are able to shift more of our dependency graph from raw git to crates.io.

[cuviper]

This would be nice for crates.io-index too, not just git dependencies. A current full clone of the index takes 39MB network and 48MB .git/ on disk, whereas --depth 1 is only 4.5MB network and 6MB on disk.

[alexcrichton]

@cuviper oh it's worth pointing out that for the index at least we planned on this happening over time (clone being large) and we wanted to protect against that. It should be implemented that at any time we can roll the entire index into one commit (e.g. one huge squash) and force push that up. That, in effect, should reset everyone to a "shallow clone".

We've never done that yet so we don't quite have a procedure and/or threshold to do that, but I should probably verify that we can actually do that soon :). Other than that all we'd need to do is briefly turn off crates.io, do the squash, force push, and then turn crates.io back. All instances of Cargo should then pick up the new commit and hard reset to it when they next need to update the registry.

[cuviper]

Hmm, will the history be saved to a branch before the reset? I'm not sure it's important, but it seems like we shouldn't just throw it all away. And FWIW git has a --single-branch option (implied by --depth) which we would want to use to get only the squashed master. It seems libgit2 can also do this by adjusting the remote refspec.

[alexcrichton]

Ah excellent point, I think we'd definitely want a backup somewhere. I don't personally know a huge amount about git refspecs, but my gut is that Cargo is cloning all branches of the index today (and all historical versions of Cargo). I think it'd be easy to update that though to only fetch master!

[cuviper]

Yeah, the refspec just becomes "refs/heads/master:refs/remotes/origin/master".

[alexcrichton]

I've filed an issue to update that

[cztomsik]

Hey, any changes to this? I'm depending on my own forks of glfw & yoga which I'd like to rather not publish to crates.io yet - fetching it takes considerable time.

[Kerollmops]

I think the progress of this issue is completely related to https://github.com/libgit2/libgit2/pull/5254, because IIRC cargo use the git2 rust library and it has recently made progress on shallow clones.