Go: cmd/go: use shallow clones for new git checkouts

[mengzhuo]

Currently go get * will download all changes data of repositories.
We can improve the download waiting time/size by passing depth argument.

I have already submit a code-review for this at
https://go-review.googlesource.com/#/c/16360/
as @bradfitz asked.

[rakyll]

Some users depend on go get to clone the repos. How is this going to affect them?

[garukun]

+1 for the CL. Maybe add go clone to support @rakyll's case?

[ALTree]

A suggestion: some data about the magnitude of the improvement in the size of the downloaded repository could probably help the reviewers.

[bradfitz]

It's already the case that "go get" won't always fetch the full history in some cases (subversion), if that precedent matters.

I'm also in favor or switching to shallow clones. Personal motivation: Camlistore's Git repo is huge, but mostly because we used to have some huge assets and other libraries in it, since gone. But that means for a user just wanting a few files, they have to download megabytes.

People who want history can go into the git directories and run the git commands.

Or we can add a new -full or -deep flag to go get.

[bradfitz]

/cc @adg @rsc

[fsouza]

+1 for the feature and for @bradfitz's -full flag, although I think users can always unshallow their repositories manually running git fetch --unshallow.

[adg]

I do think it would be worth switching to shallow clones.

I don't see much value in adding a flag to the go tool.

[fsouza]

Maybe having an environment variable for the git command (as well as other vcs commands) would do the trick and make it more flexible, if that's the case. Just like git allows customizing the ssh command with the GIT_SSH_COMMAND variable. Probably off-topic, though.

[davecheney]

I think that these requests are outside the remit of go get.

We all know go get is just a wrapper around git clone. As time goes on, a few little tweaks have been added to the clone model, and the result is more requests for more tweaks, more flags, etc.

This is natural and expected, go get is a simplified wrapper around a more complicated command so it is a logical conclusion that go get itself would have to become complex in turn.

Rather than this outcome, I suggest that we document the limitations of go get and recommend to people who need more that they simply skip go get, and use git clone; giving them the full power of the tool for their specific scenario without having to further complicate the simple model that go get provides.

[mengzhuo]

@davecheney
I agree that we should think about what "go get" for at the beginning.
IMHO "get" means "get the right source code" for build/compile instead of "get the whole repository for investigation/study" and git --depth arguments making "go get" do the right thing.

[rakyll]

For the sake of clarity...

Even though, it is not the core job of the go tool, I raised to issue of being able to deep copy because of the canonical URLs.

I simply want to be able to run

go get -d golang.org/x/tools/...

rather than figuring out what exact repo it is pointing to. But I simply can run git fetch --unshallow once it is cloned shallowly.

[omeid]

+1 shallow clone.

I just tried to do go get github.com/openshift/origin/cmd/oc and after a very long wait I ended up selectively shallow cloning docker/docker, kubernetes/kubernetes, openshift/origin and couple of other dependencies.

[mengzhuo]

@davecheney
@adg

The title is miss leading.
It should be:
cmd/go: improve git pull download size and speed (by adding --depth arguments into git VCS command)
It doesn't add any params into "go get" cmd
Sorry for my poor English (sigh)

[hyper0x]

+1 I agree with what @bradfitz said.

[mattn]

+1 to -deep

[fsouza]

Just to make it clear: I'm -1 to add a new flag to go get, I think it should _always_ do shallow clones, and if the user wants the full repository, he/she can cd into the repository directory and unshallow it.

[bradfitz]

I'm also -1 on a new flag. I probably shouldn't have even mentioned it.

I'll just make my https://github.com/bradfitz/gitutil/tree/master/git-allgoupdate code unshallow everything it finds in GOPATH instead.

I think we're all in agreement that we should go with shallow clones. The difference in size between shallow and deep for some of the repos is quite large.

[mattn]

I'm thinking that is good to provide flag or environment variable to enable or disable full/shallow because CI server hope to get shallow.

[rsc]

No flags, no environment variables, none of that complexity or configuration. The go command should do the right thing. People who want something else can type git commands. It sounds like there is general agreement that the right thing is shallow clones.

[mengzhuo]

@rsc @bradfitz I had fixed patch to depth=1

[bradfitz]

@mengzhuo, did you forget to upload it? I don't see it at https://go-review.googlesource.com/#/c/16360/

[mengzhuo]

@bradfitz Sorry, I'm new to Gerrit. I had clicked "fix" on Russ's comment and changed into 1.

[mengzhuo]

@bradfitz First of all, I'm sorry for the trouble.
After Russ reviewed, I found that none of us ( including testcases) missed "go get -u" which will raise an git error "missing option --ff-only".

It appears to be git's behavior that "--ff-only" trigger merge however "--depth" trigger fetch.
To fix this issue, we can delete --depth option in downloadCmd
https://go-review.googlesource.com/#/c/16360/5/src/cmd/go/vcs.go

My git version: 1.9

[mengzhuo]

@bradfitz new commit https://go-review.googlesource.com/16798

[bradfitz]

This is being reverted in https://go-review.googlesource.com/#/c/16832/ so re-opening this bug.

This will need better tests to re-introduce.

[mengzhuo]

Yes, This is my fault.
As Russ's comment, we should ONLY use it in clone.

---

Light up the darkness

2015-11-12 16:35 GMT+08:00 Brad Fitzpatrick [email protected]:

This is being reverted in https://go-review.googlesource.com/#/c/16832/
so re-opening this bug.

This will need better tests to re-introduce.

—
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/13078#issuecomment-156033583.

[rsc]

Complication: some servers do not support --depth=1.
Possible complication: do any servers appear to support it but not support it correctly? I've seen notes to that effect on StackOverflow but they were in fairly old comment threads.

[rsc]

It's now too late to do this in Go 1.6. It would need to be reintroduced in a non-freeze time, such as early in Go 1.7. That said, please consider whether doing a shallow clone fetches all the necessary tags for the go command to do its tag-based checkout selection. If a shallow clone does not fetch all the necessary tags, then a separate tag fetch would be needed, and if that's not possible then we can't do shallow clones at all.

[kokizzu]

+1 for -depth 1, sometimes go getting some repository takes too long for country with slow internet (like mine: Indonesia)

[bradfitz]

@kokizzu, we don't vote on issues. We all want this. That's not the issue. The issue is how to implement it correctly.

[alexkreidler]

Any new developments on this? (no pun intended)

[rsc]

No new developments. In the long term I think the answer will be that the go command will move to downloading snapshots instead of git repos, and then everything will be "shallow".

[celesteking]

Let's wait another 20 years right until github realizes go get's been downloading entire internet history on each invocation and puts the site behind a paywall.

[fsouza]

@celesteking afaik, go does shallow clones when using modules, so there's progress x)