Dep: Improve performance

[sdboyer]

dep is pretty slow right now. The initial costs are hard to defray - we have to clone down repositories - but for subsequent runs, there's a lot of potential for speedups via
caching.

All of the work to be done here is in gps, but there's a series of issues that need to be addressed first.

[smyrman]

Since there is a lot of network wait, and since this is _Go_, is there not room for increased concurrency? Caching is good, but while waiting for a network resource, why not do something else...

UPDATE: I understand that there is already quite some concurrency going on.

[sdboyer]

@smyrman yep, there's a lot of concurrency already happening 😄

There's two main issues here:

1. The fundamental nature of the solving algorithm precludes doing too much predictive lookahead with network activity
2. There are also happy paths where we can largely avoid touching the network at all; we also have as an eventual goal that we want to be able to run in offline/local-only data mode

So, it's a bit of a balancing act.

Subjectively, it does appear that some of the high concurrency networking we already do may have adverse effects - e.g., throwing thirty git ls-remote at github all at once seems to cause them all to be rather slow. That's something that would be super helpful for someone to look into 😄

[beono]

Hello,

@sdboyer
Can dep use something like git clone --depth=1 and not clone the full history with changes.
I guess it will decrease the time of fetching dependencies.

There is something like this in other package managers (composer has --prefer-dist option which instead of cloning the repo just downloads the source code)

[sdboyer]

@beono hiya - unfortunately, our requirements end up being more complex than would be well-served by using --depth=1. taking that approach would not only entail nontrivial complexity additions in the codebase, but often also establishing multiple network connections within a single run. it's just not worth the tradeoff.

there are circumstances where we could determine that a tarball could be used in place of a full clone. this could be especially useful in CI. just, haven't had time to get there, yet.

[nathany]

From https://github.com/golang/go/issues/13078#issuecomment-156215459

Complication: some servers do not support --depth=1.

[take-cheeze]

I've created a command to just download things specified in Gopkg.lock: https://github.com/take-cheeze/dep-dl
It should be useful for application that doesn't update package at all.
I've tried some comparison on my local environment and initial download performance was not bad.
Though it was slower than the cached dep ensure.
BTW does tarball downloading from github help improve performance?

[yeka]

PHP Composer has composer/installed.json file under vendor directory. When the hash of dependencies inside installed.json is equal with those in .lock file, it skips re-installing those specific dependencies. It would make the overall dep ensure experience faster.

[sdboyer]

@take-cheeze oh, awesome! super cool that you put that together. i'm working on doing a direct tarball-grab automatically, when a local clone doesn't already exist. Might obviate the need for your separate tool, but i hope we can agree that's a good thing! 😄

@yeka yeah, i'm going to be adding something analogous to that pretty soon here.