Dep: Feedback: things take a long time

[calmh]

I've been playing with this for literally less than ten minutes, so take this with a huge grain of salt. And please temper any criticism with a background of YAY THIS IS HAPPENING I'M SO EXCITED!

The first thing that strikes me when trying it out is "wow, this is slow". I suspect this is because there are a lot of network operations, and there is perhaps limited or no caching yet? I'm on a fast fiber connection so it's not that my internet is really crap. Starting out on Syncthing, I saw:

• dep init - the first command I ran. I would have expected this to be instantaneous:ish a la git init, but it took almost 30 seconds. Or at least it felt like an eternity, I didn't time it.

• dep ensure -update - failed the first time due to a constraint error. This was due to repos in my GOPATH having branches that didn't exist upstream. I fixed that in the manifest.json. Second run took quite a while, but I can expect this as I suppose it's checking out repos.

• I then entered something of a loop playing with dep ensure $package@$someversion and doing dep status to check the result. Each dep status invocation takes 23 seconds for me. Maybe this is the latest fetching? But it quickly got repetitive...

jb@unu:~/s/g/s/syncthing $ time dep ensure github.com/calmh/luhn@^1.0.0

real    0m3.247s
user    0m2.453s
sys 0m2.472s
jb@unu:~/s/g/s/syncthing $ time dep status
PROJECT                                   CONSTRAINT     VERSION        REVISION  LATEST   PKGS USED
github.com/AudriusButkevicius/cli         branch master  branch master  7f561c7   7f561c7  1  
github.com/AudriusButkevicius/go-nat-pmp  branch master  branch master  452c976   452c976  1  
github.com/bkaradzic/go-lz4               branch master  branch master  7224d8d   7224d8d  1  
github.com/calmh/du                       branch master  v1.0.0         3c0690c   3c0690c  1  
github.com/calmh/luhn                     ^1.0.0         v1.0.0         0c8388f   v1.0.0   1  
github.com/calmh/xdr                      branch master  branch master  08e072f   08e072f  1  
github.com/chmduquesne/rollinghash        branch master  branch master  53c059e   53c059e  1  
github.com/cznic/b                        *              branch master  bcff30a   bcff30a  1  
github.com/cznic/fileutil                 *              branch master  e618435   e618435  1  
github.com/cznic/internal                 *              1.0.0          cef02a8   1.0.0    2  
github.com/cznic/lldb                     *              v1.0.5         7376b3b   v1.0.5   1  
github.com/cznic/mathutil                 *              branch master  4609a45   4609a45  1  
github.com/cznic/ql                       branch master  v1.1.0         7a63cd7   7a63cd7  1  
github.com/cznic/sortutil                 *              branch master  4c73428   4c73428  1  
github.com/cznic/strutil                  *              branch master  7cacf66   7cacf66  1  
github.com/cznic/zappy                    *              branch master  2533cb5   2533cb5  1  
github.com/d4l3k/messagediff              *              v1.1.0         7b70699   v1.1.0   1  
github.com/edsrzf/mmap-go                 *              branch master  935e0e8   935e0e8  1  
github.com/gobwas/glob                    branch master  v0.2.2         bea32b9   bea32b9  1  
github.com/gogo/protobuf                  branch master  branch master  265e960   265e960  2  
github.com/golang/groupcache              branch master  branch master  72d04f9   72d04f9  1  
github.com/golang/snappy                  *              branch master  7db9049   7db9049  1  
github.com/jackpal/gateway                branch master  branch master  5795ac8   5795ac8  1  
github.com/kardianos/osext                branch master  branch master  c2c54e5   c2c54e5  1  
github.com/lib/pq                         branch master  branch master  67c3f2a   67c3f2a  1  
github.com/minio/sha256-simd              branch master  branch master  e82e73b   e82e73b  1  
github.com/mitchellh/go-wordwrap          branch master  branch master  ad45545   ad45545  1  
github.com/oschwald/geoip2-golang         branch master  v1.0.0         496a3cb   496a3cb  1  
github.com/oschwald/maxminddb-golang      *              v1.1.0         cd5998e   v1.1.0   1  
github.com/rcrowley/go-metrics            *              branch master  1f30fe9   1f30fe9  1  
github.com/sasha-s/go-deadlock            *              v0.1.0         09aefc0   v0.1.0   1  
github.com/syndtr/goleveldb               branch master  branch master  23851d9   23851d9  6  
github.com/thejerf/suture                 branch master  branch master  766acec   766acec  1  
github.com/vitrun/qart                    branch master  0.1            bf64b92   bf64b92  1  
golang.org/x/crypto                       branch master  branch master  41d678d   41d678d  1  
golang.org/x/net                          branch master  branch master  f249948   f249948  5  
golang.org/x/sys                          *              branch master  d75a526   d75a526  2  
golang.org/x/text                         branch master  branch master  11dbc59   11dbc59  1  
golang.org/x/time                         branch master  branch master  f51c127   f51c127  1  

real    0m22.657s
user    0m0.897s
sys 0m0.585s
jb@unu:~/s/g/s/syncthing $ 

I suspect that my dependency graph is quite small compared to many out there. :)

[sdboyer]

I've been playing with this for literally less than ten minutes, so take this with a huge grain of salt. And please temper any criticism with a background of YAY THIS IS HAPPENING I'M SO EXCITED!

🎉 🎉 🎉 🎉 🎉 🎉 🎉

totally, got it :) thanks for the feedback!!

Second run took quite a while, but I can expect this as I suppose it's checking out repos.

Yep, and this is a one-time cost (unless you flush out the cache dir) - basically unavoidable.

I then entered something of a loop playing with dep ensure $package@$someversion and doing dep status to check the result. Each dep status invocation takes 23 seconds for me. Maybe this is the latest fetching? But it quickly got repetitive...

Only two things that should be happening there are, yes, getting the latest version info for all your deps, and

In general, we have a LOT of headroom for performance improvements. Some more details in #67, but we're moving towards some caching that, assuming warm caches, should have this command below 1s. If current design plans hold, then I'd expect all such caches would be warmed by your first run of the tool.

We might also add some options that allow you to omit that Latest column, mostly obviating the need for network activity at all.

[flibustenet]

It could be more verbose, specially in the alpha stage. I thought it freezed on the first run.

[sdboyer]

@flibustenet did you run with -v?

[peterbourgon]

Yep, and this is a one-time cost (unless you flush out the cache dir) - basically unavoidable.

This is repeated feedback. I think we need to have init print out what it's doing, even at a coarse level ("cloning foo/bar/baz") and even without -v.

[sdboyer]

That seems fair to me. It does feel like other go tooling generally eschews such informational output. But, as @flibustenet pointed out, this is alpha - I think we can not worry about that right now.

[calmh]

I think it's a passing phase. At the moment no one knows what to expect. Once the tool is established and its operation generally understood it probably makes sense to adopt a less verbose default.

[flibustenet]

I was so excited i didn't try -v and -h ! (to put in the readme ?) #62

[nathany]

Is git init reaching out to the network? It seems like it's shelling out to git quite a lot.

If it was possible to make it significantly faster, I don't think the feedback (#62) without -v would be necessary.

[sdboyer]

First runs are often quite slow, as there's a lot of initial cloning/bzr branching/whatever to do. Not much we'll really be able to do there, ever. But subsequent dep inits of the same project should be much faster - the clone is a one-time thing. And there's a lot more potential speedups in the works - #67

[erizocosmico]

Couldn't the clones be run concurrently (if they are not yet)? That would certainly speed things up (don't really know if you can clone two repos at the sime time, though).

[sdboyer]

Yeah, they're already concurrent. No wins to be had there, unfortunately.

I'll point to #67 again - it really lays out most of the paths forward here :)

[sdboyer]

closing in favor of #289