Dep: "dep ensure -update foo/bar/baz && dep prune" takes ages to complete

[dcelasun]

Full description below.

What version of dep are you using (dep version)?

v0.3.0-142-g6ca8a48

What dep command did you run?

dep ensure -update foo/bar/baz && dep prune

What did you expect to see?

I expected a single isolated (i.e, not depending on or imported by anything else) dependency to be updated in a few seconds.

What did you see instead?

The whole thing took ~10 minutes to complete.

---

I wanted to update a single dependency with dep ensure -update. Since dep -update (unintuitively, imho) restores every single already-pruned stuff during -update I also had to run dep prune again.

Still, updating a single isolated dependency should not:

• take more than a few seconds
• affect all the other, completely unrelated dependencies
• undo unrelated previously pruned packages

Not sure if it's helpful, but Gopkg.lock is below.

Is there anything I should be doing differently? Is it a bug or expected behaviour?

# This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'.


[[projects]]
  branch = "master"
  name = "private/stuff/p1"
  packages = ["."]
  revision = "e17bb5e1c62f888e3d69071ec17b5754dc2fa6d7"

[[projects]]
  branch = "master"
  name = "private/stuff/p2"
  packages = ["."]
  revision = "0a7924b7b3c7490bf15a94cb9ed644020fa0e0b1"

[[projects]]
  branch = "master"
  name = "private/stuff/p3"
  packages = ["."]
  revision = "26597a89f60c58361c003ad290243f1ca85f605c"

[[projects]]
  branch = "master"
  name = "private/stuff/p4"
  packages = ["."]
  revision = "62eeb0956c3dda89c32dbd54fc1bf5aa3cb08bba"

[[projects]]
  branch = "master"
  name = "private/stuff/p5"
  packages = ["."]
  revision = "a16255a8e6356ad3bb4bfe5816d8ec47239fca2d"

[[projects]]
  branch = "master"
  name = "private/stuff/p6"
  packages = ["service/hooks"]
  revision = "e8c5dd8615a69e2f32d4da3afa07588d73895287"

[[projects]]
  branch = "master"
  name = "github.com/Gurpartap/logrus-stack"
  packages = ["."]
  revision = "89c00d8a28f43c567d92eb81a2945301a6a9fbb9"

[[projects]]
  branch = "v1"
  name = "github.com/Masterminds/squirrel"
  packages = ["."]
  revision = "20f192218cf52a73397fa2df45bdda720f3e47c8"

[[projects]]
  branch = "master"
  name = "github.com/armon/consul-api"
  packages = ["."]
  revision = "dcfedd50ed5334f96adee43fc88518a4f095e15c"

[[projects]]
  branch = "master"
  name = "github.com/armon/go-proxyproto"
  packages = ["."]
  revision = "48572f11356f1843b694f21a290d4f1006bc5e47"

[[projects]]
  name = "github.com/blang/semver"
  packages = ["."]
  revision = "2ee87856327ba09384cabd113bc6b5d174e9ec0f"
  version = "v3.5.1"

[[projects]]
  name = "github.com/codegangsta/negroni"
  packages = ["."]
  revision = "fde5e16d32adc7ad637e9cd9ad21d4ebc6192535"
  version = "v0.2.0"

[[projects]]
  name = "github.com/coreos/etcd"
  packages = ["client","pkg/pathutil","pkg/srv","pkg/types","version"]
  revision = "c31bec0f29facff13f7c3e3d948e55dd6689ed42"
  version = "v3.2.4"

[[projects]]
  name = "github.com/coreos/go-semver"
  packages = ["semver"]
  revision = "8ab6407b697782a06568d4b7f1db25550ec2e4c6"
  version = "v0.2.0"

[[projects]]
  branch = "master"
  name = "github.com/facebookgo/stack"
  packages = ["."]
  revision = "751773369052141c013c6e827a71e8f35c07879c"

[[projects]]
  name = "github.com/fsnotify/fsnotify"
  packages = ["."]
  revision = "629574ca2a5df945712d3079857300b5e4da0236"
  version = "v1.4.2"

[[projects]]
  branch = "master"
  name = "github.com/getsocial-rnd/ip2location-go"
  packages = ["."]
  revision = "4df59af9693f152785acc5edb1ea257b156344b1"

[[projects]]
  name = "github.com/go-sql-driver/mysql"
  packages = ["."]
  revision = "a0583e0143b1624142adab07e0e97fe106d99561"
  version = "v1.3"

[[projects]]
  name = "github.com/gorilla/context"
  packages = ["."]
  revision = "1ea25387ff6f684839d82767c1733ff4d4d15d0a"
  version = "v1.1"

[[projects]]
  branch = "master"
  name = "github.com/gorilla/schema"
  packages = ["."]
  revision = "8b1100835db5bbdae88541510f70d114b91a7e4d"

[[projects]]
  branch = "master"
  name = "github.com/hashicorp/hcl"
  packages = [".","hcl/ast","hcl/parser","hcl/scanner","hcl/strconv","hcl/token","json/parser","json/scanner","json/token"]
  revision = "392dba7d905ed5d04a5794ba89f558b27e2ba1ca"

[[projects]]
  branch = "master"
  name = "github.com/jmoiron/sqlx"
  packages = [".","reflectx"]
  revision = "d9bd385d68c068f1fabb5057e3dedcbcbb039d0f"

[[projects]]
  branch = "master"
  name = "github.com/lann/builder"
  packages = ["."]
  revision = "f22ce00fd9394014049dad11c244859432bd6820"

[[projects]]
  branch = "master"
  name = "github.com/lann/ps"
  packages = ["."]
  revision = "62de8c46ede02a7675c4c79c84883eb164cb71e3"

[[projects]]
  name = "github.com/magiconair/properties"
  packages = ["."]
  revision = "be5ece7dd465ab0765a9682137865547526d1dfb"
  version = "v1.7.3"

[[projects]]
  name = "github.com/mattes/migrate"
  packages = [".","database","database/mysql","source","source/file"]
  revision = "035c07716cd373d88456ec4d701402df52584cb4"
  version = "v3.0.1"

[[projects]]
  branch = "master"
  name = "github.com/mediocregopher/radix.v2"
  packages = ["cluster","pool","pubsub","redis","util"]
  revision = "47da4d0e14739b95177d6e1e8521bb4f2ac84f59"

[[projects]]
  branch = "master"
  name = "github.com/mitchellh/mapstructure"
  packages = ["."]
  revision = "d0303fe809921458f417bcf828397a65db30a7e4"

[[projects]]
  name = "github.com/oschwald/maxminddb-golang"
  packages = ["."]
  revision = "d19f6d453e836d12ee8fe895d0494421e93ef8c1"
  version = "v1.2.0"

[[projects]]
  name = "github.com/pelletier/go-buffruneio"
  packages = ["."]
  revision = "c37440a7cf42ac63b919c752ca73a85067e05992"
  version = "v0.2.0"

[[projects]]
  name = "github.com/pelletier/go-toml"
  packages = ["."]
  revision = "5ccdfb18c776b740aecaf085c4d9a2779199c279"
  version = "v1.0.0"

[[projects]]
  name = "github.com/satori/go.uuid"
  packages = ["."]
  revision = "879c5887cd475cd7864858769793b2ceb0d44feb"
  version = "v1.1.0"

[[projects]]
  name = "github.com/sirupsen/logrus"
  packages = ["."]
  revision = "a3f95b5c423586578a4e099b11a46c2479628cac"
  version = "1.0.2"

[[projects]]
  branch = "master"
  name = "github.com/skip2/go-qrcode"
  packages = [".","bitset","reedsolomon"]
  revision = "0bb60b5a716144fa55da373bab578cca49a537d7"

[[projects]]
  branch = "master"
  name = "github.com/spf13/afero"
  packages = [".","mem"]
  revision = "9be650865eab0c12963d8753212f4f9c66cdcf12"

[[projects]]
  name = "github.com/spf13/cast"
  packages = ["."]
  revision = "acbeb36b902d72a7a4c18e8f3241075e7ab763e4"
  version = "v1.1.0"

[[projects]]
  branch = "master"
  name = "github.com/spf13/jwalterweatherman"
  packages = ["."]
  revision = "0efa5202c04663c757d84f90f5219c1250baf94f"

[[projects]]
  name = "github.com/spf13/pflag"
  packages = ["."]
  revision = "e57e3eeb33f795204c1ca35f56c44f83227c6e66"
  version = "v1.0.0"

[[projects]]
  branch = "master"
  name = "github.com/spf13/viper"
  packages = [".","remote"]
  revision = "25b30aa063fc18e48662b86996252eabdcf2f0c7"

[[projects]]
  branch = "master"
  name = "github.com/ua-parser/uap-go"
  packages = ["uaparser"]
  revision = "e0b1fe757c944e839263af90b3c74b76d816558e"

[[projects]]
  branch = "master"
  name = "github.com/ugorji/go"
  packages = ["codec"]
  revision = "5efa3251c7f7d05e5d9704a69a984ec9f1386a40"

[[projects]]
  branch = "master"
  name = "github.com/xordataexchange/crypt"
  packages = ["backend","backend/consul","backend/etcd","config","encoding/secconf"]
  revision = "b2862e3d0a775f18c7cfe02273500ae307b61218"

[[projects]]
  name = "goji.io"
  packages = [".","internal","pat","pattern"]
  revision = "0d89ff54b2c18c9c4ba530e32496aef902d3c6cd"
  version = "v2.0"

[[projects]]
  branch = "master"
  name = "golang.org/x/crypto"
  packages = ["cast5","openpgp","openpgp/armor","openpgp/elgamal","openpgp/errors","openpgp/packet","openpgp/s2k"]
  revision = "558b6879de74bc843225cde5686419267ff707ca"

[[projects]]
  branch = "master"
  name = "golang.org/x/net"
  packages = ["context","idna","lex/httplex"]
  revision = "f5079bd7f6f74e23c4d65efa0f4ce14cbd6a3c0f"

[[projects]]
  branch = "master"
  name = "golang.org/x/sys"
  packages = ["unix","windows"]
  revision = "396c9fc8fb0ce27fc7a1ff136dc5af797bba258a"

[[projects]]
  branch = "master"
  name = "golang.org/x/text"
  packages = ["internal/gen","internal/tag","internal/triegen","internal/ucd","language","secure/bidirule","transform","unicode/bidi","unicode/cldr","unicode/norm","unicode/rangetable"]
  revision = "3bd178b88a8180be2df394a1fbb81313916f0e7b"

[[projects]]
  name = "gopkg.in/tylerb/graceful.v1"
  packages = ["."]
  revision = "4654dfbb6ad53cb5e27f37d99b02e16c1872fbbb"
  version = "v1.2.15"

[[projects]]
  branch = "v2"
  name = "gopkg.in/yaml.v2"
  packages = ["."]
  revision = "25c4ec802a7d637f88d584ab26798e94ad14c13b"

[solve-meta]
  analyzer-name = "dep"
  analyzer-version = 1
  inputs-digest = "2982228be97ef72b60341b541134eefdbcb57cb92f88dd8f359f5c21908c1376"
  solver-name = "gps-cdcl"
  solver-version = 1

[darkowlzz]

@dcelasun hi, thanks for creating an issue for this. Sadly, this is the expected behavior right now. We write the whole vendor every time, and that brings back the pruned files. There's work going on to perform prune automatically #1020 .

Due to vendor rewrite every time, dep needs all the dependencies in its cache. And if they don't already exist in cache, dep would clone all of them, and I think that's the reason for it taking so long to update in this case.

Vendor rewrite is done to ensure that vendor/ has all the dependencies in their expected state. There's work going on to implement vendor verification ( #121 ), which would eventually help us avoid full vendor rewrites.

Hope this helps.

[dcelasun]

@darkowlzz Thanks, I'll just have to wait for vendor verification and hope for the best then :)

[sdboyer]

good morning!

so, wrt slowness, we have an FAQ entry with plenty of specific information - and, fortunately, cases like these _will_ eventually be able to be completed in a matter of seconds, if not <1s, because this is mostly struggling with the segments of dep that are _extremely_ cacheable. but i'll explain things further in the context of this particular case, because i understand how this could _seem_ like an easier problem than it is.

I expected a single isolated (i.e, not depending on or imported by anything else) dependency to be updated in a few seconds.
Still, updating a single isolated dependency should not:

• take more than a few seconds
• affect all the other, completely unrelated dependencies

so, the problem here is these expectations :)

you might happen to know that these things are true beforehand for the particular dependency you're upgrading, but dep doesn't know it. dep has to _prove_ it. isolation of any change within a depgraph can never be assumed safe _a priori_, because any update could add any new import, which could have its own new imports, which could end up overlapping with existing imports...

so, yes, we absolutely must recheck the entire graph in order to be safe - and dep's first goal is safety/reliability/predictability. this is why the command is dep ensure -update, and not dep update - because the former is a slightly stronger indicator that you're not "just updating one dependency." rather, dep is ensuring everything is still in sync, while also updating that particular dependency.

you'll note, however, in the dep ensure -v output, that _if_ it turns out to be true that the dependency you're updating is isolated, that it takes only one attempt at every _other_ dependency to recheck their validity in the new solution. algorithmically, that's the best case scenario. even then, though, that process can be slow, because we're still having to run git checkouts under the hood for each version of each dependency, then statically analyzing the code in those dependencies. however, we're working on a persistent caching layer for the analyzed results - #431 - that will obviate the need for all that analysis. work on that is well underway, though my ballpark is that it's still 3-4 months away, because...y'know, caching is hard, and it'd cause a lot of user pain to get that wrong.

• undo unrelated previously pruned packages

so again, this is a mental model thing - in dep's world, vendor is dead code. and at the moment, it's basically bilious curds vomited out at the end of a previous run, because we can't yet verify it - #121 - though we're actually getting pretty close on that. a month, maybe two, i think.

because we can't yet verify anything in vendor, it means we have to regenerate vendor fully, on every single dep ensure run. this is one of the major reasons why dep is slow - that entire vendor dir is getting regenerated. this is also why prune is "undone" - because pruning, right now, is an afterthought of the model - again, vendor is dead code. however, we're working on absorbing prune back into ensure - #944 - which will make it a first-class part of the process (as well as providing more granular controls). that means pruning will never be "undone," as it'll just be the way that ensure _works_.

[sdboyer]

ah, i had this queued up and didn't see @darkowlzz' response :) oh well, double-covered!