Dep: fatal: failed to unpack tree object

[stevenroose]

I'm running dep ensure on commit this repo & commit, getting following error.

dep was installed with go get -u at commit 7d5cd199ce454707f81c63b7ea4299151b8b981d (most recent at time of posting).

Looks somewhat similar to https://github.com/golang/dep/issues/1439.

$ dep ensure -v
Gopkg.lock was already in sync with imports and Gopkg.toml
(1/39) Wrote gopkg.in/macaroon.v2@bed2a428da6e56d950bed5b41fcbae3141e5b0d0
(2/39) Wrote git.schwanenlied.me/yawning/bsaes.git@master
(3/39) Wrote github.com/jessevdk/go-flags@f88afde2fa19a30cf50ba4b05b3d13bc6bae3079
(4/39) Wrote github.com/jrick/logrotate@a93b200c26cbae3bb09dd0dc2c7c7fe1468a034a
(5/39) Wrote github.com/juju/loggo@master
(6/39) Wrote github.com/kkdai/bstream@f391b8402d23024e7c0f624b31267a89998fca95
(7/39) Wrote github.com/lightninglabs/gozmq@master
(8/39) Wrote github.com/davecgh/go-spew@8991bc29aa16c548c550c7ff78260e27b9ab7c73
(9/39) Wrote github.com/aead/chacha20@master
(10/39) Wrote github.com/aead/siphash@master
(11/39) Wrote github.com/awalterschulze/gographviz@761fd5fbb34e4c2c138c280395b65b48e4ff5a53
(12/39) Wrote github.com/btcsuite/btclog@84c8d2346e9fc8c7b947e243b9c24e6df9fd206a
(13/39) Wrote github.com/btcsuite/fastsha256@master
(14/39) Wrote github.com/btcsuite/go-socks@master
(15/39) Wrote github.com/btcsuite/websocket@master
(16/39) Wrote github.com/btcsuite/golangcrypto@master
(17/39) Wrote github.com/coreos/bbolt@4f5275f4ebbf6fe7cb772de987fa96ee674460a7
(18/39) Wrote github.com/Yawning/aez@4dad034d9db2caec23fb8f69b9160ae16f8d46a3
(19/39) Wrote github.com/go-errors/errors@a6af135bd4e28680facf08a3d206b454abc877a4
(20/39) Wrote github.com/lightninglabs/neutrino@601b7eda6e5c9e8ca91c097f0bb7be2664802ab3
(21/39) Wrote github.com/lightningnetwork/lightning-onion@6d4b1353e2835def84ee240f40499464521c6046
(22/39) Wrote github.com/grpc-ecosystem/grpc-gateway@f2862b476edcef83412c7af8687c9cd8e4097c0f
(23/39) Wrote github.com/golang/protobuf@bbd03ef6da3a115852eaf24c8a1c46aeb39aa175
(24/39) Wrote github.com/miekg/dns@79bfde677fa81ff8d27c4330c35bda075d360641
(25/39) Wrote github.com/ltcsuite/ltcd@5f654d5faab99ee2b3488fabba98e5f7a5257ee3
(26/39) Wrote github.com/roasbeef/btcutil@c3ff179366044979fb9856c2feb79bd4c2184c7a
(27/39) Wrote github.com/rogpeppe/fastuuid@master
(28/39) Wrote github.com/tv42/zbase32@501572607d0273fc75b3b261fa4904d63f6ffa0e
(29/39) Wrote github.com/urfave/cli@1efa31f08b9333f1bd4882d61f9d668a70cd902e
(30/39) Wrote github.com/roasbeef/btcd@e6807bc4dd5ddbb95b4ab163f6dd61e4ad79463a
(31/39) Wrote golang.org/x/crypto@49796115aa4b964c318aad4f3084fdb41e9aa067
(32/39) Wrote golang.org/x/net@ae89d30ce0c63142b652837da33d782e2b0a9b25
(33/39) Wrote google.golang.org/genproto@df60624c1e9b9d2973e889c7a1cff73155da81c4
(34/39) Wrote google.golang.org/grpc@b3ddf786825de56a4178401b7e174ee332173b66
(35/39) Wrote gopkg.in/macaroon-bakery.v2@94012773d2874a067572bd16d7d11ae02968b47b
(36/39) Wrote gopkg.in/errgo.v1@v1
(37/39) Wrote golang.org/x/sys@master
(38/39) Failed to write github.com/roasbeef/btcwallet@45445d1b09670109410174cb01fab0b133e3a904
(39/39) Failed to write golang.org/x/[email protected]
grouped write of manifest, lock and vendor: error while writing out vendor tree: failed to write dep tree: failed to export github.com/roasbeef/btcwallet: fatal: failed to unpack tree object 45445d1b09670109410174cb01fab0b133e3a904
: exit status 128

[ralfthewise]

Also getting this:

dep ensure -vendor-only
grouped write of manifest, lock and vendor: error while writing out vendor tree: failed to write dep tree: failed to export gopkg.in/fsnotify.v1: fatal: failed to unpack tree object c2828203cd70a50dcccfb2761f8b1f8ceef9a8e9

[stevenroose]

Hmm, for me it got solved by removing the $GOPATH/pkg/dep/sources/<repo> directory for the repo mentioned in the error.

[ralfthewise]

The logic in deduce.go specific to https://gopkg.in is broken. If you have a dependency to gopkg.in/fsnotify.v1 dep tries to construct the source on its own by prepending a go- rather than doing a ?go-get=1 request (as it probably should, although I don't know the details of why it operates the way it does). However, https://github.com/go-fsnotify/fsnotify is a placeholder repo and thus dep fetches garbage.

Sigh, why couldn't they have just copied one of the numerous dependency management tools from other languages that actually works...

[pkrull]

I was able to get this working by adding the following to my Gopkg.toml file.

[[override]]
  name = "gopkg.in/fsnotify.v1"
  version = "1.2.9"
  source = "[email protected]:fsnotify/fsnotify"

[ralfthewise]

Ah, great suggestion @pkrull , I'm guessing that will fix it for me. Thanks!

[stevenroose]

Does dep hard-code gopkg.in logic? Instead of just have gopkg.in do what it
does: redirect to github using the therefore intended protocol..

On Fri, 23 Mar 2018, 22:11 Tim Garton, [e-mail] wrote:

The logic in deduce.go specific to https://gopkg.in is broken. If you
have a dependency to gopkg.in/fsnotify.v1 dep tries to construct the
source on its own by prepending a go- rather than doing a ?go-get=1
request (as it probably should, although I don't know the details of why it
operates the way it does). However,
https://github.com/go-fsnotify/fsnotify is a placeholder repo and thus
dep fetches garbage.

Sigh, why couldn't they have just copied one of the numerous dependency
management tools from other languages that actually works...

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/golang/dep/issues/1760#issuecomment-375811861, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA0F3F96Nj9hYUKLhbH-h1PqFxOV_W9Oks5thXMkgaJpZM4SvR8-
.

[ralfthewise]

That fixes it for me @pkrull , thanks again for the suggestion. @stevenroose yes I believe that dep does hard-code gopkg.in logic, if I am reading the code here correctly: https://github.com/golang/dep/blob/master/gps/deduce.go#L284

[mattayes]

fsnotify is failing for me as well, @pkrull's workaround works, though I'd rather dep defer to gopkg.in for URLs, rather than hard-coding (incorrect) logic.

[sdboyer]

We have separate hard-coded logic for gopkg.in because we already had to have something special in order to filter the available versions list in a way that makes sense for the implied constraint within the gopkg.in package path, so i figured it'd be fine to skip the HTTP request and just do the work locally.

All of that was fine as long as gopkg.in acted according to its specified, regular behaviors. It stopped doing so as of a week ago - added a special rule for fsnotify. @niemeyer, i don't suppose you have any plans to return to the regular rules?

If not, we'll probably have to reorganize the way that our gopkg.in support works to reintegrate the HTTP request. Might just add the fsnotify exception rule as a stopgap, though - might be a bit of a tricky refactor.

Sigh, why couldn't they have just copied one of the numerous dependency management tools from other languages that actually works...

super.

[ralfthewise]

@sdboyer I apologize for my comment above, I realize you spend a lot of your own time working on this so that other people can benefit (myself being one of them), so thank you for that. The go dependency scene has been frustrating over the years and I have spent (hard not to say wasted) dozens of hours at this point wrestling with dependency issues what with all the different tools (godeps, glide, dep, now perhaps the upcoming vgo?). But still doesn't excuse snarky comments like the one I posted, my bad :(

[niemeyer]

@sdboyer Per the long discussion in issue #1 of go-fsnotify/fsnotify, this was a necessary workaround to an unfortunate sequence of events in a popular package. I have no plans to keep adding such workarounds.

At the same time, I can't make any promises. Just a few days ago I'd have claimed we'd not have redirects at all.

[stevenroose]

@sdboyer IMO, if there is an issue with version-discovery using gopkg.in and the default ?go-get=1 protocol, then that's not something dep should fix. That's the decision of the publisher of the package that suggests using gopkg.in instead of github or of the user that imports using gopkg.in instead of github.

With dependency managers like dep, gopkg.in becomes quite useless actually, so it's use with dep should be discouraged.

I still believe the best solution here is to remove all gopkg.in-specific logic like suggested in https://github.com/golang/dep/issues/1782.

[mattayes]

The best part of all of this is the fsnotify package recommends using github.com/fsnotify/fsnotify; the gopkg.in/fsnotify.v1 import is deprecated :stuck_out_tongue_closed_eyes: https://godoc.org/github.com/fsnotify/fsnotify

[sdboyer]

@niemeyer got it - we'll take compensating steps on our end. and yeah, predicting the future is not a business any of us want to be in 😢

@ralfthewise i appreciate it, thanks.

@stevenroose as i noted on the other issue, we can re-institute the ?go-get=1 lookup, but we cannot remove all gopkg.in-specific logic, as we have to do additional version filtering. i think, though, that you're thinking the deducer is "all" the logic, which is not the case - but the specialized deducer is the part we can remove.

[mvdan]

Dep was officially deprecated earlier this year, and the proposal to archive this repository was accepted. As such, I'm closing outstanding issues before archiving the repository. For any further comments, please use the proposal thread on the Go issue tracker. Thanks!