Dep: `Failed to write` and `failed to unpack tree object` during dep ensure

[Southclaws]

What version of dep are you using (dep version)?

0.3.2 and tip

What dep command did you run?

dep ensure

What did you expect to see?

A successful ensure - my dependencies haven't changed in a while so this was completely unexpected.

What did you see instead?

It first failed in CI (CI does a go-get so it's always tip - I see why that is a bad idea now!) which produced this error

Then when I tried it locally, I got this:

grouped write of manifest, lock and vendor: error while writing out vendor tree: failed to write dep tree: failed to export github.com/minio/go-homedir: fatal: failed to unpack tree object 21304a94172ae3a09dee2cd86a12fb6f842138c7

So I deleted ./vendor and tried again, same result. Not sure if an issue with go-homedir, it recently got a couple of changes but I don't see how that could break dep at all.

I'm now getting this with 0.3.2, never seen it before:

$ dep ensure -v
Gopkg.lock was already in sync with imports and Gopkg.toml
(1/33) Wrote github.com/mattn/[email protected]
(2/33) Wrote github.com/google/[email protected]
(3/33) Wrote github.com/davecgh/[email protected]
(4/33) Wrote github.com/jbenet/go-context@master
(5/33) Wrote github.com/mgutz/ansi@master
(6/33) Wrote gopkg.in/yaml.v2@v2
(7/33) Wrote github.com/Masterminds/[email protected]
(8/33) Wrote github.com/docker/[email protected]
(9/33) Wrote github.com/michaelbironneau/garbler@master
(10/33) Wrote github.com/ghodss/[email protected]
(11/33) Wrote github.com/mattn/[email protected]
(12/33) Wrote github.com/Microsoft/[email protected]
(13/33) Wrote github.com/mitchellh/go-homedir@master
(14/33) Wrote github.com/pkg/[email protected]
(15/33) Failed to write github.com/pmezard/[email protected]
(16/33) Failed to write github.com/docker/[email protected]
(17/33) Failed to write golang.org/x/net@master
(18/33) Failed to write gopkg.in/AlecAivazis/[email protected]
(19/33) Failed to write github.com/nbutton23/[email protected]
(20/33) Failed to write golang.org/x/sys@master
(21/33) Failed to write github.com/xanzy/ssh-agent@master
(22/33) Failed to write gopkg.in/src-d/[email protected]
(23/33) Failed to write github.com/sergi/go-diff@master
(24/33) Failed to write github.com/docker/[email protected]
(25/33) Failed to write github.com/minio/go-homedir@master
(26/33) Failed to write github.com/stretchr/[email protected]
(27/33) Failed to write golang.org/x/text@master
(28/33) Failed to write golang.org/x/crypto@master
(29/33) Failed to write gopkg.in/src-d/[email protected]
(30/33) Failed to write github.com/docker/[email protected]
(31/33) Failed to write github.com/src-d/[email protected]
grouped write of manifest, lock and vendor: error while writing out vendor tree: failed to write dep tree: failed to export github.com/pmezard/go-difflib: cannot stat C:\Users\Southclaw\Go\pkg\dep\sources\https---github.com-pmezard-go--difflib\.git\index: CreateFile C:\Users\Southclaw\Go\pkg\dep\sources\https---github.com-pmezard-go--difflib\.git\index: The system cannot find the file specified.

I'm now going to try deleting everything and running init from scratch...

[sdboyer]

hi, welcome! if you run tip on your local, i think it should automatically heal the issue with pmezard/go-difflib. the problem with go-minio is more concerning, though - does it reliably re-occur?

[Southclaws]

Yeah the other issue fixed itself, I just deleted the source directory and re-ran it.

The other issue is consistently happening to go-homedir but not the same error, this time it was failed to export: https://travis-ci.org/Southclaws/sampctl/builds/313085584#L1345

This is completely blocking these builds from deploying and the only thing I can think of doing is to fork go-homedir and directing my Gopkg.toml to the fork.

[sdboyer]

i've just merged a PR into tip that'll give us more insight into the actual git error output. if you could trigger another build so that we could see what's going on (assuming your builds are still running tip), that'd be helpful.

[lclarkmichalek]

I'm getting this with this in the Gopkg.lock (not a dep generated lock file, so likely wholly incorrect in various ways, but that shouldn't be relevant for this issue).

[[projects]]
  name = "github.com/dsnet/compress"
  packages = []
  revision = "0ae8e136a5df9e3caf6c0f69983608b07438411b"

Tracking down the repo, and running the command that is failing:

$ git read-tree 0ae8e136a5df9e3caf6c0f69983608b07438411b
fatal: failed to unpack tree object 0ae8e136a5df9e3caf6c0f69983608b07438411b

I can see the commit on github: https://github.com/dsnet/compress/commit/0ae8e136a5df9e3caf6c0f69983608b07438411b

But taking a look at the history of the files touched by that commit using git directly:

$ git log -- bzip2/internal/sais/sais_gen.go
commit 9ab2d87e091afdb4d89ddbd736e667c998adc475
Author: Joe Tsai <[email protected]>
Date:   Thu May 25 13:35:15 2017 -0700

    bzip2/internal/sais: prevent sais_gen.go from being considered generated code

    Any source that matches "^// Code generated .* DO NOT EDIT\.$"
    is considered generated source. Factor this string out from the
    source template to avoid sais_gen.go from falsely being identified
    as generated code.

commit d7bc70ae03efe5ac206eb8f9e1dc51bb6bc3e402
Author: Joe Tsai <[email protected]>
Date:   Wed May 24 17:29:38 2017 -0700

    bzip2/internal/sais: use generators for sais_byte.go and sais_int.go

And the commit seems to have a different hash. There's a chance I'm being really stupid here, and my git is rusty (have you heard good news about our lord and saviour mercurial?), but that seems like the kind of thing that shouldn't happen. This also happens when I clone the repo fresh, so maybe my git is just fsked? Also works with the github.com/minio/go-homedir@21304a94172ae3a09dee2cd86a12fb6f842138c7 example above

[euank]

@lclarkmichalek your dnsnet/compress commit is in the master branch of a fork, specifically https://github.com/gy-games-libs/compress/commits/master (https://github.com/gy-games-libs/compress/commit/0ae8e136a5df9e3caf6c0f69983608b07438411b).

Github lets the "/commit/" url for a repository also display commits across all forks of the repo. I don't really know why, but I've run into that before.

To get a similar result to github's commit view, you need to also add all public forks as remotes and fetch all their branches (and probably all their tags?).

In the case of go-homedir, that commit lives in the master branch of "storageio"'s fork, here https://github.com/storageio/go-homedir/commits/master, so it's the same deal; it's a commit that lives in another fork and can be found via github, but not via git clone unless the fork's is used as the remote instead.

[Southclaws]

I'm running into this again, with a reliable repro this time:

grouped write of manifest, lock and vendor: error while writing out vendor tree: failed to write dep tree: failed to export github.com/minio/go-homedir: fatal: failed to unpack tree object 21304a94172ae3a09dee2cd86a12fb6f842138c7

Just pulled master tip:

dep:
version : v0.3.2
build date : 2017-10-19
git hash : 8ddfc8a
go version : go1.9
go compiler : gc
platform : windows/amd64

The project I'm doing this on is sampctl if that helps debugging on your end, it seems to be happening every time now - when I reported this, it happened once then disappeared mysteriously...

Last time when I hit the other issue, I just deleted my vendor folder and re-ensured and it worked as a quick workaround but this time I'm not getting the vendor folder back at all. I also tried clearning the dep folder in $GOPATH/pkg but that isn't working either...

Not entirely sure what to do now, I can't build my project at all now.

[euank]

@Southclaws per my above comment, the commit it's erroring about (21304a94, the one currently in your Gopkg.lock) is not present in that repository.

The Gopkg.lock has had that commit, which exists in the storageio fork, in it since the original dep vendoring.

If you wish to use commits in that repository, you should add source = "https://github.com/storageio/go-homedir" to the go-homedir constraint.

It seems likely that you got into this situation in the first place by having a $GOPATH which included the storageio fork (whether because another project depended on it, or because you actively cloned it).

Since your lockfile contains a commit for the project (as described in your Gopkg.toml) doesn't exist, I don't think dep can really do anything better here. Your repo's in a broken state. Whatever got it into that state, be it a dirty gopath or older version of dep or both, seems like a more likely culprit.

[Vaishnavij]

Running dep ensure -update fixed the issue. But the digest & revision version of the packages in Gopkg.lock were updated.