Dep: "dep ensure -add" fails on WSL (Bash on Windows on Linux) with "failed to strip vendor" error

Created on 30 Aug 2017  ·  20Comments  ·  Source: golang/dep

What version of dep are you using (dep version)?

dep was obtained using go get.

dep version: 

dep:
 version     : devel
 build date  :
 git hash    :
 go version  : go1.9
 go compiler : gc
 platform    : linux/amd64

cd $GOPATH/src/github.com/golang/dep && git describe --tags:

v0.3.0-204-g153953d

What dep command did you run?

dep ensure -v -add github.com/spf13/viper

What did you expect to see?

Nothing — dependency successfully added.

What did you see instead?

The following error:

...
Writing out golang.org/x/text@master
Writing out google.golang.org/api@master
Writing out google.golang.org/[email protected]
Writing out gopkg.in/[email protected]
Writing out google.golang.org/[email protected]
Writing out gopkg.in/yaml.v2@v2
Failed to write dep tree. The following errors occurred:
 *  failed to strip vendor from github.com/fatih/color: remove /tmp/dep193733944/vendor/github.com/fatih/color/vendor/golang.org/x/sys/unix: directory not empty
 *  failed to strip vendor from github.com/kris-nova/lolgopher: remove /tmp/dep193733944/vendor/github.com/kris-nova/lolgopher/vendor/golang.org/x/sys/unix: directory not empty
 *  failed to strip vendor from github.com/kris-nova/klone: remove /tmp/dep193733944/vendor/github.com/kris-nova/klone/vendor/github.com/google/go-github/github: directory not empty
grouped write of manifest, lock and vendor: error while writing out vendor tree: failed to write dep tree

I tried removing directories manually and re-running the command but it doesn't help. I also tried re-running the command but it returned the same error:

...
Writing out github.com/mitchellh/mapstructure@master
Writing out gopkg.in/yaml.v2@v2
Failed to write dep tree. The following errors occurred:
 *  failed to strip vendor from github.com/fatih/color: remove /tmp/dep696529382/vendor/github.com/fatih/color/vendor/golang.org/x/sys/unix: directory not empty
 *  failed to strip vendor from github.com/kris-nova/lolgopher: remove /tmp/dep696529382/vendor/github.com/kris-nova/lolgopher/vendor/golang.org/x/sys/unix: directory not empty
 *  failed to strip vendor from github.com/kris-nova/klone: remove /tmp/dep696529382/vendor/github.com/kris-nova/klone/vendor/github.com/google/go-github/github: directory not empty

tree /tmp/dep696529382 -d returns the following:

marko@Marko-PC:~/src/github.com/kris-nova/kubicorn$ tree /tmp/dep696529382 -d
/tmp/dep696529382
└── vendor
    └── github.com
        └── jmespath
            └── go-jmespath
                └── fuzz
                    └── corpus

6 directories

Additional information

This is my attempt to vendor github.com/spf13/viper on kubicorn project. I'm running dep on WSL, with everything updated as well as with Go 1.9.
go version:

go version go1.9 linux/amd64

go env:

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/marko"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build132465124=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

lsb_release -a:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.3 LTS
Release:        16.04
Codename:       xenial

uname -a:

Linux Marko-PC 4.4.0-43-Microsoft #1-Microsoft Wed Dec 31 14:42:53 PST 2014 x86_64 x86_64 x86_64 GNU/Linux

Running WSL on Windows 10 Pro, 64-bit, Build 1703 (Creators Update).
Output of dep hash-inputs:

github.com/digitalocean/godo
svc-^1.1.0
github.com/fatih/color
svc-^1.5.0
github.com/ghodss/yaml
svc-^1.0.0
github.com/kris-nova/lolgopher
b-master
github.com/pkg/errors
svc-^0.8.0
github.com/pkg/sftp
b-master
github.com/spf13/cobra
b-master
golang.org/x/oauth2
b-master
google.golang.org/api
b-master
-IMPORTS/REQS-
github.com/Azure/azure-sdk-for-go/arm/examples/helpers
github.com/Azure/azure-sdk-for-go/arm/network
github.com/Azure/azure-sdk-for-go/arm/resources/resources
github.com/Azure/go-autorest/autorest
github.com/Azure/go-autorest/autorest/adal
github.com/Azure/go-autorest/autorest/azure
github.com/aws/aws-sdk-go/aws
github.com/aws/aws-sdk-go/aws/awserr
github.com/aws/aws-sdk-go/aws/credentials/stscreds
github.com/aws/aws-sdk-go/aws/session
github.com/aws/aws-sdk-go/service/autoscaling
github.com/aws/aws-sdk-go/service/ec2
github.com/aws/aws-sdk-go/service/s3
github.com/digitalocean/godo
github.com/fatih/color
github.com/ghodss/yaml
github.com/kris-nova/charlie/network
github.com/kris-nova/klone/pkg/local
github.com/kris-nova/lolgopher
github.com/pkg/errors
github.com/pkg/sftp
github.com/spf13/cobra
golang.org/x/crypto/ssh
golang.org/x/crypto/ssh/agent
golang.org/x/crypto/ssh/terminal
golang.org/x/net/context
golang.org/x/net/context/ctxhttp
golang.org/x/oauth2
golang.org/x/oauth2/google
google.golang.org/api/compute/v1
google.golang.org/api/gensupport
google.golang.org/api/googleapi
google.golang.org/api/googleapi/transport
google.golang.org/api/internal
google.golang.org/api/option
google.golang.org/api/transport/bytestream/internal
google.golang.org/api/transport/grpc
google.golang.org/api/transport/http
google.golang.org/appengine
google.golang.org/appengine/socket
google.golang.org/appengine/urlfetch
google.golang.org/genproto/googleapis/bytestream
google.golang.org/grpc
google.golang.org/grpc/codes
google.golang.org/grpc/credentials
google.golang.org/grpc/credentials/oauth
google.golang.org/grpc/metadata
k8s.io/apimachinery/pkg/apis/meta/v1
-IGNORES-
-OVERRIDES-
-ANALYZER-
dep
1

Feel free to rename issue to something more appropriate.

picture xmudrii

Most helpful comment

@ibrasho I'm running on Insider Preview right now and had to use dep. I think it works flawlessly. I didn't encounter any error and actually.. it's even faster than before, at least for me.

picture xmudrii on 21 Sep 2017
👍2 🎉1

All 20 comments

Could you try that again and check the permissions on the file kept on disk? This could boil down to a file that's being created in a read-only state...

ibrasho picture ibrasho on 30 Aug 2017 
                                                                                  ‎It's using default permissions, cloned directly to the home directory. Do I need to set something special?                                                                                                                                                                                                                                                                                                                                        Sent from my BlackBerry 10 smartphone.                                                                                                                                                                                  ‎
xmudrii picture xmudrii on 30 Aug 2017

You shouldn't need to do anything. But is os.RemoveAll is failing then try looking for a file/directory it couldn't delete because​ of permissions. Can you check the permissions for the folder dep failed to delete? And any files it contains?

One other reason would be if the file descriptor​ wasn't closed when dep tried to remove the directory, but os.RemoveAll should handle that as well... This is quite puzzling. 😕

ibrasho picture ibrasho on 30 Aug 2017

here's the ls /tmp:

drwx------ 0 marko marko  512 Aug 30 12:54 dep193733944
drwx------ 0 marko marko  512 Aug 30 12:30 dep319658213
drwx------ 0 marko marko  512 Aug 30 12:46 dep334913890
drwx------ 0 marko marko  512 Aug 30 14:09 dep696529382

I ran the same command on Windows, directly, and it worked. Not sure why it fails on WSL.

xmudrii picture xmudrii on 30 Aug 2017

I was asking specifically about the files that weren't deleted but should've been like:

 *  failed to strip vendor from github.com/fatih/color: remove /tmp/dep193733944/vendor/github.com/fatih/color/vendor/golang.org/x/sys/unix: directory not empty
 *  failed to strip vendor from github.com/kris-nova/lolgopher: remove /tmp/dep193733944/vendor/github.com/kris-nova/lolgopher/vendor/golang.org/x/sys/unix: directory not empty
 *  failed to strip vendor from github.com/kris-nova/klone: remove /tmp/dep193733944/vendor/github.com/kris-nova/klone/vendor/github.com/google/go-github/github: directory not empty
ibrasho picture ibrasho on 30 Aug 2017

They don't exist. I post tree of directory in the initial post:

marko@Marko-PC:~/src/github.com/kris-nova/kubicorn$ tree /tmp/dep696529382 -d
/tmp/dep696529382
└── vendor
    └── github.com
        └── jmespath
            └── go-jmespath
                └── fuzz
                    └── corpus

6 directories

marko@Marko-PC:/mnt/c/Users/mudri$ ls /tmp/dep193733944/vendor/github.com/fatih/color/vendor/golang.org/x/sys/unix
ls: cannot access '/tmp/dep193733944/vendor/github.com/fatih/color/vendor/golang.org/x/sys/unix': No such file or directory
marko@Marko-PC:/mnt/c/Users/mudri$ ls /tmp/dep193733944/vendor/github.com/kris-nova/lolgopher/vendor/golang.org/x/sys/unix
ls: cannot access '/tmp/dep193733944/vendor/github.com/kris-nova/lolgopher/vendor/golang.org/x/sys/unix': No such file or directory
xmudrii picture xmudrii on 30 Aug 2017

Very weird. Is this behavior always happening for you on WSL?

ibrasho picture ibrasho on 30 Aug 2017

Yes, I tried many times but no avail.

xmudrii picture xmudrii on 30 Aug 2017

Can you try this in a different GOPATH?

ibrasho picture ibrasho on 30 Aug 2017

I can't right now, but will do. Will $HOME/go do the job?

xmudrii picture xmudrii on 30 Aug 2017
❤1

Sure. I'm just trying to figure out what's the reason. A different directory might give us a hint.

ibrasho picture ibrasho on 30 Aug 2017

@ibrasho I changed $GOPATH to $HOME/go but it didn't help.
Here's my env:

marko@Marko-PC:~/go/src/github.com/kris-nova/kubicorn$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/marko/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build160552007=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

Here's the error:

Writing out google.golang.org/[email protected]
Writing out gopkg.in/[email protected]
Writing out gopkg.in/yaml.v2@v2
Writing out github.com/kris-nova/charlie@master
Failed to write dep tree. The following errors occurred:
 *  failed to strip vendor from github.com/fatih/color: remove /tmp/dep789075680/vendor/github.com/fatih/color/vendor/golang.org/x/sys/unix: directory not empty
 *  failed to strip vendor from github.com/kris-nova/lolgopher: remove /tmp/dep789075680/vendor/github.com/kris-nova/lolgopher/vendor/golang.org/x/sys/unix: directory not empty
 *  failed to strip vendor from github.com/kris-nova/klone: remove /tmp/dep789075680/vendor/github.com/kris-nova/klone/vendor/github.com/google/go-github/github: directory not empty
grouped write of manifest, lock and vendor: error while writing out vendor tree: failed to write dep tree

I cleaned dep folders out from /tmp before running the command.

Additional details:

marko@Marko-PC:~/go/src/github.com/kris-nova/kubicorn$ ls /tmp
total 1.2M
drwx------ 0 marko marko  512 Aug 31 17:55 dep789075680

marko@Marko-PC:~/go/src/github.com/kris-nova/kubicorn$ ls /tmp/dep789075680/vendor/github.com/fatih/color/vendor/golang.org/x/sys/unix
ls: cannot access '/tmp/dep789075680/vendor/github.com/fatih/color/vendor/golang.org/x/sys/unix': No such file or directory

marko@Marko-PC:~/go/src/github.com/kris-nova/kubicorn$ tree -d /tmp/dep789075680/
/tmp/dep789075680/
└── vendor
    └── github.com
        └── jmespath
            └── go-jmespath
                └── fuzz
                    └── corpus

6 directories
xmudrii picture xmudrii on 31 Aug 2017

I'm having a similar problem with WSL, and precisely with the same package (github.com/jmespath/go-jmespath/fuzz/corpus). I'm getting the following error when I run dep ensure -v:

grouped write of manifest, lock and vendor: cannot delete /tmp/dep189020658/vendor: remove /tmp/dep189020658/vendor/github.com/jmespath/go-jmespath/fuzz/corpus: directory not empty

Additional details:

$ tree -d /tmp/dep189020658
/tmp/dep189020658
└── vendor
    └── github.com
        └── jmespath
            └── go-jmespath
                └── fuzz
                    └── corpus

6 directories

$ l /tmp/dep189020658/vendor/github.com/jmespath/go-jmespath/fuzz/corpus
total 160K
drwxrwxrwx 0 iuri iuri 512 Sep  6 08:49 .
drwxrwxrwx 0 iuri iuri 512 Sep  6 08:49 ..
-rw-rw-rw- 1 iuri iuri  10 Sep  6 08:46 expr-503
-rw-rw-rw- 1 iuri iuri   9 Sep  6 08:46 expr-504
-rw-rw-rw- 1 iuri iuri   8 Sep  6 08:46 expr-505
-rw-rw-rw- 1 iuri iuri   7 Sep  6 08:46 expr-506
-rw-rw-rw- 1 iuri iuri  10 Sep  6 08:46 expr-507
-rw-rw-rw- 1 iuri iuri   8 Sep  6 08:46 expr-508
-rw-rw-rw- 1 iuri iuri   9 Sep  6 08:46 expr-509
-rw-rw-rw- 1 iuri iuri  18 Sep  6 08:46 expr-51
-rw-rw-rw- 1 iuri iuri   7 Sep  6 08:46 expr-510
-rw-rw-rw- 1 iuri iuri   6 Sep  6 08:46 expr-511
-rw-rw-rw- 1 iuri iuri   8 Sep  6 08:46 expr-512
-rw-rw-rw- 1 iuri iuri  11 Sep  6 08:46 expr-513
-rw-rw-rw- 1 iuri iuri   7 Sep  6 08:46 expr-514
-rw-rw-rw- 1 iuri iuri   9 Sep  6 08:46 expr-515
-rw-rw-rw- 1 iuri iuri   8 Sep  6 08:46 expr-516
-rw-rw-rw- 1 iuri iuri   9 Sep  6 08:46 expr-517
-rw-rw-rw- 1 iuri iuri   9 Sep  6 08:46 expr-518
-rw-rw-rw- 1 iuri iuri  12 Sep  6 08:46 expr-519
-rw-rw-rw- 1 iuri iuri  22 Sep  6 08:46 expr-52
-rw-rw-rw- 1 iuri iuri  12 Sep  6 08:46 expr-520
-rw-rw-rw- 1 iuri iuri  11 Sep  6 08:46 expr-521
-rw-rw-rw- 1 iuri iuri   8 Sep  6 08:46 expr-508
-rw-rw-rw- 1 iuri iuri   9 Sep  6 08:46 expr-509
-rw-rw-rw- 1 iuri iuri  18 Sep  6 08:46 expr-51
-rw-rw-rw- 1 iuri iuri   7 Sep  6 08:46 expr-510
-rw-rw-rw- 1 iuri iuri   6 Sep  6 08:46 expr-511
-rw-rw-rw- 1 iuri iuri   8 Sep  6 08:46 expr-512
-rw-rw-rw- 1 iuri iuri  11 Sep  6 08:46 expr-513
-rw-rw-rw- 1 iuri iuri   7 Sep  6 08:46 expr-514
-rw-rw-rw- 1 iuri iuri   9 Sep  6 08:46 expr-515
-rw-rw-rw- 1 iuri iuri   8 Sep  6 08:46 expr-516
-rw-rw-rw- 1 iuri iuri   9 Sep  6 08:46 expr-517
-rw-rw-rw- 1 iuri iuri   9 Sep  6 08:46 expr-518
-rw-rw-rw- 1 iuri iuri  12 Sep  6 08:46 expr-519
-rw-rw-rw- 1 iuri iuri  22 Sep  6 08:46 expr-52
-rw-rw-rw- 1 iuri iuri  12 Sep  6 08:46 expr-520
-rw-rw-rw- 1 iuri iuri  11 Sep  6 08:46 expr-521
-rw-rw-rw- 1 iuri iuri   9 Sep  6 08:46 expr-522
-rw-rw-rw- 1 iuri iuri  14 Sep  6 08:46 expr-523
-rw-rw-rw- 1 iuri iuri  11 Sep  6 08:46 expr-524
-rw-rw-rw- 1 iuri iuri  10 Sep  6 08:46 expr-525
-rw-rw-rw- 1 iuri iuri  11 Sep  6 08:46 expr-526
-rw-rw-rw- 1 iuri iuri   9 Sep  6 08:46 expr-527
-rw-rw-rw- 1 iuri iuri   9 Sep  6 08:46 expr-528
-rw-rw-rw- 1 iuri iuri  11 Sep  6 08:46 expr-529
-rw-rw-rw- 1 iuri iuri  19 Sep  6 08:46 expr-53
-rw-rw-rw- 1 iuri iuri  13 Sep  6 08:46 expr-530
-rw-rw-rw- 1 iuri iuri  11 Sep  6 08:46 expr-531
-rw-rw-rw- 1 iuri iuri   9 Sep  6 08:46 expr-532
-rw-rw-rw- 1 iuri iuri   3 Sep  6 08:46 expr-533
-rw-rw-rw- 1 iuri iuri   6 Sep  6 08:46 expr-534
-rw-rw-rw- 1 iuri iuri   8 Sep  6 08:46 expr-535
-rw-rw-rw- 1 iuri iuri   6 Sep  6 08:46 expr-536
-rw-rw-rw- 1 iuri iuri   7 Sep  6 08:46 expr-537
-rw-rw-rw- 1 iuri iuri   3 Sep  6 08:46 expr-538
-rw-rw-rw- 1 iuri iuri   1 Sep  6 08:46 expr-539
-rw-rw-rw- 1 iuri iuri  20 Sep  6 08:46 expr-54
-rw-rw-rw- 1 iuri iuri   3 Sep  6 08:46 expr-540
-rw-rw-rw- 1 iuri iuri   5 Sep  6 08:46 expr-541
-rw-rw-rw- 1 iuri iuri   4 Sep  6 08:46 expr-542
-rw-rw-rw- 1 iuri iuri   2 Sep  6 08:46 expr-543
-rw-rw-rw- 1 iuri iuri   3 Sep  6 08:46 expr-544
-rw-rw-rw- 1 iuri iuri   3 Sep  6 08:46 expr-545
-rw-rw-rw- 1 iuri iuri   7 Sep  6 08:46 expr-546
-rw-rw-rw- 1 iuri iuri   7 Sep  6 08:46 expr-547
-rw-rw-rw- 1 iuri iuri   6 Sep  6 08:46 expr-548
-rw-rw-rw- 1 iuri iuri   6 Sep  6 08:46 expr-549
-rw-rw-rw- 1 iuri iuri  16 Sep  6 08:46 expr-55
-rw-rw-rw- 1 iuri iuri   7 Sep  6 08:46 expr-550
-rw-rw-rw- 1 iuri iuri   9 Sep  6 08:46 expr-551
-rw-rw-rw- 1 iuri iuri  14 Sep  6 08:46 expr-552
-rw-rw-rw- 1 iuri iuri  27 Sep  6 08:46 expr-96
-rw-rw-rw- 1 iuri iuri  19 Sep  6 08:46 expr-97
-rw-rw-rw- 1 iuri iuri  20 Sep  6 08:46 expr-98

My env (running dep version v0.3.0):

$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/iuri/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build410158927=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

$ go version
go version go1.9 linux/amd64
iuriaranda picture iuriaranda on 6 Sep 2017

That folder (github.com/jmespath/go-jmespath/fuzz/corpus) has 645 files in it, maybe that has something to do with this issue, maybe dep tries to delete the folder before all the files are properly deleted. 🤔

iuriaranda picture iuriaranda on 6 Sep 2017

I'm seeing this consistently on WSL as well, when running dep ensure I get the following issue:
grouped write of manifest, lock and vendor: cannot delete /tmp/dep314869223/vendor: remove /tmp/dep314869223/vendor/github.com/Shopify/sarama: directory not empty

Checking the folder lists "github.com/pierrec/lz4/fuzz/corpus" which is imported in Shopify/sarama/message.go. I'm unable to resolve this on WSL/windows

eirikbell picture eirikbell on 8 Sep 2017

@xmudrii @iuriaranda @eirikbell

Could please check #1136?

Quoting @redbaron:

subscribe to Insiders Preview builds in your update settings to make it work or wait for Fall Creator's Update.

ibrasho picture ibrasho on 20 Sep 2017

@ibrasho I'm running on Insider Preview right now and had to use dep. I think it works flawlessly. I didn't encounter any error and actually.. it's even faster than before, at least for me.

xmudrii picture xmudrii on 21 Sep 2017
👍2 🎉1

Glad to hear that. I'll close this issue then.

If you face any other issues, don't hesitate to report them.

ibrasho picture ibrasho on 21 Sep 2017
🎉1 👍1

FYI: This issue should be related to https://github.com/golang/go/issues/20841 .
And may be fixed in Windows 10 Fall Creators Update ("RS3", build 16299) according to https://github.com/golang/go/issues/20841#issuecomment-350799311 .

Or if you want to avoid this error without upgrading Windows 10, you must apply patch like:

tyru picture tyru on 27 Dec 2017

I've also bee hitting this issue in WSL even with the latest Windows updates, by sharing the $GOPATH between WSL and Windows.

The workaround here is to set DEPCACHEDIR on WSL, as introduced by https://github.com/golang/dep/pull/1234.

radu-matei picture radu-matei on 31 Oct 2018
Was this page helpful?
0 / 5 - 0 ratings

Related issues

fabulous-gopher picture fabulous-gopher  ·  27Comments
davecheney picture davecheney  ·  32Comments
sdboyer picture sdboyer  ·  31Comments
cpapidas picture cpapidas  ·  50Comments
pbennett picture pbennett  ·  30Comments