Go: cmd/go: go mod vendor has started to report error when encountering permission errors

[Pawda]

What version of Go are you using (go version)?

$ go version
go version go1.15 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

From Docker golang:1.15-alpine

go env Output

$ go env

GO111MODULE="on"

GOARCH="amd64"

GOBIN=""

GOCACHE="/home/localuser/.cache/go-build"

GOENV="/home/localuser/.config/go/env"

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="linux"

GOINSECURE=""

GOMODCACHE="/go/pkg/mod"

GONOPROXY=""

GONOSUMDB=""

GOOS="linux"

GOPATH="/go"

GOPRIVATE=""

GOPROXY="https://proxy.golang.org,direct"

GOROOT="/usr/local/go"

GOSUMDB="sum.golang.org"

GOTMPDIR=""

GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"

GCCGO="gccgo"

AR="ar"

CC="gcc"

CXX="g++"

CGO_ENABLED="1"

GOMOD="/src/go.mod"

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build160550825=/tmp/go-build -gno-record-gcc-switches"

What did you do?

The same happens on Ubuntu but also on alpine so here is a reproducible Dockerfile

ARG GO_VERSION=1.15

FROM golang:${GO_VERSION}-alpine
ENV GO111MODULE=on

RUN adduser -D localuser
RUN mkdir /src && chown localuser /src
WORKDIR /src
USER localuser

RUN go mod init gomodisbroken
RUN echo $'\
package main \n\
\n\
import (\n\
    "fmt"\n\
    "github.com/keltia/leftpad"\n\
)\n\
\n\
func main() { \n\
    fmt.Println(leftpad.Pad("this should work", 42))\n\
}\n' | tee main.go

RUN go env

RUN mkdir -p utils/protected && chmod 000 utils/protected

RUN go mod vendor && go mod tidy

RUN CGO_ENABLED=0 go build -o a.out .

ENTRYPOINT [ "/src/a.out" ]

Then run for different go version:
go 1.15 breaking: docker build --build-arg GO_VERSION=1.15 -t gomodisbroken . && docker run --rm gomodisbroken
go 1.14 working: docker build --build-arg GO_VERSION=1.14 -t gomodisbroken . && docker run --rm gomodisbroken

What did you expect to see?

Expected go mod vendor to ignore silently folders it cannot read and exit with a 0 code.
Same as go1.14:

Step 12/14 : RUN go mod vendor && go mod tidy
 ---> Running in 56076b11eabe
go: finding module for package github.com/keltia/leftpad
go: downloading github.com/keltia/leftpad v0.1.0
go: found github.com/keltia/leftpad in github.com/keltia/leftpad v0.1.0
Removing intermediate container 56076b11eabe
 ---> 5a218931e08a
Step 13/14 : RUN CGO_ENABLED=0 go build -o a.out .
 ---> Running in 6fa88481a2ed
Removing intermediate container 6fa88481a2ed
 ---> e58b469d8dfa
Step 14/14 : ENTRYPOINT [ "/src/a.out" ]
 ---> Running in 41f90efe6d1c
Removing intermediate container 41f90efe6d1c
 ---> a35dbd80b4da
Successfully built a35dbd80b4da
Successfully tagged gomodisbroken:latest
                          this should work <nil>

What did you see instead?

with go 1.15: go mod vendor exited with code 1 due to permission denied on the folder protected.

Step 12/14 : RUN go mod vendor && go mod tidy
 ---> Running in 8a3fa42265cd
go: finding module for package github.com/keltia/leftpad
go: downloading github.com/keltia/leftpad v0.1.0
go: found github.com/keltia/leftpad in github.com/keltia/leftpad v0.1.0
pattern ...: open /src/utils/protected: permission denied
The command '/bin/sh -c go mod vendor && go mod tidy' returned a non-zero code: 1

[Pawda]

Seems to have be introduced on purpose by https://golang.org/cl/232579

As found here:
https://github.com/golang/go/blob/5c7748dc9de9c9e0a6844bf72faaf5b484004ba9/src/cmd/go/internal/modload/search.go#L72
A workaround is to have the protected folder named testdata or prefix it with "." or "_"

[dmitshur]

I agree this appears to be working as intended.

/cc @bcmills @matloob @jayconrod FYI.

[bcmills]

Yes, this is an intentional bug-fix. The go command cannot distinguish between an accidental permission error (such as when a directory is accidentally created as root) and an intentional one.

Note that you can also use a go.mod file to indicate that parts of the filesystem tree are intentionally excluded from the main module. So you could either add /src/utils/go.mod to prune out /src/utils and its subdirectories, or rename /src/utils/protected to /src/utils/_protected.