Go: x/vgo: vgo removed -insecure flag

[saibing]

What version of Go are you using (go version)?

go version go1.10.2 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/baas/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/baas/git/go"
GORACE=""
GOROOT="/home/baas/my/go"
GOTMPDIR=""
GOTOOLDIR="/home/baas/my/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build608113082=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ curl -I http://rnd-github.huawei.com/db/parser/sql?go-get=1
HTTP/1.1 404 Not Found
Via: 1.1 172.18.32.117 (McAfee Web Gateway 7.6.2.14.0.23766)
Date: Thu, 17 May 2018 03:47:45 GMT
Server: GitHub.com
Status: 404 Not Found
X-Runtime: 0.006361
Content-Type: text/html; charset=utf-8
X-Request-Id: 715f56762358f8dbcf8968484163ceb3
Content-Length: 129318
X-Frame-Options: deny
Proxy-Connection: Keep-Alive
X-XSS-Protection: 1; mode=block
X-GitHub-Request-Id: 0f75ee93-d875-48d1-8dc7-8bcfa529656f
X-Content-Type-Options: nosniff
Content-Security-Policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; child-src 'self'; connect-src 'self' http://rnd-github.huawei.com https://status.github.com rnd-github.huawei.com; font-src 'self'; form-action 'self' rnd-github.huawei.com; frame-ancestors 'none'; frame-src 'self'; img-src * data:; media-src 'none'; script-src 'self'; style-src 'unsafe-inline' 'self'
Connection: close

$ vgo get -insecure rnd-github.huawei.com/db/parser/sql

What did you expect to see?

get the rnd-github.huawei.com/db/parser/sql from github enterprise

What did you see instead?

flag provided but not defined: -insecure
usage: get [build flags] [packages]
Run 'go help get' for details.

[kardianos]

@rsc Was the removal of the -insecure flag intentional? Should it stay removed?

[kardianos]

Insecure was probably removed because many commands can now hit remote servers. Adding insecure flags to all such commands is probably not great idea.

Perhaps if this remains a requirement we could introduce a new environment variable that takes a list of host prefixes to whitelist.

GO_WHITELIST_ALLOW_INSECURE=internal.mycorp.local/vcs:vcs.test.local

[as]

I would prefer if this was a flag. I would like to see it with the ps command as that is easier to audit. Environment variables are significantly harder to trace in this manner.

[rsc]

This is the first report I've seen of GitHub Enterprise running over HTTP instead of HTTPS. Is it really not available under HTTPS in your environment?

One workaround is to put something in your ~/.gitconfig to tell git to rewrite https://your.server to http://your.server.

[saibing]

@rsc

Thank you for your answer very much.

However, this method does not solve the problem, because vgo does not read .gitconfig file, it still access https://rnd-github.huawei.com/db/parser?go-get=1.

baas@SZX1000428520:~/git/go/src/sql$ git config --get url.http://rnd-github.com/.insteadof
https://rnd-github.com/
baas@SZX1000428520:~/git/go/src/sql$ vgo get rnd-github.huawei.com/db/parser
FindRepo: Get https://rnd-github.huawei.com/db/parser?go-get=1: cannotconnect
vgo get rnd-github.huawei.com/db/parser: Get https://rnd-github.huawei.com/db/parser?go-get=1: cannotconnect
vgo: finding rnd-github.huawei.com/db/parser v0.0.0-20180125184044-316ea1d6f45f
FindRepo: Get https://rnd-github.huawei.com/db/parser?go-get=1: cannotconnect
vgo: rnd-github.huawei.com/db/parser: Get https://rnd-github.huawei.com/db/parser?go-get=1: cannotconnect
vgo: finding rnd-github.huawei.com/go/otto v1.0.0
FindRepo: Get https://rnd-github.huawei.com/go/otto?go-get=1: cannotconnect
vgo: rnd-github.huawei.com/go/otto: Get https://rnd-github.huawei.com/go/otto?go-get=1: cannotconnect
vgo: finding rnd-github.huawei.com/go/otto v1.0.0
FindRepo: Get https://rnd-github.huawei.com/go/otto?go-get=1: cannotconnect
vgo: rnd-github.huawei.com/go/otto: Get https://rnd-github.huawei.com/go/otto?go-get=1: cannotconnect
vgo get: Get https://rnd-github.huawei.com/go/otto?go-get=1: cannotconnect
baas@SZX1000428520:~/git/go/src/sql$

[myitcv]

It looks like your git config isn't quite right (missing a colon). Here's an example:

git config url.http://localhost:8080/:.insteadof https://github.com/

Notice the : before the .insteadof.

Also, you'll need to ensure that you don't have old configs that are potentially clashing:

git config -l | grep -i insteadof

But like @rsc asked, do you really not have https available?

[saibing]

@myitcv

Thank you very much.

I tested it, this does not solve the problem

baas@SZX1000428520:~/git/go/src/sql$ git config -l | grep -i insteadof
url.http://rnd-github.com/:.insteadof=https://rnd-github.com/
baas@SZX1000428520:~/git/go/src/sql$ vgo build
vgo: finding rnd-github.huawei.com/db/parser v0.0.0-20180125184044-316ea1d6f45f
FindRepo: Get https://rnd-github.huawei.com/db/parser?go-get=1: cannotconnect
vgo: rnd-github.huawei.com/db/parser: Get https://rnd-github.huawei.com/db/parser?go-get=1: cannotconnect

For the follow problem,　I have used nginx agent to solve this problem:）

But like @rsc asked, do you really not have https available?

[saibing]

@rsc @myitcv

Thank you for your support, I have use a nginx proxy to support https

[myitcv]

Just for the record, it looks like the insteadof config didn't work because you were using the incorrect domain. vgo was trying to resolve rnd-github.huawei.com yet your config was for rnd-github.com.

So you would need to have the following:

git config url.http://rnd-github.huawei.com/:.insteadof https://rnd-github.huawei.com/

I have use a nginx proxy to support https

Glad you have a solution.

[saibing]

@myitcv

I feel sorry for my mistake

I tested it again, but it still can not solve the problem

baas@SZX1000428520:~/git/go/src/sql$ git config -l | grep .insteadof
url.http://rnd-github.huawei.com/:.insteadof=https://rnd-github.huawei.com/
baas@SZX1000428520:~/git/go/src/sql$ vgo build
vgo: finding rnd-github.huawei.com/db/parser v0.0.0-20180125184044-316ea1d6f45f
FindRepo: Get https://rnd-github.huawei.com/db/parser?go-get=1: cannotconnect

I found that vgo does not use .gitconfig's url map：

func lookupCustomDomain(path string) (Repo, error) {
    var body io.ReadCloser
    err := web.Get(
        "https://"+path+"?go-get=1",
        web.Non200OK(),
        web.Body(&body),
    )
    if body != nil {
        defer body.Close()
    }
        //...
}

[myitcv]

@saibing are you using the latest commit of vgo?

Looks like you are using an old version, pre https://github.com/golang/go/issues/24915

Can you check:

go get -u golang.org/x/vgo
(command cd $(go list -f "{{.Dir}}" golang.org/x/vgo); git rev-parse HEAD)

outputs:

for me.

[saibing]

@myitcv

me too:)

baas@SZX1000428520:~/git/go/src$ (command cd $(go list -f "{{.Dir}}" github.com/golang/vgo); git rev-parse HEAD)
6a94eb3b5ccc04453d2fb45c23641e5993118068

[myitcv]

Sorry, I've also missed something obvious here. rnd-github.huawei.com is a custom domain. Hence the issue we're seeing here happens _before_ git is called; because it's the resolution of https://rnd-github.huawei.com/db/parser?go-get=1 that is failing here. This step is required so that vgo can understand where the underlying VCS repo lives, and which tool to use (git, svn etc).

I'll defer to @rsc on whether we want to align this with the go tool or not.

[AlekSi]

Do you really not have https available?

If a company runs GitHub Enterprise or something similar behind VPN, they typically do not use HTTPS because setting it up is a hassle. For example, if they use their own custom domain like .internal it is not possible to get a "real" (trusted by major CAs) certificate, and creating their own CA and make it trusted on all hosts is painful.

So -insecure flag should ignore certificate check errors (if HTTPS is used with a self-signed certificate or certificate is issued by a custom CA), and should try HTTP is HTTPS is not available.