Go: cmd/go: validate module proxy URLs received from go-get=1 queries

Created on 13 May 2019  路  7Comments  路  Source: golang/go

What version of Go are you using (go version)?

Go tip at CL 170879 or later.

Does this issue reproduce with the latest release?

no

What operating system and processor architecture are you using (go env)?

go env Output


$ go env



GOARCH="amd64"

GOBIN=""

GOCACHE=SHOWS_CORRECTLY

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="darwin"

GOOS="darwin"

GOPATH=SHOWS_CORRECTLY

GOPROXY=""

GORACE=""

GOROOT="/usr/local/Cellar/go/1.12.2/libexec"

GOTMPDIR=""

GOTOOLDIR="/usr/local/Cellar/go/1.12.2/libexec/pkg/tool/darwin_amd64"

GCCGO="gccgo"

CC="clang"

CXX="clang++"

CGO_ENABLED="1"

GOMOD=SHOWS_CORRECTLY

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/j0/qxnj38ld075bj80p9hs2dqlw0000gn/T/go-build144819988=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

on travis, when running tests against master, modules are unable to be downloaded. i am unsure if this is because of travis or if it can be replicated elsewhere

What did you expect to see?

successful download of modules

What did you see instead?


go: github.com/golang-migrate/migrate/[email protected] requires
    github.com/fsouza/[email protected] requires
    cloud.google.com/[email protected] requires
    golang.org/x/[email protected] requires
    dmitri.shuralyov.com/app/[email protected]: Get https:///dmitri.shuralyov.com/app/changes/@v/v0.0.0-20180602232624-0a106ad413e3.info: http: no Host in request URL
The command "go mod download" failed and exited with 1 during .

specifically, the 3x slashes (/) in the url. removing the slash resolves correctly

NeedsFix help wanted modules
Source
picture maikelmclauflin

Most helpful comment

I've edited the original issue to make it more visible that this affects Go tip only. Go 1.12 and Go 1.11 are not affected.

I think there are two problems, one in cmd/go and the other in the way dmitri.shuralyov.com/app/changes vanity import path is currently served:

  1. The go command can return a better error message when it encounters an invalid/unsupported module proxy URL. "https://" is a valid URL: it specifies the HTTPS scheme, but no host and no path; see https://play.golang.org/p/vmTPnZffjDf. But I don't think it's a valid module proxy URL. I think a module proxy URL must have a non-empty host.

  2. The dmitri.shuralyov.com/app/changes vanity import path currently uses "https://" as the module proxy URL. This used to work in Go 1.11 and 1.12, perhaps unintentionally (there weren't any test cases for it), and it no longer works in Go tip after CL 170879. I think the vanity import path should change to return a module proxy URL with a non-empty host, so that it'll continue to work in Go 1.13 and later.

    Edit on May 18, 2019: This has been fixed in commit shurcooL/home@17791dca3219d26ab08ab73c7b0f0a9d81d925ae, see https://github.com/golang/go/issues/32006#issuecomment-493705089.

Since it was only affecting tip and not currently released stable versions of Go, I was using the aforementioned vanity import path as an opportunity to understand the new Go tip behavior better and learn what the best way to handle such URLs would be.

I plan to fix the dmitri.shuralyov.com/app/changes vanity import path in the coming days, but please let me know if it not working in Go tip is a more disruptive problem than I'm aware of and I'll do it sooner.

Background

The module proxy protocol documentation did not specify very formally how the URL of the module proxy was joined with the rest of the path. It just said:

The GET requests sent to a Go module proxy are:

GET $GOPROXY/\

The GET $GOPROXY/<module>/@v/list text doesn't make it clear what kind of valid URLs are permitted for $GOPROXY. "https://" happened to work in 1.11 and 1.12.

The advantage of that is that it allowed a vanity import path to serve a module at a URL that did not stutter. I.e., for a vanity import path example.com/foo, its @v/list endpoint could be served at "https://example.com/foo/@v/list" instead of "https://example.com/example.com/foo/@v/list" or "https://example.com/moduleproxy/example.com/foo/@v/list", which is more verbose.

However, after spending more time thinking about it and discussing with some people, I think the $GOPROXY URL needs to have a supported scheme and a non-empty host. I was not able to find a coherent and simple way that a URL with an empty host could be made to work.

Unfortunately, that means a module with a vanity import path being served by its own domain will have module proxy URLs that are more verbose and stutter-y, but that seems to be a favorable trade-off to make. I'm happy to hear if anyone has more thoughts on this.

picture dmitshur on 13 May 2019
👍3

All 7 comments

@jayconrod @bcmills

andybons picture andybons on 13 May 2019

dmitri.shuralyov.com is currently serving very unusual go-import tags:

~/go/src$ curl -L dmitri.shuralyov.com/app/changes?go-get=1
<meta name="go-import" content="dmitri.shuralyov.com/app/changes git https://dmitri.shuralyov.com/app/changes">
<meta name="go-import" content="dmitri.shuralyov.com/app/changes mod https://">
<meta name="go-source" content="dmitri.shuralyov.com/app/changes https://dmitri.shuralyov.com/app/changes https://gotools.org/dmitri.shuralyov.com/app/changes https://gotools.org/dmitri.shuralyov.com/app/changes#{file}-L{line}">

Note the second line: it includes a bare https://, relying on the module proxy protocol to fill in the hostname. According to @dmitshur that used to work (in Go 1.11), but it no longer does and I suspect that that's a good thing.

bcmills picture bcmills on 13 May 2019

Assigning to Dmitri to either update his server, or make a convincing argument that we should allow the bare https://. 馃檪

bcmills picture bcmills on 13 May 2019
👍1

馃檪thanks @bcmills

maikelmclauflin picture maikelmclauflin on 13 May 2019

I've edited the original issue to make it more visible that this affects Go tip only. Go 1.12 and Go 1.11 are not affected.

I think there are two problems, one in cmd/go and the other in the way dmitri.shuralyov.com/app/changes vanity import path is currently served:

  1. The go command can return a better error message when it encounters an invalid/unsupported module proxy URL. "https://" is a valid URL: it specifies the HTTPS scheme, but no host and no path; see https://play.golang.org/p/vmTPnZffjDf. But I don't think it's a valid module proxy URL. I think a module proxy URL must have a non-empty host.

  2. The dmitri.shuralyov.com/app/changes vanity import path currently uses "https://" as the module proxy URL. This used to work in Go 1.11 and 1.12, perhaps unintentionally (there weren't any test cases for it), and it no longer works in Go tip after CL 170879. I think the vanity import path should change to return a module proxy URL with a non-empty host, so that it'll continue to work in Go 1.13 and later.

    Edit on May 18, 2019: This has been fixed in commit shurcooL/home@17791dca3219d26ab08ab73c7b0f0a9d81d925ae, see https://github.com/golang/go/issues/32006#issuecomment-493705089.

Since it was only affecting tip and not currently released stable versions of Go, I was using the aforementioned vanity import path as an opportunity to understand the new Go tip behavior better and learn what the best way to handle such URLs would be.

I plan to fix the dmitri.shuralyov.com/app/changes vanity import path in the coming days, but please let me know if it not working in Go tip is a more disruptive problem than I'm aware of and I'll do it sooner.

Background

The module proxy protocol documentation did not specify very formally how the URL of the module proxy was joined with the rest of the path. It just said:

The GET requests sent to a Go module proxy are:

GET $GOPROXY/\

The GET $GOPROXY/<module>/@v/list text doesn't make it clear what kind of valid URLs are permitted for $GOPROXY. "https://" happened to work in 1.11 and 1.12.

The advantage of that is that it allowed a vanity import path to serve a module at a URL that did not stutter. I.e., for a vanity import path example.com/foo, its @v/list endpoint could be served at "https://example.com/foo/@v/list" instead of "https://example.com/example.com/foo/@v/list" or "https://example.com/moduleproxy/example.com/foo/@v/list", which is more verbose.

However, after spending more time thinking about it and discussing with some people, I think the $GOPROXY URL needs to have a supported scheme and a non-empty host. I was not able to find a coherent and simple way that a URL with an empty host could be made to work.

Unfortunately, that means a module with a vanity import path being served by its own domain will have module proxy URLs that are more verbose and stutter-y, but that seems to be a favorable trade-off to make. I'm happy to hear if anyone has more thoughts on this.

dmitshur picture dmitshur on 13 May 2019
👍3

I've made a change to the module proxy URL used to serve modules beginning with the path dmitri.shuralyov.com in commit shurcooL/home@17791dca3219d26ab08ab73c7b0f0a9d81d925ae. The URL now contains a non-empty host:

$ curl -L 'https://dmitri.shuralyov.com/app/changes?go-get=1'
<meta name="go-import" content="dmitri.shuralyov.com/app/changes git https://dmitri.shuralyov.com/app/changes">
<meta name="go-import" content="dmitri.shuralyov.com/app/changes mod https://dmitri.shuralyov.com/api/module">
<meta name="go-source" content="dmitri.shuralyov.com/app/changes https://dmitri.shuralyov.com/app/changes https://gotools.org/dmitri.shuralyov.com/app/changes https://gotools.org/dmitri.shuralyov.com/app/changes#{file}-L{line}">

As a result, those modules can be fetched successfully with Go 1.11, 1.12, and tip.

To make it possible to continue to test this issue and make improvements to the error reporting done by the go command when it encounters an invalid module proxy URL, I've left a test module that still uses "https://" as the module proxy URL here:

https://dmitri.shuralyov.com/test/modtest2

Here's the current error message from latest Go tip (commit 1ab063ce532f72851cef735238ba656cc7680b66) when trying to fetch it (directly, without a local GOPROXY):

$ gotip version
go version devel +1ab063c Fri May 17 22:32:30 2019 +0000 darwin/amd64
$ export GO111MODULE=on
$ export GOPROXY=direct
$ export GOPATH=$(mktemp -d)
$ cd $(mktemp -d)
$ gotip mod init m
go: creating new go.mod: module m
$ gotip get dmitri.shuralyov.com/test/modtest2
go: finding dmitri.shuralyov.com/test/modtest2 v0.0.0
go: dmitri.shuralyov.com/test/[email protected]: Get https:///dmitri.shuralyov.com/test/modtest2/@v/v0.0.0.info: http: no Host in request URL
dmitshur picture dmitshur on 18 May 2019

Change https://golang.org/cl/191945 mentions this issue: cmd/go: validate module proxy URLs received from go get queries

gopherbot picture gopherbot on 30 Aug 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

longzhizhi picture longzhizhi  路  3Comments
stub42 picture stub42  路  3Comments
natefinch picture natefinch  路  3Comments
enoodle picture enoodle  路  3Comments
mingrammer picture mingrammer  路  3Comments