Go: Packages under rsc.io are no longer available

Created on 28 Jul 2020 · 18Comments · Source: golang/go

What version of Go are you using (`go version`)?

Go 1.12, Go 1.14

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

I don't think this matters but here it goes. This failed both on my laptop with Go 1.14 and in GitHub Action that uses Go1.12

go env Output



$ go env

GO111MODULE=""

GOARCH="amd64"

GOBIN=""

GOCACHE="/Users/yenlinc/Library/Caches/go-build"

GOENV="/Users/yenlinc/Library/Application Support/go/env"

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="darwin"

GOINSECURE=""

GONOPROXY=""

GONOSUMDB=""

GOOS="darwin"

GOPATH="/Users/yenlinc/go"

GOPRIVATE="*"

GOPROXY="direct"

GOROOT="/usr/local/Cellar/go/1.14.3/libexec"

GOSUMDB="off"

GOTMPDIR=""

GOTOOLDIR="/usr/local/Cellar/go/1.14.3/libexec/pkg/tool/darwin_amd64"

GCCGO="gccgo"

AR="ar"

CC="clang"

CXX="clang++"

CGO_ENABLED="1"

GOMOD="/Users/yenlinc/workplace/yenlinc/amazon-kinesis-streams-for-fluent-bit/go.mod"

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/v4/nbxvs8tj4ms68w5ghhnzn_f1ckngqg/T/go-build594642831=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Ran go build command

What did you expect to see?

Modules should be fetched successfully and compilation should succeed.

What did you see instead?

Since this morning, 2 packages under rsc.io have been failing:

go build -buildmode c-shared -o ./bin/kinesis.so ./
go: github.com/aws/[email protected] requires
    github.com/golang/[email protected] requires
    rsc.io/quote/[email protected]: unrecognized import path "rsc.io/quote/v3": reading https://rsc.io/quote/v3?go-get=1: 503 Service Unavailable

$ go mod download

go: finding github.com/aws/amazon-kinesis-firehose-for-fluent-bit v1.2.2-0.20200717080229-e3ffb88e79bb
go: finding github.com/aws/aws-sdk-go v1.33.13
go: finding github.com/lestrrat-go/strftime v1.0.2-0.20200618102204-1b0bc59fa4ab
go: finding github.com/stretchr/objx v0.3.0
go: finding github.com/sirupsen/logrus v1.6.0
go: finding github.com/yuin/goldmark v1.1.33
go: rsc.io/[email protected]: unrecognized import path "rsc.io/sampler" (parse https://rsc.io/sampler?go-get=1: no go-import meta tags ())
go: finding golang.org/x/crypto v0.0.0-20200709230013-948cd5f35899
go: finding golang.org/x/text v0.3.3
go: finding github.com/google/gofuzz v1.1.0
go: finding gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776
go: finding github.com/cenkalti/backoff v2.2.1+incompatible
go: finding golang.org/x/net v0.0.0-20200707034311-ab3426394381
go: finding github.com/fluent/fluent-bit-go v0.0.0-20200707230002-2a28684e2382
go: finding github.com/stretchr/testify v1.6.1
go: finding github.com/json-iterator/go v1.1.10
go: finding gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f
go: finding golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae
go: finding golang.org/x/tools v0.0.0-20200717024301-6ddee64345a6
go: finding github.com/golang/mock v1.4.3
go: error loading module requirements

Builders NeedsInvestigation

Source

hencrice

👀1 👍1

Most helpful comment

@menghanl Fixing rsc.io/quote is being tracked in issue https://github.com/rsc/quote/issues/4.

dmitshur on 28 Jul 2020

👍4

All 18 comments

https://rsc.io/sampler?go-get=1 and https://rsc.io/quote/v3?go-get=1 show

Go 1.9 is no longer available. Please refer to https://goo.gl/aESk5L for more information.

Seems like the service that serve these pages are deprecated by Google Apps engine

hencrice on 28 Jul 2020

this is mitigated by using the proxy, which you appear to have disabled with GOPROXY=direct

but @rsc probably needs to update their app engine server

seankhliao on 28 Jul 2020

👍1

@seankhliao Thanks for the reply. I've actually tried that and it failed with the same error:

This is what I tried:

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/yenlinc/Library/Caches/go-build"
GOENV="/Users/yenlinc/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY="*"
GONOSUMDB="*"
GOOS="darwin"
GOPATH="/Users/yenlinc/go"
GOPRIVATE="*"
GOPROXY="https://proxy.golang.org"
GOROOT="/usr/local/Cellar/go/1.14.3/libexec"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.14.3/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/yenlinc/workplace/yenlinc/amazon-kinesis-streams-for-fluent-bit/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/v4/nbxvs8tj4ms68w5ghhnzn_f1ckngqg/T/go-build312867321=/tmp/go-build -gno-record-gcc-switches -fno-common"

hencrice on 28 Jul 2020

not sure then, but they're in there https://proxy.golang.org/rsc.io/quote/v3/@v/v3.1.0.zip and https://proxy.golang.org/rsc.io/sampler/@v/v1.99.99.zip

seankhliao on 28 Jul 2020

cc @rsc @dmitshur @toothrot

While this would probably not normally be considered a Go issue, it is breaking the -longtest builders.

e.g., https://storage.googleapis.com/go-build-log/3c347861/linux-amd64-longtest_bb29e807.log

prattmic on 28 Jul 2020

👍2 👀1

Just wanted to chime in and say that it's breaking for my builds (go 1.12) as well:

go test -coverprofile=coverage.txt -covermode=count ./...
go: finding github.com/golang/mock v1.4.3
go: finding github.com/go-kit/kit v0.9.0
go: finding github.com/prometheus/client_golang v1.3.0
go: finding github.com/golang/protobuf v1.4.2
go: finding github.com/kelseyhightower/envconfig v1.4.0
go: finding github.com/streadway/handy v0.0.0-20190108123426-d5acb3125c2a
go: finding github.com/stretchr/testify v1.6.1
go: finding google.golang.org/grpc v1.30.0
go: finding github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
go: finding github.com/golang/protobuf v1.3.2
go: finding github.com/go-logfmt/logfmt v0.4.0
go: finding github.com/go-stack/stack v1.8.0
go: rsc.io/quote/[email protected]: unrecognized import path "rsc.io/quote/v3" (parse https://rsc.io/quote/v3?go-get=1: no go-import meta tags ())
go: finding github.com/json-iterator/go v1.1.8
go: finding github.com/stretchr/testify v1.3.0
go: finding github.com/prometheus/client_model v0.1.0
go: finding github.com/prometheus/procfs v0.0.8
go: finding golang.org/x/tools v0.0.0-20190425150028-36563e24a262
go: finding github.com/kr/pretty v0.1.0
go: finding go.opencensus.io v0.22.1
go: finding golang.org/x/sys v0.0.0-20191220142924-d4481acd189f
go: finding github.com/cespare/xxhash/v2 v2.1.1
go: finding gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
go: finding github.com/beorn7/perks v1.0.1
go: finding github.com/davecgh/go-spew v1.1.0
go: finding golang.org/x/net v0.0.0-20190628185345-da137c7871d7
go: finding github.com/pmezard/go-difflib v1.0.0
go: finding gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127
go: finding github.com/davecgh/go-spew v1.1.1
go: finding github.com/stretchr/objx v0.1.0
go: finding github.com/prometheus/common v0.7.0
go: finding github.com/golang/mock v1.3.1
go: finding github.com/satori/go.uuid v1.2.0
go: finding cloud.google.com/go/datastore v1.1.0
go: error loading module requirements

AntonNguyen on 28 Jul 2020

👍1

@hencrice I can't reproduce this with Go 1.14 with a default configuration:

$ go version
go version go1.14.6 darwin/amd64
$ cd $(mktemp -d)
$ go mod init example.com/m
go: creating new go.mod: module example.com/m
$ go mod download github.com/aws/[email protected]
$ echo $?
0

~~What is your go env GOPROXY value? Is it modified from the default value of https://proxy.golang.org,direct? I'm guessing it does not use the module mirror.~~ Edit: I see GOPROXY="direct" in the original issue report now.

Is it an option for you to start using the module mirror (https://proxy.golang.org) by reverting GOPROXY to its default value of https://proxy.golang.org,direct?

Making cmd/go tests not depend on external services is tracked in issue #28856. We should see if the current -longtest failures should be in scope of that issue. I'll see how quickly Russ expects to be able to fix rsc.io to determine the best next steps for it.

dmitshur on 28 Jul 2020

@dmitshur I've tried to set GOPROXY to direct, https://proxy.golang.org and https://proxy.golang.org,direct. All failed with the same error message.

hencrice on 28 Jul 2020

Okay, setting GOPROXY fixed it for me:

$ export GOPROXY="https://proxy.golang.org"

Then when I directly download binaryregexp, it succeeds:

$ go get -u rsc.io/binaryregexp
go: finding github.com/go-openapi/errors v0.17.0
go: finding github.com/go-openapi/jsonreference v0.19.2
go: finding github.com/mitchellh/mapstructure v1.1.2
go: finding golang.org/x/sys v0.0.0-20190616124812-15dcb6c0061f
go: finding github.com/globalsign/mgo v0.0.0-20180905125535-1ca0a4f7cbcb
go: finding github.com/asaskevich/govalidator v0.0.0-20180720115003-f9ffefc3facf
go: finding github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329
go: finding github.com/kr/pty v1.1.5
go: finding golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8
go: finding github.com/go-openapi/jsonpointer v0.19.2
go: finding rsc.io/binaryregexp v0.2.0
go: finding github.com/PuerkitoBio/purell v1.1.1
go: finding github.com/go-openapi/loads v0.19.0
go: finding github.com/globalsign/mgo v0.0.0-20181015135952-eeefdecb41b8
go: finding golang.org/x/net v0.0.0-20190613194153-d28f0bde5980
go: finding github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63
go: finding github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578
go: finding golang.org/x/tools v0.0.0-20190614205625-5aca471b1d59
go: finding github.com/go-openapi/runtime v0.19.0
go: finding github.com/go-openapi/errors v0.18.0
go: finding github.com/go-openapi/jsonreference v0.18.0
go: finding golang.org/x/sys v0.0.0-20190321052220-f7bb7a8bee54
go: finding golang.org/x/net v0.0.0-20190320064053-1272bf9dcd53
go: finding golang.org/x/crypto v0.0.0-20190320223903-b7391e95e576
go: finding github.com/go-openapi/validate v0.18.0
go: finding github.com/go-openapi/jsonpointer v0.18.0
go: finding github.com/go-openapi/strfmt v0.18.0
go: finding github.com/go-openapi/spec v0.17.0
go: finding golang.org/x/net v0.0.0-20181005035420-146acd28ed58
go: finding github.com/mailru/easyjson v0.0.0-20190312143242-1de009706dbe
go: finding github.com/go-openapi/swag v0.17.0
go: finding github.com/docker/go-units v0.3.3
go: finding github.com/go-openapi/swag v0.18.0
go: finding github.com/go-openapi/jsonreference v0.17.0
go: finding github.com/go-openapi/loads v0.18.0
go: finding github.com/go-openapi/runtime v0.0.0-20180920151709-4f900dc2ade9
go: finding github.com/go-openapi/spec v0.18.0
go: finding github.com/go-openapi/analysis v0.0.0-20180825180245-b006789cd277
go: finding github.com/go-openapi/jsonpointer v0.17.0
go: finding github.com/PuerkitoBio/purell v1.1.0
go: finding github.com/go-openapi/strfmt v0.17.0
go: finding github.com/pborman/uuid v1.2.0
go: finding github.com/go-openapi/analysis v0.17.0
go: finding github.com/go-openapi/loads v0.17.0
go: finding github.com/go-openapi/analysis v0.18.0
go: finding github.com/google/uuid v1.0.0
go: downloading rsc.io/binaryregexp v0.2.0
go: extracting rsc.io/binaryregexp v0.2.0

AntonNguyen on 28 Jul 2020

@AntonNguyen Could you show your full go env? I tried setting GOPROXY but it still failed for me.

hencrice on 28 Jul 2020

@hencrice Please post the full output of running go version, then go env, then the exact command you ran that's still giving an error, so we have full information.

It should work okay if GOPROXY is set to use the module mirror and the value is used when you run go build or go mod download, so perhaps the problem was your configuration change wasn't picked up by the other command.

dmitshur on 28 Jul 2020

@AntonNguyen Could you show your full go env? I tried setting GOPROXY but it still failed for me.
@hencrice

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/circleci/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GOPROXY="https://proxy.golang.org"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/go/src/github.com/user/repo/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build689553606=/tmp/go-build -gno-record-gcc-switches"

AntonNguyen on 28 Jul 2020

👍1

@dmitshur

go version outputs:

go version go1.14.3 darwin/amd64

go env outputs:

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/yenlinc/Library/Caches/go-build"
GOENV="/Users/yenlinc/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY="*"
GONOSUMDB="*"
GOOS="darwin"
GOPATH="/Users/yenlinc/go"
GOPRIVATE="*"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.14.3/libexec"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.14.3/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/yenlinc/workplace/yenlinc/amazon-kinesis-firehose-for-fluent-bit/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/v4/nbxvs8tj4ms68w5ghhnzn_f1ckngqg/T/go-build709008692=/tmp/go-build -gno-record-gcc-switches -fno-common"

Content of go.mod:

module github.com/aws/amazon-kinesis-firehose-for-fluent-bit

go 1.12

require (
    github.com/aws/aws-sdk-go v1.33.13
    github.com/cenkalti/backoff v2.2.1+incompatible
    github.com/fluent/fluent-bit-go v0.0.0-20190614024040-c017a8579953
    github.com/golang/mock v1.4.3
    github.com/json-iterator/go v1.1.10
    github.com/lestrrat-go/strftime v1.0.2-0.20200618102204-1b0bc59fa4ab
    github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
    github.com/modern-go/reflect2 v1.0.1 // indirect
    github.com/sirupsen/logrus v1.6.0
    github.com/stretchr/testify v1.6.1
)

Commands ran:

$ mkdir -p ./bin && go build -buildmode c-shared -o ./bin/firehose.so ./

go: github.com/golang/[email protected] requires
    rsc.io/quote/[email protected]: unrecognized import path "rsc.io/quote/v3": reading https://rsc.io/quote/v3?go-get=1: 503 Service Unavailable

One thing I discovered is if I just:

remove the go.mod and go.sum
reinitialize the module via go mod init github.com/aws/amazon-kinesis-firehose-for-fluent-bit
Change the go version in go.mod from 1.14 to 1.12
Re-run go build
Then it compiles successfully. But I suspect that's just because the indirect dependency on rsc.io/sampler and rsc.io/quote were removed

hencrice on 28 Jul 2020

👀1 👍1

@hencrice The GOPRIVATE environment variable in go env output is set to *, which tells the go command to consider all modules as private and not to use the module proxy and checksum database for them.

To be able to fetch rsc.io/quote/v3 and other public modules from the mirror, that environment variable will need to be updated to something narrower. See https://golang.org/cmd/go/#hdr-Module_configuration_for_non_public_modules for more information about it.

dmitshur on 28 Jul 2020

👍1

This break gRPC's tests for go 1.12 as well: https://travis-ci.org/github/grpc/grpc-go

While we try to figure the reason behind the failure, is there any intent to fix rsc.io/quote?
rsc.io/quote still shows Go 1.9 is no longer available. Please refer to https://goo.gl/aESk5L for more information.

menghanl on 28 Jul 2020

@menghanl Fixing rsc.io/quote is being tracked in issue https://github.com/rsc/quote/issues/4.

dmitshur on 28 Jul 2020

👍4

rsc.io is working again now (see https://github.com/rsc/quote/issues/4#issuecomment-665300101). Making cmd/go tests not depend on external services is tracked in issue #28856. I don't think there's anything else left to do here, so I'll close this.

dmitshur on 28 Jul 2020

👍1

Russ's personal website (https://swtch.com/), which also hosts a lot of semi-official Go-related content like the Go versioning and Go proposal documents, is down with the same issue. I have a ton of respect for Russ (and Google, on good days), and these things can happen to anyone -- but one does have to see the funny side of this. As an acquaintance of mine put it: "heh, that is funny ... Go project lead's web site down due to using an EOL version of the language ... film at 11 :)"