Go: proxy.golang.org: does not work in China

[ipfans]

What version of Go are you using (go version)?

$ go version 1.12.4

Does this issue reproduce with the latest release?

Yes

What did you do?

China have one of biggest Gopher communities. For reasons known to all, golang.org can not be accessed from China. But we have golang.google.cn mirrors to access golang.org site.

Now we have proxy.golang.org and sum.golang.org to accelerate go modules, but those still can not works in China.

What did you expect to see?

1. proxy.golang.org to proxy.golang.google.cn
2. sum.golang.org to sum.golang.google.cn

[beoran]

While having a mirror is fine, maybe some brave Chinese citizens could contact their government and let them know that they are blocking what is basically a free and open source handout of technology and knowledge to their own detriment. IIRC, the block on Github in china was lifted after similar complaints.

[googollee]

@beoran It's hard to do that because golang.org is hosted by Google servers, and government blocked all Google servers (or trying to do that). That's why Go team created a mirror site golang.google.cn for Chinese access. It's reasonable that Go team could bring proxy and sum services to golang.google.cn to resolve this problem.

[ALTree]

cc @andybons

[bradfitz]

@googollee, it's not that simple. A requirement for Google getting Chinese approval for golang.google.cn was that we didn't serve any user-generated content (UGC). That's why the playground is disabled on the Chinese domain, as play.golang.org's "Share" functionality lets people share user-generated content (code).

It's hard to run a caching module proxy mirror without user generated content. All the code in the modules is user-generated.

It might be possible, however, to run https://sum.golang.org/ on a *.cn domain (requiring approval) since that doesn't serve any UGC... only cryptographic sums. Unless the GitHub module names themselves could be considered UGC, which they might. Then people can run a proxy inside China, but at least everybody can be on the same checksum database. (See @FiloSottile's comments an https://go-review.googlesource.com/c/go/+/174338/4#message-3b3b14410b157d8ebb620dd6c08268efb94f8763)

[oiooj]

@ipfans You can try gosum.io first, but as @FiloSottile said, we recommend that you use sum.golang.org if you can.

[aofei]

It's not that we are not brave, @beoran. This is an extremely complicated matter in our country. It's because we can't do anything about it, so some of us need help from the Go team.

And no UGC is impossible at all. So, since Go team can not set up proxies on golang.google.cn, we should talk about the #31757. At least we can set up proxies ourselves. But once the proxy.golang.org becomes the default, teaching beginners to learn Go will be a very difficult or even illegal thing (we need to teach them how to break through the network blockade) in China and countries that @davecheney mentioned in https://github.com/golang/go/issues/25530#issuecomment-469583362, right?

So let's head to #31757 and continue the discussion, what do you think, boss? @bradfitz

[bradfitz]

So let's head to #31757 and continue the discussion, what do you think, boss? @bradfitz

As I said there, let's use this bug. It predates your bug and is mostly overlapping. Both are about China & the experience for users there.

Disabling GOPROXY might help China, but doesn't help the rest of the world.

We should find a solution that works for both those firewalled and those not.

[go101]

Is sum checking a must for projects using modules? Can we skip it?

[aofei]

@go101 It's necessary, see #25530.

[oiooj]

Is sum checking a must for projects using modules? Can we skip it?

@go101 You can set environment GOSUMDB="off" to disable it now.

From docs:

GOSUMDB defaults to "sum.golang.org" when GOPROXY="https://proxy.golang.org"
and otherwise defaults to "off". NOTE: The GOSUMDB will later default to
"sum.golang.org" unconditionally.

If GOSUMDB is set to "off", or if "go get" is invoked with the -insecure flag,
the checksum database is never consulted, but at the cost of giving up the
security guarantee of verified repeatable downloads for all modules.
A better way to bypass the checksum database for specific modules is
to use the GONOSUMDB environment variable.

[ipfans]

@oiooj gosum.io is cool. But for security reason, I think we can not trust 3rd-party sum database, right?

We still need an official database to check the 3rd party go module proxy is trustable.

Ref：XcodeGhost Security Attack.

[aofei]

Okay, as you wish, boss. @bradfitz

As I said in #31757. I started the goproxy/goproxy.cn with Qiniu Cloud mainly to make everyone better use Go modules (or just Go) in the mainland of China. Note that it's not like the goproxy.io created by @oiooj (thank you for doing this of course). The goproxy.cn will be a business-supported project rather than a personal project. This means that the goproxy.cn will be fully legal in China and we will try our best to spread it. However, when GOPROXY has a default value (a blocked endpoint), not only will our goproxy.cn become more difficult to spread, but we must even say "You must first change your GOPROXY to something else before you start Go" in every single Go beginner's teaching article. As I said, imagine how disappointing the Chinese (and other countries with the same problems) Go beginners would be if they couldn't run a simple "hello, world" with the gin framework or something else.

And I know disabling GOPROXY by default will not help the rest of the world. Of course, we are not selfish like that. So let's find a more perfect way.

What about reminding users to modify their GOPROXY when the go tool gets a full timeout while getting modules?

[go101]

@oiooj glad to hear there is a choice. Still a concern on the NOTE The GOSUMDB will later default to "sum.golang.org" unconditionally.

[googollee]

@bradfitz Thanks for details. That's reasonable.

[ianlancetaylor]

What will happen with a default GOPROXY setting if the proxy machine is not accessible?

[aofei]

Hi @ianlancetaylor, as I said above, we (Chinese Gophers) will not able to go get any third-party modules if we do nothing about the default GOPROXY at first.

[ianlancetaylor]

What I mean is: what is the exact failure mode? You are suggesting that we don't want to have to tell people to change their GOPROXY value. What happens if GOPROXY is not set at all? And what exactly happens if GOPROXY is set to an inaccessible value? Can we fall back from the inaccessible machine to whatever we do when GOPROXY is not set?

[aofei]

Ah, maybe I'm too entangled in the proxy.golang.org itself. And I think the fallback mechanism you mentioned is ideal. But how? How do we judge when we need to perform this fallback? When getting a full timeout? Well, I don't have any well-thought-out idea right now... :-(

[bradfitz]

How do we judge when we need to perform this fallback? When getting a full timeout?

What's a "full" timeout? We can control the timeout.

Maybe we could do some heuristic check to detect whether proxy.golang.org is firewalled and dynamically control the timeout. Rather than something too short (like 1 second, which penalized slow/lossy/mobile internet connections), or too long (like 30 seconds, which is a bad experience for firewalled users), perhaps at t=1s we start detecting whether proxy.golang.org is firewalled and then fail earlier once we determine that, say, other domains are accessible. Then we could fall back to direct.

[FiloSottile]

As I mentioned on Gerrit, the key and tree of the checksum database are disjoint from who serves them.

There is no need to request trust in a new entity in order to provide the service to a client that can't reach sum.golang.org. The design of the checksum database is such that any untrusted proxy can mirror the main log. Even better, proxies can act as auditors, and ensure the log is run consistently and honestly.

The core advantage of a single checksum database is that it ensures everyone agrees on the same content for each version, and that everyone is seeing the same comprehensive list of versions. You wouldn't want an attacker to be able to single out gosum.io users with different contents, or to show them a version that the author (who monitors sum.golang.org) is not aware of.

Proxying and auditing sum.golang.org makes everyone safer, while splitting off a separate log makes everyone less safe.

It's important that checksum database checks are enforced precisely because they allow third-party proxies or direct fallbacks without losing most of the security properties.

The checksum database is designed as it is to make it easier for users to select alternative proxies without having to make a trust judgement call.

[beoran]

First of all, I'd like to apologise, I was shooting from the hip, no offense intended.

Secondly, stepping back, I realized this issue is not just an issue for China, it is an issue all over the world. Not only governments, but also other large organizations such as corporations may be more interested to protect themselves from outside influences then to allow their developers unimpeded access to the internet. Therefore many people all over the world work in environments that restrict and firewall access to the internet.

The problem is that if there are default GOPROXY and GOSUMDB, this creates a single point of failure. I'm not quite sure what the better solution would be, but technically, I am in favor of a more distributed approach.

[aofei]

@williamzion Really? It's already 9102, still so racist? And, what exactly is the "FREE WORLD" you talked about? A world without China? A world without 1.42 billion population? Should we give you some time for you to snap your fingers? Well, I think I'll just ignore your existence, my "FREE WORLD" friend.

Hi again @beoran, it's okay. In fact, I admit that our country may not be so good. But all of us are working hard for a better future, and I believe that there will be some changes in this kind of things in our country in that future. So, yeah, let's skip this "China" topic and move on. I also realize that it's too late to talk about things like "There should be no default GOPROXY and GOSUMDB". So I think your concerns should be answered directly by the Go team in #25530.

I suggest that we should treat this issue as a broader issue. I mean, what if a proxy endpoint is not blocked but is down? Should we fall back to direct?

And @bradfitz, wow boss, you're always so efficient. I think the "heuristic check" you mentioned may be a viable option. I plan to try it after the holiday. Thank you. :-)

[ALTree]

@williamzion @aofei @beoran Please, let's keep this thread on track. It's okay to discuss technical issues with the caching and checksum servers; and possible solutions that would make the module system usable from mainland China. Discussions about the free world or general remarks about internet censorship do not belong on the Go github issue tracker. Thanks.

[beoran]

OK, technical solutions it is! :)

First of all, I think, go modules and their checksums should be distributable off-line. Now they are zip files, which could be good enough, if we had the tooling to make it easy. Something like go mod zip could create the zip files and a go.sum file of that zip file. Then, you can do go mod install <module zip file> <go.sum file> could then install the module to $GOPATH/pkg/mod . Like that, the modules could be placed on a USB stick or SD card and transmitted like that easily even where there is no internet connectivity at all.

Secondly, it should be easy to set up a module proxy and sumdb yourself without having to install any third party software. So, I'd say we need to include a go proxy together with the go distribution, something like go mod serve, which will serve the contents of the $GOPATH/pkg/mod and and also supports the sumdb protocol for proxies, so people on an intranet can easily share modules after they have installed them somewhere locally using the aforementioned offline methods.

[beoran]

@bradfitz One thing, the heuristic you propose is only useful for proxy.golang.org. Other proxies that clone repositories "on demand" might actually be extremely slow without being down.

[bradfitz]

@bradfitz One thing, the heuristic you propose is only useful for proxy.golang.org. Other proxies that clone repositories "on demand" might actually be extremely slow without being down.

One of the probes could be a HEAD request to your proxy's / root path, which should be fast. If that's up, but a get of a zip is taking awhile, we can assume the proxy is reachable but slow/working.

[beoran]

That sounds reasonable. Please document the probes well, though for those amongst us who must implement a go module proxy, like yours truly.

[bradfitz]

To be clear, I'm just throwing out hypothetical designs. There is no official plan yet.

[aofei]

go help goproxy:

...
A Go module proxy is any web server that can respond to GET requests for
URLs of a specified form. The requests have no query parameters, so even
a site serving from a fixed file system (including a file:/// URL)
can be a module proxy.
...

If we're going to use net/http, then I think we should use GET instead of HEAD to do this check. Not only because of go help goproxy says so, but because it's possible that some proxies doesn't support the HEAD requests.

Moreover, if we're trying to implement this idea, then I think we should use net.Dial instead of net/http ways. Maybe something like:

conn, err := net.Dial("tcp", "proxy.golang.org:443")
if err != nil {
    // Do something here...
}

conn.Close()

This may be more efficient and I think it's enough for us to determine if a proxy can be reached.

[leafduo]

Are there security implications for disabling GOPROXY and GOSUMDB?

[bradfitz]

Moreover, if we're trying to implement this idea, then I think we should use net.Dial instead of net/http ways. Maybe something like:

I think we'd want to get past the (valid) TLS handshake, though.

[aofei]

What I'm trying to say is that we only need to check if we can reach the proxy machine through the provided address, not whether it can handle GOPROXY requests (or just any HTTP requests). But of course, a more comprehensive check is better.

So let's say we will send a GET request. But to where? The proxy's root path may be an option. But some proxies may use their root path for other purposes, such as showing some guidance, those may slow down our checks. So I think we should take the initiative to trigger some errors, such as 404 Not Found (in fact, no matter what the error is). I think this way may improve TTFB more or less. It's just another unthinking view of mine.

client := &http.Client{
    Timeout: 3 * time.Second,
}

res, err := client.Get("https://proxy.golang.org/foo/bar") // 400 Bad Request
if err != nil {
    // Do something here...
}

// No body is needed, headers is more than enough.
// I think this should be almost the same as sending a HEAD request.
res.Body.Close()

[aofei]

I just sorted out my thoughts and wrote a more substantial snippet to better express them.

Code:

const (
    GOPROXY                        = "https://proxy.golang.org"
    GOPROXY_REACHABILITY_CHECK_URL = GOPROXY + "/foo/bar"
)

req, err := http.NewRequest(http.MethodGet, GOPROXY+"/golang.org/x/text/@latest", nil)
if err != nil {
    return err
}

ctx, cancel := context.WithCancel(context.Background())
req = req.WithContext(ctx)

isProxyRequestReturned := false

go func() { // Check in parallel whether GOPROXY can be reached
    time.Sleep(time.Second) // I think this may be what @bradfitz said
    if isProxyRequestReturned { // Minimize unnecessary checks
        return
    }

    reachabilityCheckClient := &http.Client{
        Timeout: 3 * time.Second,
    }

    res, err := reachabilityCheckClient.Get(GOPROXY_REACHABILITY_CHECK_URL)
    if err == nil { // Everything is fine
        res.Body.Close() // No body is needed, headers is more than enough
        return
    }

    if !err.(*url.Error).Timeout() { // Everything is fine too, even if there is an error here
        return
    }

    cancel() // Cancel the proxy request
    // Fall back and get the module directly from the origin
}()

res, err := http.DefaultClient.Do(req) // Do the proxy request
isProxyRequestReturned = true
if err != nil {
    return err
}
defer res.Body.Close()

// Get the module from the proxy

[ankushchadha]

hey, not to derail the conversation or anything but there is an alternative to Go's proxy called GoCenter which is available from within CN.

[aofei]

Thank you @ankushchadha, but we're not looking for a GOPROXY service here. In fact, we've developed a business-supported GOPROXY service ourselves, which is avaliable at goproxy.cn. What we really need is to find a way for us to use Go modules without having to modify GOPROXY beforehand.

[ankushchadha]

Thanks, @aofei. The proposal here is to set https://gocenter.io as one of the default GOPROXY since it's accessible to the golang developers all over the world.

[aofei]

@ankushchadha

If we can add other defaults for GOPROXY, then I'd of course strongly recommend making the goproxy.cn one of them. After all, it's as fast as the proxy.golang.org now and it can be accessed in the mainland of China (tested by https://tools.ipip.net/newping.php).

However, I think we can't do this, at least I think the Go team will never agree to do this. In fact, a lot of Gophers may not agree, think about how hard it is for us to accept the proxy.golang.org.

According to the email sent by the Go team to @astaxie, the Go team can now offer two solutions for Chinese Gophers:

1. Using go env -w to configure the go command before go get:

go env -w GOPROXY=direct
go env -w GOSUMDB=sum.golang.google.cn # Yeah, they're willing to build a Chinese variant of the sum.golang.org

1. Making the Go distribution downloaded from the golang.google.cn/dl different from the golang.org/dl. As I discussed with @astaxie, in this solution, we can make the go command default like this:

export GOPROXY=https://goproxy.cn
export GOSUMDB=sum.golang.org # Since the goproxy.cn already supports proxying checksum databases, so no need to build the sum.golang.google.cn

For the first solution, I don't think it solves the problem of not being able to access the proxy.golang.org in the mainland of China. And just a Chinese variant of the sum.golang.org is not that helpful, IMO.

For the last solution, I don't know how many people know the golang.google.cn, at least I rarely hear people around me mention it. And I believe most people are using something like Homebrew or Snapd to install and manage the Go distribution. So, I think this solution is not that perfect too.

I think the best solution at the moment is the "heuristic check and fallback" mentioned by @bradfitz and @ianlancetaylor.

However, I'm still grateful to the Go team for being concerned about this issue, which means a lot to us. 🙏😊

[FiloSottile]

For the first solution, I don't think it solves the problem of not being able to access the proxy.golang.org in the mainland of China. And just a Chinese variant of the sum.golang.org is not that helpful, IMO.

Can you elaborate on this? GOPROXY=direct should make it unnecessary to access proxy.golang.org, while sum.golang.google.cn will provide all the sumdb security guarantees. It should be at least as functioning as Go has been so far, and more secure.

[aofei]

I know that GOPROXY=direct can prevents the go command from accessing the default proxy.golang.org. And I also know the importance of an official sumdb that can be accessed in China without using any proxies. What I'm trying to do is to highlight the current issue, that is, we're eager for the Go team to announce an OFFICIAL Go module proxy that can be accessed in China. You may not know, the Go module proxy feature is extremely important to our Chinese Gophers. I'm pretty sure this feature is much more acceptable in China than in all other countries in the world, because if we don't use this feature, then we'll need a lot of other tools for getting the usual packages like the famous golang.org/x/text.

But, of course, I also know that Go team can't build a proxy.golang.google.cn because of the no UGC policy. So, I was thinking, if Google can't build it, then why can't Go team work with a trusted Chinese company to build one and then officially announce it? Is Go official stuff can only run by Google? I don't think so. So, building another official Go module proxy running in China and using a domain name that has been ICP filed in the MIIT of China, by doing so, this Go module proxy will be fully legal in China and can serve Chinese Gophers without any interference. If the Go team is willing to do this, then I think I can help you to promote this. In fact, I have donated the goproxy.cn to a Chinese company (Qiniu Cloud), and this domain name has been ICP filed by them, not me (see the picture below), so it's now fully legal in China, and it's quite fast now (CDNed). I'm currently working with Qiniu Cloud's employees to make the goproxy.cn as stable as the proxy.golang.org. Our plan is to officially advertise it on the day that Go 1.13 is released. So I think the goproxy.cn is eligible to be the official Go module proxy for Go in China. If we can do this, then I'll be happy to help the Go team and Qiniu Cloud to achieve this cooperation. And, of course, I don't think they need to get any money from the Go team, so this cooperation will be more easier. 😊

[andybons]

Thanks everyone. We do care deeply about this issue and are doing everything we can to ensure everyone has a good experience with Go 1.13 and beyond.

As @aofei mentioned, we are able to to have Go’s sumdb available in mainland China and are currently working on this. However, the Go project is currently unable to offer proxy.golang.org inside mainland China. Fortunately, many similar services are already being hosted by Go community members in mainland China.

The Go project will not endorse any 3rd party service as “official”. The tooling is designed to make it easy to choose any of the many compatible proxies. In our blog post announcing the new changes, we will provide clear instructions on how to configure your environment to any proxy.

Thanks again for you continued patience on this issue.

Andy for the Go team

[aofei]

Understood. Then I think there is only one thing we can do on our side now: after the release of Go 1.13, we spread the GOPROXY configuration instructions as widely as possible in our Go communities, forums, blogs, and books etc. That's all.

And, of course, @andybons, thank you very much for your response and Go team's efforts. We really appreciate the Go team for caring about this issue. 🙏

[ipfans]

I think we can close this issue when Go 1.13 release.

[andybons]

Hi folks,
Go 1.13 has been released and https://sum.golang.google.cn is now generally available. If you’re using Go 1.13, you can set your GOSUMDB environment variable using go env -w GOSUMDB=sum.golang.google.cn. If proxy.golang.org is not available for you, you must set GOPROXY=direct or to a service that is available (some are mentioned above).

Thanks everyone for your patience on this. Closing as there is not much more we can do. If there are any issues with the sumdb in mainland China please feel free to file a separate issue.

Andy for the Go Team