Bazel: Could not connect to mirror.bazel.server with proxy in mainland China since a few days ago

[YangleiZouIntel]

My bazel is not able to connect to mirror.bazel.server with company proxy from mainland China since a few days ago. The server seems to refuse connection from a certain range of ip. A private VPN might work but it is not convenient for me. Hope someone can help. Thank you so much.

[oquenchil]

@coeuvre do you think you can help here?

[coeuvre]

Can you provide more context:

• Output of bazel info release
• Steps to setup the proxy
• Error messages related to the connection issues printed by bazel

[YangleiZouIntel]

I am using Ubuntu 16.04
Bazelisk version: v1.1.0
export USE_BAZEL_VERSION=~/Downloads/bazel-3.4.1-linux-x86_64

Output of bazelisk info release
Starting local Bazel server and connecting to it...
release 3.4.1

Steps to setup the proxy

in ~/.bashrc
export http_proxy=http://my_company_proxy:port
export HTTP_PROXY=http://my_company_proxy:port
export https_proxy=https://my_company_proxy:port
export HTTPS_PROXY=https://my_company_proxy:port

sudo visudo and added the following lines
Defaults env_keep+="http_proxy"
Defaults env_keep+="https_proxy"
Defaults env_keep+="HTTP_PROXY"
Defaults env_keep+="HTTPS_PROXY"

Error messages related to the connection issues printed by bazel

WARNING: Download from https://mirror.bazel.build/openjdk/azul-zulu11.37.17-ca-jdk11.0.6/zulu11.37.17-ca-jdk11.0.6-linux_x64.tar.gz failed: class javax.net.ssl.SSLException Read timed out
ERROR: An error occurred during the fetch of repository 'remotejdk11_linux':
java.io.IOException: Error downloading [https://mirror.bazel.build/openjdk/azul-zulu11.37.17-ca-jdk11.0.6/zulu11.37.17-ca-jdk11.0.6-linux_x64.tar.gz] to /home/yangleiz/.cache/bazel/_bazel_yangleiz/941f2d8f0f6033e4d146732bd806074f/external/remotejdk11_linux/zulu11.37.17-ca-jdk11.0.6-linux_x64.tar.gz: Read timed out
ERROR: /home/yangleiz/.cache/bazel/_bazel_yangleiz/941f2d8f0f6033e4d146732bd806074f/external/bazel_tools/tools/jdk/BUILD:497:6: @bazel_tools//tools/jdk:remote_jdk11 depends on @remotejdk11_linux//:jdk in repository @remotejdk11_linux which failed to fetch. no such package '@remotejdk11_linux//': java.io.IOException: Error downloading [https://mirror.bazel.build/openjdk/azul-zulu11.37.17-ca-jdk11.0.6/zulu11.37.17-ca-jdk11.0.6-linux_x64.tar.gz] to /home/yangleiz/.cache/bazel/_bazel_yangleiz/941f2d8f0f6033e4d146732bd806074f/external/remotejdk11_linux/zulu11.37.17-ca-jdk11.0.6-linux_x64.tar.gz: Read timed out
ERROR: Analysis of target '//tools/test/CoverageOutputGenerator/java/com/google/devtools/coverageoutputgenerator:coverage_output_generator_zip' failed; build aborted: Analysis failed
INFO: Elapsed time: 174.553s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (27 packages loaded, 290 targets configured)

Update
I am able to connect to mirror.bazel.build from my cell phone. I used it as a hotspot and downloaded the required files as a walkaround. But using company proxy still timed out, tested from both Beijing and Shanghai. Sometimes I can get a few kb data, but most of the time it is just timed out. Not sure if it is bazel server issue or company proxy server issue.

[coeuvre]

Would you please try download the file using curl with your company proxy and share the outputs:

$ curl -o /dev/null -D - https://mirror.bazel.build/openjdk/azul-zulu11.37.17-ca-jdk11.0.6/zulu11.37.17-ca-jdk11.0.6-linux_x64.tar.gz

Please make sure execute above command at the same network condition as you ran bazel. e.g. immediately run curl command after bazel build failed.

If it works, it's probably an issue with Bazel.
If it fails in the same way as Bazel, let's see if it works with http without encryption:

$ curl -o /dev/null -D - http://mirror.bazel.build/openjdk/azul-zulu11.37.17-ca-jdk11.0.6/zulu11.37.17-ca-jdk11.0.6-linux_x64.tar.gz

If it works now, there might be an issue with the proxy not correctly supporting HTTPS.

It might be interesting to see if the file can be downloaded directly (bypassing the Load Balancer):

$ curl -o /dev/null -D - https://storage.googleapis.com/bazel-mirror/openjdk/azul-zulu11.37.17-ca-jdk11.0.6/zulu11.37.17-ca-jdk11.0.6-linux_x64.tar.gz

(repeat with http:// if it fails)

[YangleiZouIntel]

Below please see my test result. Using http gives an 301 Moved Permanently error.
Thank you so much.

$ curl -o /dev/null -D - https://mirror.bazel.build/openjdk/azul-zulu11.37.17-ca-jdk11.0.6/zulu11.37.17-ca-jdk11.0.6-linux_x64.tar.gz

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0HTTP/1.1 200 Connection established

0 0 0 0 0 0 0 0 --:--:-- 0:03:54 --:--:-- 0
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to mirror.bazel.build:443

curl -o /dev/null -D - http://mirror.bazel.build/openjdk/azul-zulu11.37.17-ca-jdk11.0.6/zulu11.37.17-ca-jdk11.0.6-linux_x64.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0HTTP/1.1 301 Moved Permanently
Server: nginx/1.16.1
Date: Thu, 30 Jul 2020 02:02:23 GMT
Content-Type: text/html
Content-Length: 169
Location: https://mirror.bazel.build/openjdk/azul-zulu11.37.17-ca-jdk11.0.6/zulu11.37.17-ca-jdk11.0.6-linux_x64.tar.gz
Via: 1.1 google, 1.1 shzdmzpr03, 1.1 shzintpr03
Proxy-Connection: Keep-Alive
Connection: Keep-Alive

100 169 100 169 0 0 292 0 --:--:-- --:--:-- --:--:-- 292

curl -o /dev/null -D - https://storage.googleapis.com/bazel-mirror/openjdk/azul-zulu11.37.17-ca-jdk11.0.6/zulu11.37.17-ca-jdk11.0.6-linux_x64.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0HTTP/1.1 200 Connection established

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0HTTP/1.1 200 OK
X-GUploader-UploadID: AAANsUmUVbjADaQcJfxotXpfuB9dmMVs0veLSk_RHFJlYK-6i3U3woi1MgmFl7kv2NokLmIndOtSudfSZhnQd4UXK7GEgR7Pdg
Expires: Thu, 30 Jul 2020 02:52:25 GMT
Date: Thu, 30 Jul 2020 01:52:25 GMT
Cache-Control: public, max-age=3600
Last-Modified: Fri, 14 Feb 2020 12:43:39 GMT
ETag: "93f4cca7c9e85af9971503949b759e2f"
x-goog-generation: 1581684219045238
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 200144536
Content-Type: application/gzip
x-goog-hash: crc32c=8001fg==
x-goog-hash: md5=k/TMp8noWvmXFQOUm3WeLw==
x-goog-storage-class: STANDARD
Accept-Ranges: bytes
Content-Length: 200144536
Server: UploadServer
Alt-Svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

100 190M 100 190M 0 0 6361k 0 0:00:30 0:00:30 --:--:-- 8101k

[philwo]

The only difference in HTTP response headers between the Google Cloud Load Balancer (GCLB) and Google Cloud Storage (GCS) is the Alt-Svc part:

GCLB: alt-svc: clear

GCS: alt-svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

Doesn't look likely to cause trouble to the proxy server.

The SSL config of GCLB and GCS is also pretty similar according to a check via ssllabs.com, so that doesn't look like it might cause trouble either.

What I could imagine is that some kind of firewall or behavior in the proxy does not like the IP address of mirror.bazel.build and blocks or interrupts HTTPS traffic with it, because it is not well-known and just a normal static IPv4 assigned to us via Google Cloud. No one except the Bazel project uses this IP address.

However storage.googleapis.com is used by many many people and websites and might have special exception rules to allow traffic in any case (as otherwise a lot of stuff on the internet would break if you block it). 🤔

If possible, I would recommend to ask the people who manage your corporate proxy whether they can reproduce and whether they see any issues in their config or logs.

At this point, I don't have a good idea how we could help on our side, but I will think about it for some more time.

[YangleiZouIntel]

The only difference in HTTP response headers between the Google Cloud Load Balancer (GCLB) and Google Cloud Storage (GCS) is the Alt-Svc part:

GCLB: alt-svc: clear

GCS: alt-svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

Doesn't look likely to cause trouble to the proxy server.

The SSL config of GCLB and GCS is also pretty similar according to a check via ssllabs.com, so that doesn't look like it might cause trouble either.

What I could imagine is that some kind of firewall or behavior in the proxy does not like the IP address of mirror.bazel.build and blocks or interrupts HTTPS traffic with it, because it is not well-known and just a normal static IPv4 assigned to us via Google Cloud. No one except the Bazel project uses this IP address.

However storage.googleapis.com is used by many many people and websites and might have special exception rules to allow traffic in any case (as otherwise a lot of stuff on the internet would break if you block it).

If possible, I would recommend to ask the people who manage your corporate proxy whether they can reproduce and whether they see any issues in their config or logs.

At this point, I don't have a good idea how we could help on our side, but I will think about it for some more time.

Thank you. I am contacting people managing the proxy. It seems certain websites are blocked in China and the company proxy only allows access to a few well-known websites like github.com and google.com. I will check if they can unblock mirror.bazel.build.