Clasp: Support proxy network

[hanguokai]

To use proxy network, like http(s) or socks5, I use http_proxy and https_proxy environment variable that are supported by many tools. Currently, clasp can't work in proxy environment, for example clasp login and clasp clone.

[grant]

I'm not sure what this would change would entail.
Can you be more specific about repro steps, some links, and a more detailed proposal on what commands this would be needed for?

It looks like this request would want to add something like:
--proxy "socks5://127.0.0.1/"
to every command.

Similar issue: https://github.com/rg3/youtube-dl/issues/402#issuecomment-213744197

[hanguokai]

Yes, clasp need to add --proxy option, like youtube-dl, curl, gcloud, npm, git, etc.

http_proxy environment variable is the same meaning as --proxy, but detect by command if no --proxy option specified.

I am using socks5 protocol, there are others like http, https.

[grant]

@hanguokai Can you add proxy rule to the URL googleapis.com so that every googleapis.com request is proxied?

[grant]

Pull requests are welcome.
I'm closing this since I don't see much demand for this feature. Will reopen if there's demand or more discussion.

[HeavensBlade]

I would also like to see a proxy option.
Since I'm developing behind a corporate proxy I always need to activate my smartphone hotspot.

[nepsilon]

I'm using clasp from a censored internet where all things Google are blocked and need the proxy feature. Using proxy environment variables doesn't work.

[grant]

Why doesn't using proxy environment variables not work? Every request goes through http://googleapis.com/.

Does every application you use require a proxy feature?

[hanguokai]

Aha, please look at the same issue for firebase command line tool https://github.com/firebase/firebase-tools/issues/155 . Nearly 2 years and 81 comments! Under certain conditions, people can't connect internet directly. That's it.

[nepsilon]

@grant Yes, all blocked domains and IP requires a VPN or proxy. It's very standard in places like China.

[grant]

Unfortunately G Suite and many of Google's services are blocked in China. I can't solve that issue. I'm guessing the Firebase tool would probably prefer to not be blocked too to solve that linked issue.

I don't understand why environment variables don't work, do you really need every single tool you use to implement a custom --proxy option?

Underneath it all, we use googleapis. It supports a proxy (HTTP_PROXY, HTTPS_PROXY). Does that not work?

To implement this feature, I'd need:

• A reasonable number of people to request and use the proxy feature (👍 the issue)
• A design doc for this feature.
• Ideally a pull request with an initial implementation.

To be honest, I have no use for this feature and still don't understand why you can't use a proxy elsewhere in the tech stack.

[valen22br]

Hello, I'm running through the same problem. I tried setting up my user variables with proxy settings, and I'm still receiving the error below:
Error retrieving access token: Error: connect ETIMEDOUT 172.217.18.170:443

Is there a workaround for this issue?

[grant]

I'm in favor of adding a proxy option, but to be honest, I don't know how to run a proxy and I don't know why env variables don't work like in the googleapis README:

export HTTP_PROXY=http://localhost:3128
export http_proxy=http://localhost:3128
export https_proxy=http://localhost:3128
export HTTPS_PROXY=http://localhost:3128

I did a bit more research.

• If someone wants to try this, add a proxy variable to this line of code: https://github.com/google/clasp/blob/master/src/auth.ts#L40
  Then rebuild clasp and test it out, and it works, that'd be awesome. Then I can update clasp to include this proxy param.
• If someone (@hanguokai, @nepsilon) describes exact steps on how to download a proxy, see the error in clasp using some tool, I'll try to reproduce the problem and create a solution.
• If you need help with a pull request, or building clasp, LMK too.

This is the most upvoted issue, and it looks like user variables doesn't work, so I'm in favor of support this feature in clasp, but I'll need some help for this one.

[valen22br]

Thanks for the reply!
I'll take a look at it. I work for a big company that enforces us to use a
proxy. That's why for us, having a way to support proxy is extremely
important. I don't need a specific solution as the --proxy option, but a
watever solution that works behind a proxy.

Best,
Luis
Gustavo

On Mon, Aug 20, 2018 at 6:12 PM Grant Timmerman [e-mail]
wrote:

I'm in favor of adding a proxy option, but to be honest, I don't know how
to run a proxy and I don't know why env variables don't work like in the
googleapis README:

export HTTP_PROXY=http://localhost:3128
export http_proxy=http://localhost:3128
export https_proxy=http://localhost:3128
export HTTPS_PROXY=http://localhost:3128

I did a bit more research.

• If someone wants to try this, add a proxy variable to this line of
  code: https://github.com/google/clasp/blob/master/src/auth.ts#L40
  Then rebuild clasp and test it out, and it works, that'd be awesome.
  Then I can update clasp to include this proxy param.
• If someone (@hanguokai https://github.com/hanguokai, @nepsilon
  https://github.com/nepsilon) describes exact steps on how to
  download a proxy, see the error in clasp using some tool, I'll try to
  reproduce the problem and create a solution.
• If you need help with a pull request, or building clasp, LMK too.

This is the most upvoted issue, and it looks like user variables doesn't
work, so I'm in favor of support this feature in clasp, but I'll need
some help for this one.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/google/clasp/issues/8#issuecomment-414480372, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AZEcdMhPAkxXsw_MGk1jHS4ONV_NjIlWks5uSzRkgaJpZM4Rfrqw
.

[valen22br]

Can you give me directions of how to build clasp locally for testing
purpose? Them if works, you can help me with the pull request.
Best,
Luis Gustavo

On Tue, Aug 21, 2018 at 8:10 AM Gustavo Valensuela [e-mail]
wrote:

Thanks for the reply!
I'll take a look at it. I work for a big company that enforces us to use a
proxy. That's why for us, having a way to support proxy is extremely
important. I don't need a specific solution as the --proxy option, but a
watever solution that works behind a proxy.

Best,
Luis
Gustavo

On Mon, Aug 20, 2018 at 6:12 PM Grant Timmerman [e-mail]
wrote:

I'm in favor of adding a proxy option, but to be honest, I don't know how
to run a proxy and I don't know why env variables don't work like in the
googleapis README:

export HTTP_PROXY=http://localhost:3128
export http_proxy=http://localhost:3128
export https_proxy=http://localhost:3128
export HTTPS_PROXY=http://localhost:3128

I did a bit more research.

• If someone wants to try this, add a proxy variable to this line of
  code: https://github.com/google/clasp/blob/master/src/auth.ts#L40
  Then rebuild clasp and test it out, and it works, that'd be awesome.
  Then I can update clasp to include this proxy param.
• If someone (@hanguokai https://github.com/hanguokai, @nepsilon
  https://github.com/nepsilon) describes exact steps on how to
  download a proxy, see the error in clasp using some tool, I'll try to
  reproduce the problem and create a solution.
• If you need help with a pull request, or building clasp, LMK too.

This is the most upvoted issue, and it looks like user variables doesn't
work, so I'm in favor of support this feature in clasp, but I'll need
some help for this one.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/google/clasp/issues/8#issuecomment-414480372, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AZEcdMhPAkxXsw_MGk1jHS4ONV_NjIlWks5uSzRkgaJpZM4Rfrqw
.

[grant]

@valen22br Follow these instructions https://github.com/google/clasp#develop
Let me know if you can't build clasp.

[valen22br]

@grant the instructions worked for me. I will start playing with the proxy settings and will let you know the results.

[hanguokai]

I did some research about google-api-nodejs-client. It says it support proxy.

But in fact, googleapis use axios as it's http client library that doesn't support socks proxies. So export https_proxy=socks5://127.0.0.1:1080 or setting (socks) proxy option in code doesn't work.

In theory, googleapis should support http/https proxy like export http_proxy=http://localhost:3128. I didn't research whether there is a problem with this method.

PS: I know gcloud command support proxy including socks proxy. And firebase command had problem for supporting proxy, but recently it seems support http proxy(not socks proxy) in 4.1.2.

[grant]

I don't know the difference between socks proxies and non-socks proxies. Will a non-socks proxy work?

[valen22br]

I don't know either.
From my tests I got the following results:

1. No HTTP_PROXY or HTTPS_PROXY environment setup. In this scenario, the error was: Error retrieving access token: Error: connect ETIMEDOUT 172.217.18.170:443
2. With HTTP_PROXY or HTTPS_PROXY environment setup. Now I got another error: Error retrieving access token: Error: write EPROTO 101057795:error:140770FC:SSL
   routines:SSL23_GET_SERVER_HELLO:unknown protocol:openssl\ssl\s23_clnt.c:827:

It seems that we are going from a proxy error to some OpenSSL incompatibility. Do you have such OpenSSL error message before?

[valen22br]

Hi @grant , the call to the method client.getToken(authCode) at auth.ts was not able to provide a return. Do you know how to debug that method?

[hanguokai]

I don't know why env variables don't work like in the googleapis README - grant
Will a non-socks proxy work? - grant

I wrote a sample to test env variables for googleapis directly. I found googleapis itself has issue for supporting http(non-socks) proxy. Someone filed an issue here. Axios, googleapis use it as http client library, also has issue about http proxy.

So when googleapis fixed http proxy issue, let's review this issue again in clasp. Currently, I use VPN as a workaround method.

[valen22br]

@hanguokai , good point in your research. My tests also revealed that apparently, the environment variables with proxy settings are working. The problem is when the clasp tries to use the token got from the authentication process to use it against the google API. In this step, it seems that someone, (Google API as you pointed) is messing around with https requests.
Here in our company, we have a single proxy that can handle both HTTP and https requests without distinction.

I'll take a look at the links you provided. Can you explain your workaround using the VPN behind the corporate proxy?

[grant]

(I moved some off-topic comments to a separate thread to keep this about a proxy FR. See https://github.com/google/clasp/issues/297#issuecomment-415641264)

[aakasaka]

It seems googleapis fixed the problem with proxy.
Could you please update the packages (googleapis and google-auth-library) to latest version?

[grant]

I'm in the process of upgrading the packages.

[grant]

Can you install the latest clasp, 1.6.1? LMK if it's fixed with the new googleapis module.
@hanguokai @valen22br @aakasaka @HeavensBlade

[aakasaka]

I was able to clasp login and clasp create with the latest version, thank you !
FYI, I still should replace the code inOnline() with true coz is-online module doesn't work with proxy. (mentioned in #336 )

[grant]

OK, thanks. I'll remove the isOnline feature with #332 since it's not super useful and blocks proxy users. Sorry @arjun-rao!

[valen22br]

Tomorrow I will run some tests with a proxy configuration.
Thanks for the fix!

[grant]

@aakasaka I removed the check to isOnline if HTTP_PROXY/HTTPS_PROXY is present with 1.6.3.
Let me know if a proxy works out of the box now, so I can close this issue.

[hanguokai]

I simply tested clasp 1.6.3 with login, logout, status, clone, pull and push commands. It seems Ok now for http_proxy/https_proxy. Thanks for updating!

[grant]

👍 Thanks for testing.
Closing this top clasp FR now!