Volta: Can volta using proxy?

[YucongDing]

I come from china, due to the firewall, when using 'volta install ' command, the net speed is so slow.
I configured socks proxy to speedup the http/https/git, the installer script works, but 'volta install ' command faild.
Can you add this feature to volta?

[charlespierce]

Hi @YucongDing, I think we may be able to set that up, though it looks like it will take a little bit of work. Out of curiosity, how are you configuring your proxy? Using the HTTP_PROXY and HTTPS_PROXY environment variables?

[YucongDing]

Hi @YucongDing, I think we may be able to set that up, though it looks like it will take a little bit of work. Out of curiosity, how are you configuring your proxy? Using the HTTP_PROXY and HTTPS_PROXY environment variables?

Yes, using HTTP_PROXY & HTTPS_PROXY & ALL_PROXY env variables.
By the way, how does volta fetch the nodejs installer? If I configured the proxy, it should download through the ip:port or I need to add the domain to my rule list?

[charlespierce]

Good to know. It looks like the library we’re using for HTTP requests doesn’t support Proxies, which is why it’s not working for you. We can do some investigation to see if there’s a different library that would support it.

As for how we make the requests, they are all with domain names (nodejs.org, yarnpkg.com, github.com and npmjs.com, I believe are the ones we use by default)

[YucongDing]

Thank you very much. Hope Volta can support it as soon as possible. It works smoothly and nice.👍

[charlespierce]

Glad to hear it’s working well for you, despite the proxy issues!

[codereflection]

+1 for proxy support. Volta looks promising for our build chain.

[gregjopa]

I'm also looking for proxy support with Volta. My use case is the public npm registry is restricted and can only be accessed through a proxy. Developers have this in their ~/.npmrc file which doesn't seem to work with Volta:

registry=https://registry.npmjs.org/
proxy=http://proxy.giantcorp.com:8080/
https_proxy=http://proxy.giantcorp.com:8080/

@charlespierce I'm pretty new to Rust but I'd like to help with this feature request. Any recommendations on where to start?

[charlespierce]

Hi @gregjopa, we would definitely be happy to take a contribution to resolve this! I did some preliminary investigation and I _think_ the most straightforward way would be to use an HTTP library that natively supports the HTTP_PROXY and HTTPS_PROXY environment variables. The only one that seems to fit is reqwest (ironically, we used to use reqwest, but switched off for compile-time and binary size reasons).

So I believe the high-level approach would be to switch from attohttpc to reqwest everywhere (inside crates/archive and crates/volta-core), then update the code that calls attohttpc::get to use reqwest::get and adjust to the small differences in API between the two libraries. IIRC from when we switched away, the two APIs are mostly similar to one another, but slightly different so it takes a little bit of work at each call site to make them line up.

[gregjopa]

Thanks for the direction @charlespierce! It's been fun hacking on this. I was able to successfully replace attohttpc with reqwest. I feel bad reverting the perf gains from #733. The latest version of reqwest is split up into optional features so hopefully that helps minimize the impact of this larger dependency.

I have the changes pushed here and will create a PR soon. With this set up, I can successfully use the proxy by setting an environment variable:

export https_proxy=http://proxy.giantcorp.com:8080/
# successfully uses the proxy
volta install npm@latest

So this solution technically unblocks folks who need to use a proxy. But setting an environment variable for the proxy is inconvenient. Ideally we would read it from the npm config since it's typically defined in the ~/.npmrc file. What do you think? I wanted to check with you on this before drafting a PR.

A potential solution is to have volta get those proxy values with npm config ls --json and then pass those values into reqwest using the client builder. This need does seem similar to the private registry feature 🤔.

[charlespierce]

@gregjopa Since the proxy may be needed for fetching Node (which could occur without npm even being available on the machine to run npm config), I think the environment variable is the best general-purpose solution.

For situations where you only need to proxy registry.npmjs.org and use the setting in ~/.npmrc, I think we may be able to handle it slightly differently: We are in the process of revamping our global package installs to better leverage the existing npm behaviors. Once that is complete, we can swap the installation of npm and yarn to use the same behavior, which would then respect ~/.npmrc in those cases as a consequence.

So for this issue, I think the swap back to reqwest for the additional features is the solution. Thankfully, most of the perf gains in the original swap were in build time, so while losing them isn't ideal, it's a fairly easy tradeoff to make.

[gregjopa]

Since the proxy may be needed for fetching Node (which could occur without npm even being available on the machine to run npm config), I think the environment variable is the best general-purpose solution.

@charlespierce that sounds good to me. Thanks for providing more context. The environment variable approach also seems a lot simpler. I went ahead and opened up PR #809 for adding reqwest.