Got: Proxy support

[sindresorhus]

_Previous issue https://github.com/sindresorhus/got/issues/7. I couldn't care less about this, but it's useful to have a place to refer people that open issues elsewhere._

A good PR would be accepted, but won't happen from my part.

I would prefer for the solution to be an external module got can just depend on as I don't want to have to maintain it. Maybe https://github.com/mikeal/tunnel-agent.

I strongly believe this is something that should be a part of Node.js and not every module doing HTTP. +1 this issue if you agree: https://github.com/nodejs/node/issues/8381

[kevva]

I created a module for this, but didn't get the tests passing IIRC. I'll try finishing it asap.

[Flet]

I've run into two great packages now that use got, but with no proxy support I can't use them :(

@kevva is there anything I can do to help with your module?

[floatdrop]

@Flet caw it is, I guess.

[kevva]

Yup, haven't been able to check if it's working correctly though.

[alexpaluzzi]

Is not working (as mentioned in other issue).

Several companies don't have an actual "proxy" but rather, an HTTPS inspection point somewhere down the line. (Like, in my case, "Check Point Firewall"). This requires some self-signed certs to work and it definitely causes some issues by default with npm and other package managers.

npm is easily solved with "strict-ssl=false". Curl has a flag for it as well.

download worked great before whatever tweaks were made a few weeks ago. @kevva's fix probably did help those that have "real" proxies.

I'm not a network admin, so I'm probably messing some of this lingo up. I did speak with my network guys here, though, and after a few days of research, this is where we landed. Hopefully this helps somehow.

[kevva]

If you look at https://github.com/koichik/node-tunnel, do you see something there that might fix it?

[alexpaluzzi]

I immediately had the same thought after I saw your caw. Unfortunately, I don't see anything there. If I have some time, I'm going to look at npm and what exactly strict-ssl is doing. Even without an actual proxy, that config is the difference between working and not working.

[alexpaluzzi]

@sindresorhus Tears of joy here.

http doesn't have a way to handle these certificates. request does, of course (with agentOptions and strictSsl). I found a couple ancient issues from 2013 asking for a strict-ssl option. I don't see one yet.

The workaround for http is setting this environment variable (not my favorite solution, but it's the only one I've found):
process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";

I think your tunnel-agent is working good for traditional proxies. I wager this can be closed now. I'm certain there were people having my exact issue here, so hopefully this will help.

[stevenvachon]

Progress?

[sindresorhus]

@stevenvachon → https://github.com/sindresorhus/got/pull/159#issuecomment-173184834

[kevva]

I'm not sure why this wouldn't work:

got('http://github.com', {
    agent: caw()
});

And to address @alexpaluzzi issue, you don't have to set the NODE_TLS_REJECT_UNAUTHORIZED env flag. Just pass rejectUnauthorized: false to got.

[sindresorhus]

New thread about adding proxy support to Node.js core: https://github.com/nodejs/node/issues/8381 (Vote/comment please)

[tjwebb]

fwiw, the node yeoman generator is also experiencing this issue: https://github.com/trailsjs/generator-node/issues/1.

Is there a patch available?

[chocolateboy]

Until this is fixed in Node, a workaround for Linux/BSD/macOS is to use proxychains-ng.

You'll need your proxy server's IP address and port:

$ echo $http_proxy
http://proxy.example.com:8080/

$ ping proxy.example.com
PING proxy.example.com (192.0.2.42) 56(84) bytes of data.

- which should be added to the end of /etc/proxychains.conf (comment out the default rule) e.g.:

# socks4 127.0.0.1 9050
http 192.0.2.42 8080

Then you're all set:

$ proxychains -q npm-check    
Checking for unused packages...

$ alias npm-name="proxychains -q npm-name"
$ npm-name proxychains-ng-rocks
✔ proxychains-ng-rocks is available

[stevenvachon]

What do you guys think of https://github.com/stevenvachon/auto-tunnel before I publish?

[kevva]

Is there any differences from https://github.com/kevva/caw? Could really use a PR if you notice anything that can be improved upon.

[paambaati]

@kevva I can see that @stevenvachon's auto-tunnel is very similar to caw.

[silverwind]

auto-tunnel looks promising. From a quick look, it seems to supports NO_PROXY which I think caw does not. I'd suggest to clean up the code and push a 1.0.0 and open a PR here.

[rightaway]

What's the recommended way to use got with HTTPS/SOCKS5 proxies?

[stevenvachon]

auto-tunnel wasn't released because:

• it is so similar to caw that I decided to instead convince @kevva to conform to some of its features. He didn't respond to everything.
• https://github.com/TooTallNate/node-proxy-agent/issues/11

[wildone]

NPM works fine with --https-proxy and --proxy, so its only logical that all modules built on top of it support this basic mechanism. I had a situation using npm with proxy where another module was using your module to build its depenedecies... well that did not work at all. I ended up fixing the issue with a workaround.

So maybe this needs a another look as "A nicer interface to the built-in http module" (from your readme) is a big call if basic http modules functions are not replicated... thats just my 2cents

[sindresorhus]

NPM works fine with --https-proxy and --proxy, so its only logical that all modules built on top of it support this basic mechanism.

Got is not built on top of npm.

So maybe this needs a another look as "A nicer interface to the built-in http module" (from your readme) is a big call if basic http modules functions are not replicated...

The Node.js core http API doesn't support proxies built-in either.

[sindresorhus]

Closing this as I don't see us adding native proxy support. It's just too complicated and would bloat Got. It can already be done by the user by following this recipe.