Ngrok: Allow ngrok to work behind an HTTP Proxy

[gauthamses]

I have just downloaded ngrok and try to run it behind a corporate proxy. The console log does not show any url that is forwarded to the local service. Could anyone please confirm if this works behind a proxy?

Console output :

ngrok                                                                                                                                 

Tunnel Status                 connecting                                                                                                              
Version                       0.14/                                                                                                                   
Protocol                      http                                                                                                                    
Forwarding                     -> 127.0.0.1:80                                                                                                        
Web Interface                 http://localhost:4040                                                                                                   
# Conn                        0                                                                                                                       
Avg Conn Time                 0.00ms      

[inconshreveable]

What sort of corporate proxy are you behind? Is this an HTTP proxy? Currently, ngrok requires that you be able to initiate TCP connections out to the public internet over any target port. It would be possible to run it entirely on ports that corporate firewalls do not restrict though. That actually seems like a better idea to avoid these types of issues.

[gauthamses]

It is a HTTP Proxy.

Yes, it would be a great feature if it works behind a HTTP proxy.

[inconshreveable]

I'm not too familiar with the common corporate firewalls/HTTP proxy setups, but I suspect we could take care of this issue by running all of ngrok's traffic over port 443.

443 needs to be open to support TLS/SSL connections, and unless it is MITMing your traffic it won't know the underlying protocol isn't really HTTP, so this seems like a reasonable strategy. Then again, I don't have this sort of setup to test against, nor do I know if proxies of this nature typically MITM your HTTPS traffic, so I might need some help to test possible solutions to this issue.

[inconshreveable]

This is now working in the refactor branch with -proxyAddr switch.

[tiekuhn]

hi,

I'm having the exact same requirement as gauthamses, so I'm totally happy that you accepted the requirement and added the proxyAddr flag :-)!

If you need help testing the new feature I'll be very happy to help you out.

All I'd need is a ngrok client built from the refactor branch which I can run on my server (which is behind a corporate firewall/http proxy).
...unfortunately I wasn't able to build the binary myself as I do not have "go" installed on my system, which is needed at build time... --> could someone please do the build on linux (SLES 11), I'd be very pleased to do the test afterwards :-)

[inconshreveable]

@tiekuhn I would absolutely love you to try this out and let me know how successful it is at working behind a corporate HTTP proxy. I don't have too many corporate networks to try this with myself =)

Please try this experimental build: https://dl.ngrok.com/linux_386/1.1/ngrok.zip and then let me know what your results are.

If it fails to work, please run it with the -log switch and send me the log file.

Thanks!

[tiekuhn]

Hi,

thanks for the build :-). I think we are quite close to a solution ;-) (but unfortunately not completely there yet)!

I think your commit would already work for "ordinary" http proxies. Unfortunately my proxy requires authentication which still causes errors :-(.

Testcall 1:

./ngrok -authtoken someauthtoken -proxyAddr http://E415_s_s415vmmt113:dd:[email protected]:3128 -log with_http_and_auth.txt 80

--> Result:

[EROR] (ngrok/log.Error:97) control recovering from failure too many colons in address http://E415_s_s415vmmt113:dd:[email protected]:3128

Testcall 2:

./ngrok -authtoken someauthtoken -proxyAddr E415_s_s415vmmt113:dd:[email protected]:3128 -log without_http_and_auth.txt 80

--> Result:

[EROR] (ngrok/log.Error:97) control recovering from failure too many colons in address E415_s_s415vmmt113:dd:[email protected]:3128

Testcall 3:

./ngrok -authtoken someauthtoken -proxyAddr s415016c.detss.someserver.net:3128 -log without_http_and_without_auth.txt 80

--> Result:

[EROR] (ngrok/log.Error:97) control recovering from failure Non-200 response from proxy server: 407 Proxy Authentication Required ( Forefront TMG requires authorization to fulfill the request. Access to the Web Proxy filter is denied.  )

Is there a chance that you add auth support for http proxies?
Another idea would be to follow the unix pattern and pull the proxy config from the default ENV values "http_proxy" and/or "https_proxy" as curl/wget and other unix tools do.

[inconshreveable]

Ah, of course. The current implementation doesn't support proxy authentication. I'll add this capability in and post a new build for you to try later today. Interesting idea about the ENV values as well. I'll look into supporting those since most people will have them configured for other tools (at least on *nix)

[inconshreveable]

@tiekuhn Please try: https://dl.ngrok.com/linux_386/1.2/ngrok.zip

The syntax is the same as you used in testcall2 and is documented when you do ngrok -help. It should look like:

./ngrok -proxyAddr="user:[email protected]:3128" 80

[tiekuhn]

Hi,

looks like we are another step further :-)

[2013/09/16 09:11:45 CEST] [INFO] (ngrok/log.Info:89) Checking for update
[2013/09/16 09:11:45 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [ctl:6bc19194] New connection to: 53.48.15.30:3128
[2013/09/16 09:11:45 CEST] [EROR] (ngrok/log.Error:97) Error while checking for update: Get https://dl.ngrok.com/update/check?arch=386&os=linux&user=someauthtoken&version=1.2: Proxy Authentication Required ( Forefront TMG requires authorization to fulfill the request. Access to the Web Proxy filter is denied.  )
[2013/09/16 09:11:45 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [ctl:6bc19194] Writing message: {"Type":"Auth","Payload":{"Version":"2","MmVersion":"1.2","User":"someauthtoken","Password":"","OS":"linux","Arch":"386","ClientId":""}}
[2013/09/16 09:12:11 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [ctl:6bc19194] Closing
[2013/09/16 09:12:11 CEST] [EROR] (ngrok/log.Error:97) control recovering from failure EOF
[2013/09/16 09:12:11 CEST] [INFO] (ngrok/log.Info:89) Waiting 1 seconds before reconnecting
[2013/09/16 09:12:12 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [view] [term] Waiting for update
[2013/09/16 09:12:12 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [ctl:2c7aa52c] New connection to: 53.48.15.30:3128
[2013/09/16 09:12:12 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [ctl:2c7aa52c] Writing message: {"Type":"Auth","Payload":{"Version":"2","MmVersion":"1.2","User":"someauthtoken","Password":"","OS":"linux","Arch":"386","ClientId":""}}
[2013/09/16 09:12:39 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [ctl:2c7aa52c] Closing
[2013/09/16 09:12:39 CEST] [EROR] (ngrok/log.Error:97) control recovering from failure EOF
[2013/09/16 09:12:39 CEST] [INFO] (ngrok/log.Info:89) Waiting 2 seconds before reconnecting

...do you think the EOF is caused by the proxy or could it be something else?

[inconshreveable]

The EOF was caused by a pretty serious bug in the ngrokd server code in the new version. I've updated the hosted ngrokd server, so please just try again and it should work!

[tiekuhn]

Hurra ;-)!

the tunnel with HTTP-proxy and auth is now working flawlessly :-)!!!!!!

Thank you for your great efforts and support :-)!

[inconshreveable]

Thank you so much for helping me test it!

I'll close out this issue after I add new documentation to the website and release the experimental versions as stable releases later this week.

[tiekuhn]

Hi,

looks like there still could be an issue regarding proxy handling...

once using the tunnel I see following messages in the log

[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [view] [term] Waiting for update
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [pxy:2d95fba6] Writing message: {"Type":"RegProxy","Payload":{"ClientId":"8d81b497c1a39838e2eeb08c33aee20e"}}
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [pxy:3bfec239] Writing message: {"Type":"RegProxy","Payload":{"ClientId":"8d81b497c1a39838e2eeb08c33aee20e"}}
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [pxy:e261bf5] Waiting to read message
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [ctl:451ff28] Reading message with length: 32
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [ctl:451ff28] Read message {"Type":"ReqProxy","Payload":{}}
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [ctl:451ff28] Waiting to read message
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [pxy:71c2ec24] New connection to: 53.48.15.30:3128
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [pxy:e261bf5] Reading message with length: 94
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [pxy:e261bf5] Read message {"Type":"StartProxy","Payload":{"Url":"http://e2emon.ngrok.com","ClientAddr":"141.113.86.96"}}
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [prv:7d75fe60] New connection to: 127.0.0.1:80
[2013/09/16 13:45:24 CEST] [INFO] (ngrok/log.(*PrefixLogger).Info:60) [prv:7d75fe60] Joined with connection pxy:e261bf5
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [ctl:451ff28] Reading message with length: 32
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [ctl:451ff28] Read message {"Type":"ReqProxy","Payload":{}}
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [ctl:451ff28] Waiting to read message
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [pxy:259e002] Waiting to read message
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [view] [term] Waiting for update
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [view] [term] [http] Got HTTP update
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [view] [term] Waiting for update
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [prv:7d75fe60] Copied 15164 bytes from to pxy:e261bf5
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [prv:7d75fe60] Closing
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [pxy:e261bf5] Closing
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [view] [term] [http] Got HTTP update
[2013/09/16 13:45:24 CEST] [WARN] (ngrok/log.(*PrefixLogger).Warn:64) [pxy:e261bf5] Copied 479 bytes to prv:7d75fe60 before failing with error read tcp 53.48.15.30:3128: use of closed network connection
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [view] [term] Waiting for update
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [prv:7d75fe60] Closing
[2013/09/16 13:45:24 CEST] [DEBG] (ngrok/log.(*PrefixLogger).Debug:56) [pxy:e261bf5] Closing

--> mind the line with "[pxy:e261bf5] Copied 479 bytes to prv:7d75fe60 before failing with error read tcp 53.48.15.30:3128: use of closed network connection"

In my browser I receive 502 errors for the requested ressources.

I think it could be related to the size of the ressource I request. Small ressources (<60 KB) seem to load fine whereas bigger ressources (+-100 KB) seem to fail.

Does that behaviour ring a bell?

[inconshreveable]

That "error", "use of closed network connection" is expected and OK. When one side of two joined connections closes, the other side must be closed forcibly and that generates that error.

Can you curl one of these resources with verbosity turned up as high as possible and show me the raw text of the 502 request?

[tiekuhn]

sure

curl -v -i http://e2emon.ngrok.com/dashboard/css/font-awesome.css
* About to connect() to e2emon.ngrok.com port 80
*   Trying 173.255.204.192... * connected
* Connected to e2emon.ngrok.com (173.255.204.192) port 80
> GET /dashboard/css/font-awesome.css HTTP/1.1
User-Agent: curl/7.12.1 (i686-redhat-linux-gnu) libcurl/7.12.1 OpenSSL/0.9.7a zlib/1.2.1.2 libidn/0.5.6
Host: e2emon.ngrok.com
Pragma: no-cache
Accept: */*

< HTTP/1.1 502 Bad Gateway
< Server: nginx/1.4.1
< Date: Mon, 16 Sep 2013 12:12:17 GMT
< Content-Type: text/html
< Content-Length: 172
< Connection: keep-alive
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

100   172  100   172    0     0    616      0 --:--:-- --:--:-- --:--:--   616
100   172  100   172    0     0    615      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host e2emon.ngrok.com left intact
* Closing connection #0
HTTP/1.1 502 Bad Gateway
Server: nginx/1.4.1
Date: Mon, 16 Sep 2013 12:12:17 GMT
Content-Type: text/html
Content-Length: 172
Connection: keep-alive

<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.4.1</center>
</body>
</html>

or even a bit more verbose with tracing enabled

> curl --trace-ascii - http://e2emon.ngrok.com/dashboard/css/font-awesome.css
== Info: About to connect() to e2emon.ngrok.com port 80
== Info:   Trying 173.255.204.192... == Info: connected
== Info: Connected to e2emon.ngrok.com (173.255.204.192) port 80
=> Send header, 208 bytes (0xd0)
0000: GET /dashboard/css/font-awesome.css HTTP/1.1
002e: User-Agent: curl/7.12.1 (i686-redhat-linux-gnu) libcurl/7.12.1 O
006e: penSSL/0.9.7a zlib/1.2.1.2 libidn/0.5.6
0097: Host: e2emon.ngrok.com
00af: Pragma: no-cache
00c1: Accept: */*
00ce: 
<= Recv header, 26 bytes (0x1a)
0000: HTTP/1.1 502 Bad Gateway
<= Recv header, 21 bytes (0x15)
0000: Server: nginx/1.4.1
<= Recv header, 37 bytes (0x25)
0000: Date: Mon, 16 Sep 2013 12:16:11 GMT
<= Recv header, 25 bytes (0x19)
0000: Content-Type: text/html
<= Recv header, 21 bytes (0x15)
0000: Content-Length: 172
<= Recv header, 24 bytes (0x18)
0000: Connection: keep-alive
<= Recv data, 172 bytes (0xac)
0000: <html>
0008: <head><title>502 Bad Gateway</title></head>
0035: <body bgcolor="white">
004d: <center><h1>502 Bad Gateway</h1></center>
0078: <hr><center>nginx/1.4.1</center>
009a: </body>
00a3: </html>
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

100   172  100   172    0     0    638      0 --:--:-- --:--:-- --:--:--   638
100   172  100   172    0     0    638      0 --:--:-- --:--:-- --:--:--     0
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.4.1</center>
</body>
</html>
== Info: Connection #0 to host e2emon.ngrok.com left intact
== Info: Closing connection #0

does it help if I do the same curl requests without the tunnel (=on the intranet) as comparrison?

[inconshreveable]

Hrm. The 502 error is coming back because when ngrokd tries to start copying data between the public connection and proxy connection, the proxy connection is being closed before it even sends a single byte. Since nginx is fronting ngrokd, it notices the connection closes without a response and issues a 502.

It doesn't make sense that it would fail with larger response sizes because the connection is being closed before a single byte is being copied. Can you reliably reproduce small response working and large response failing?

My only guess right now involves how ngrok manages proxy connections in the latest version. The newest ngrok basically asks for proxy connections up-front and then keeps them idle in a pool waiting to serve incoming connections. It's possible that a proxy would enforce a timeout policy on connections (especially if there's no read/write activity) and kill them. If it did though, I would expect ngrok to notice that the connection was killed before it starts to join the connection. So that doesn't entirely make sense. Can you duplicate this issue and then email the full client log for the session? It might help me spot something.

Other than that, I don't have much of an idea on where to look. I might try experimenting with adding a flag that asks ngrokd _not_ to prefech connections in case it really is a timeout-sensitive issue.

[tiekuhn]

hmmm strange.

but the issue definitly seems to be related to the response-size.
the "sweet spot" seems to be between 3-6 KB.

http://e2emon.ngrok.com/dashboard/css/reset.css (<1KB works)
http://e2emon.ngrok.com/dashboard/images/logo_daimler-itr.gif (2,8 KB works)

http://e2emon.ngrok.com/dashboard/css/monitoring.css (6KB --> 502)
http://e2emon.ngrok.com/dashboard/images/mercedes-benz-logo.png (8KB --> 502)
http://e2emon.ngrok.com/dashboard/css/font-awesome.css (14 KB --> 502)

there is also no "flickering"...the ressource either works every time or not at all.

on which port does ngrok communicate with your server? If I'd know the port I could do a tcpdump for further analysis...if that helps ;-).

[inconshreveable]

Wow that's really weird. If it's size related, it suggests to me that the proxy server is buffering the request and rejecting all requests over a certain size. But that would imply that you couldn't upload any large files through the proxy which seems extremely unlikely.

ngrok communicates over port 443 to ngrok.com, but obviously between ngrok and the proxy it will communicate over whatever port the proxy server is running on.

[inconshreveable]

@tiekuhn Can you try once more? I think I found a possible culprit: ngrok was proxying an HTTP url instead of an HTTPS url which means a proxy might choose to try to inspect/modify the request instead of getting out of the way and treating the TLS-encrypted communication as opaque.

Try the linked build below with the following change:
I've removed the command-line flag in favor of using the standard unix environment variable as you've suggested. Set the environment variable "http_proxy" using the same format as expected by other unix tools. Here's an example of how to invoke ngrok now to proxy:

http_proxy="http://user:password@host:3128" ngrok 80

https://dl.ngrok.com/linux_386/1.3/ngrok.zip

[inconshreveable]

Closing, no one has reported this issue since I've pushed the fixed. We can reopen if someone encounters this again.

[csms]

Hmm. I also get the bad gw 502. Appears on a file that's 9 kb. If I take the same file and strip it down do 2kb I get 200. Also behind proxy. Tried setting env variable according to (but with "my" values):
http_proxy="http://user:password@host:3128"

[inconshreveable]

@csms The more I've learned about this, the more I believe this is not a solvable issue on the ngrok side. I suspect that your http proxy has a setting for something like "max request size" and will simply drop connections that exceed a certain size.

[kenshinji]

@inconshreveable Is ngrok supporting http proxy officially?

[liuchong]

@inconshreveable
hI, thanks for the -proxyAddr feature, but where is the refactor branch?

[craigsdennis]

Hi @kenshinji and @liuchong ...coming in late here, but http proxy will take from your environment variable http_proxy. Hope that helps!

https://github.com/inconshreveable/ngrok/blob/a8e7fa486348f391f4bdf241344d798297f953bc/src/ngrok/client/config.go#L80

[tango-j]

does it support authentication with proxy? @nickpresta @inconshreveable

[hariom282538]

http_proxy="http://proxy.example.com:8123" ngrok http 5000

this also supports self-signed proxy-server certificate.
it took few moment.