Habitat: Corporate proxy not working on Mac

[hiucimon]

[metadave]

@hiucimon looks like the version of hab you have is quite old, can you try the recently released v 0.9.0? https://www.habitat.sh/docs/get-habitat/

[metadave]

I suspect that the problem is related to this line:

Using HAB_BIN=/hab/pkgs/core/hab/0.7.0/20160614230104/bin/hab

any thoughts @fnichol?

[hiucimon]

That was the version I got to work (by turning off proxy and going through my phone), and it is failing trying to fetch the newer version.

[metadave]

try:

     docker pull habitat-docker-registry.bintray.io/studio

related: https://github.com/habitat-sh/habitat/issues/1167

[metadave]

1. can you remove any previous studios you've created with hab studio rm?
2. can you gist your plan, removing anything proprietary?

[hiucimon]

Not sure what you mean by gist?

[hiucimon]

I removed the entire directory where my plan.sh was. and recreated it, and did the hab studio rm, same results. Is there a way to remove ALL the files for Habitat and start from stratch?

[hiucimon]

I deleted my .hab directory as well and did a hab setup again. Same thing. Invalid Status provided.

[metadave]

We use the hyper HTTP library, which uses httparse. As far as I can tell, it appears that there are invalid characters in the response, mostly like coming from the proxy:

https://github.com/seanmonstar/httparse/blob/master/src/lib.rs#L104-L105
https://github.com/seanmonstar/httparse/blob/master/src/lib.rs#L363

[hiucimon]

OK. The proxy is working on every thing else in the company, which is a LOT which you can see it working with Docker. Where do I go from here?

[hiucimon]

If I remove the proxies from Docker and go through my phone, works fine, but if I put the proxies, even if I have no special characters in my password, if fails.

[adamhjk]

@hiucimon thanks for helping us troubleshoot this - I know its frustrating to be stuck. Can you get us a traffic dump of your communications with the proxy? Something like this on your mac, while attempting to build:

$ sudo tcpdump -s 0 -vvv -tt -w /tmp/out.tcpdump

Would be very helpful. We appreciate it.

[hiucimon]

I captured the trace, but I can't send it (financial institute). I am pretty good at reading IP traces if you can tell what you want out of it, I can get you the packets you're looking for.

[hiucimon]

I restricted it to just the conversation with the proxy and have it down to 32 packets. You are getting an "Bad Request".

[metadave]

So sorry for the inconvenience, but your request was unable to be processed at this time.

[hiucimon]

As you can see, the access DOES work, and I do have access. This is the only piece of software on my system not working.

[hiucimon]

What resource is it trying to reach when it is getting this failure? I can show you it works from a browser or other system that honors the proxy properly.

[metadave]

our guess is that the proxy server is expecting a user agent, which we're not sending.

from some of your trace, it looks like the resource it's trying to hit is:
https://willem.habitat.sh/v1/depot/pkgs/core/node
or
https://willem.habitat.sh/v1/depot/pkgs/core/node/latest
Although, I don't see the initial HTTP request in the pcap file.

See also:
https://github.com/habitat-sh/habitat/blob/master/components/builder-depot-client/src/lib.rs#L220-L231
https://github.com/habitat-sh/habitat/blob/master/components/builder-depot-client/src/lib.rs#L281

[hiucimon]

I open both of those fine in a browser.

[{"origin":"core","name":"node","version":"4.2.6","release":"20160612143531"},{"origin":"core","name":"node","version":"4.2.6","release":"20160708162711"},{"origin":"core","name":"node","version":"4.2.6","release":"20160728175113"},{"origin":"core","name":"node","version":"4.2.6","release":"20160729200209"}] {"checksum":"1ffdccf7b076a8f4006d9011c6478d2d3ffbfe4b5a0cc0f50d26b04ff00f2035","config":"","deps":[{"name":"glibc","origin":"core","release":"20160612063629","version":"2.22"},{"name":"gcc-libs","origin":"core","release":"20160612075020","version":"5.2.0"}],"exposes":[],"ident":{"name":"node","origin":"core","release":"20160729200209","version":"4.2.6"},"manifest":"# core / node\n\n\n* __Maintainer__: The Habitat Maintainers <[email protected]>\n* __Version__: 4.2.6\n* __Release__: 20160729200209\n* __Architecture__: x86_64\n* __System__: linux\n* __Target__: x86_64-linux\n* __Upstream URL__: upstream project's website or home page is not defined\n* __License__: MIT \n* __Source__: [https://nodejs.org/dist/v4.2.6/node-v4.2.6.tar.gz](https://nodejs.org/dist/v4.2.6/node-v4.2.6.tar.gz)\n* __SHA__: `ea5e357db8817052b17496d607c719d809ed1383e8fcf7c8ffc5214e705aefdd`\n* __Path__: `/hab/pkgs/core/node/4.2.6/20160729200209`\n* __Build Dependencies__: `core/coreutils core/python2 core/gcc core/make `\n* __Dependencies__: `core/glibc core/gcc-libs `\n* __Interpreters__: `bin/node `\n\n# Plan\n\n## Build Flags\n\n```bash\nCFLAGS: -I/hab/pkgs/core/glibc/2.22/20160612063629/include -I/hab/pkgs/core/python2/2.7.12/20160729195942/include -I/hab/pkgs/core/python2/2.7.12/20160729195942/Include -I/hab/pkgs/core/gcc/5.2.0/20160612064854/include -I/hab/pkgs/core/make/4.2.1/20160729193703/include\nCPPFLAGS: -I/hab/pkgs/core/glibc/2.22/20160612063629/include -I/hab/pkgs/core/python2/2.7.12/20160729195942/include -I/hab/pkgs/core/python2/2.7.12/20160729195942/Include -I/hab/pkgs/core/gcc/5.2.0/20160612064854/include -I/hab/pkgs/core/make/4.2.1/20160729193703/include\nCXXFLAGS: -I/hab/pkgs/core/glibc/2.22/20160612063629/include -I/hab/pkgs/core/python2/2.7.12/20160729195942/include -I/hab/pkgs/core/python2/2.7.12/20160729195942/Include -I/hab/pkgs/core/gcc/5.2.0/20160612064854/include -I/hab/pkgs/core/make/4.2.1/20160729193703/include\nLDFLAGS: -L/hab/pkgs/core/glibc/2.22/20160612063629/lib -L/hab/pkgs/core/gcc-libs/5.2.0/20160612075020/lib -L/hab/pkgs/core/python2/2.7.12/20160729195942/lib -L/hab/pkgs/core/gcc/5.2.0/20160612064854/lib\nLD_RUN_PATH: /hab/pkgs/core/node/4.2.6/20160729200209/lib:/hab/pkgs/core/glibc/2.22/20160612063629/lib:/hab/pkgs/core/gcc-libs/5.2.0/20160612075020/lib\n```\n\n## Plan Source\n\n```bash\npkg_name=node\npkg_origin=core\npkg_version=4.2.6\npkg_license=('MIT')\npkg_maintainer=\"The Habitat Maintainers <[email protected]>\"\npkg_source=https://nodejs.org/dist/v${pkg_version}/${pkg_name}-v${pkg_version}.tar.gz\n# the archive contains a 'v' version # prefix, but the default value of\n# pkg_version is node-4.2.6 (without the v). This tweak makes build happy\npkg_dirname=node-v${pkg_version}\npkg_shasum=ea5e357db8817052b17496d607c719d809ed1383e8fcf7c8ffc5214e705aefdd\npkg_deps=(core/glibc core/gcc-libs)\npkg_build_deps=(core/coreutils core/python2 core/gcc core/make)\npkg_bin_dirs=(bin)\npkg_include_dirs=(include)\npkg_interpreters=(bin/node)\npkg_lib_dirs=(lib)\n\ndo_prepare() {\n # The `/usr/bin/env` path is hardcoded, so we'll add a symlink if needed.\n if [[ ! -r /usr/bin/env ]]; then\n ln -sv $(pkg_path_for coreutils)/bin/env /usr/bin/env\n _clean_env=true\n fi\n}\n\ndo_build() {\n # use --without-snapshot, because https://github.com/nodejs/node/issues/4212\n ./configure --prefix=${pkg_prefix} --without-snapshot\n make\n}\n\ndo_install() {\n do_default_install\n\n # Node produces a lot of scripts that hardcode `/usr/bin/env`, so we\n # need to fix that everywhere to point directly at the node binary. There is\n # no other good reason to add the env indirection trick on these shebangs.\n grep -nrlI '^\\#\\!/usr/bin/env' $pkg_prefix | while read target; do\n sed -e \"s#\\#\\!/usr/bin/env node#\\#\\!${pkg_prefix}/bin/node#\" -i $target\n done\n}\n\ndo_end() {\n # Clean up the `env` link, if we set it up.\n if [[ -n \"$_clean_env\" ]]; then\n rm -fv /usr/bin/env\n fi\n}\n```","tdeps":[{"name":"gcc-libs","origin":"core","release":"20160612075020","version":"5.2.0"},{"name":"glibc","origin":"core","release":"20160612063629","version":"2.22"},{"name":"linux-headers","origin":"core","release":"20160612063537","version":"4.3"}]}

[metadave]

ok, does your proxy server require a user agent to be specified?

[hiucimon]

I would not know. I would have to try and hunt down the network engineers and try to get them to dig into the specs. I think we use F5.

[hiucimon]

So, where do we go from here? Do I need to engage my network teams or do we assume it need the user agent and try it with one?

[jtimberman]

@hiucimon Does it work if you use curl?

curl https://willem.habitat.sh/v1/depot/pkgs/core/node/latest

And with a user agent?

curl -A 'Mozilla/5.0' https://willem.habitat.sh/v1/depot/pkgs/core/node/latest

You may need to try different user agent strings depending on what desktop browser you're using successfully, to get one that matches.

[hiucimon]

Works with both.

[adamhjk]

Lets create a branch with a user agent on it and have @hiucimon give it a shot.

[hiucimon]

Any estimates on when we might have something to try?

[hiucimon]

What are our next steps?

[fnichol]

Hi there, @hiucimon, I'm the developer that added the initial HTTP/proxy support and just back from some time off. I'll get on this straight away and really appreciate your feedback on this PR.

For the others playing along, the hab HTTP client is setting a User-Agent header which contains the program, version, and OS/kernel info such as User-Agent: hab/0.9.0/20160815225003 (x86_64-linux; 4.4.15-moby)--this is an example of installing core/jq-static via and https_proxy with a bit more debug tuning to see the http-related info:

# env RUST_LOG=habitat_http_client=debug,hyper=debug,warn https_proxy=http://jdoe:[email protected]:8123 hab install core/jq-static
DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem
DEBUG:habitat_http_client::api_client: Using proxy 10.200.0.104:8123...
DEBUG:habitat_http_client::api_client: User-Agent: hab/0.9.0/20160815225003 (x86_64-linux; 4.4.15-moby)
» Installing core/jq-static
DEBUG:habitat_http_client::api_client: GET https://willem.habitat.sh/v1/depot/pkgs/core/jq-static/latest with ApiClient { endpoint: "https://willem.habitat.sh/v1/depot", inner: Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None }, proxy: Some(ProxyInfo { url: "http://jdoe:[email protected]:8123/", authorization: Some(ProxyBasicAuthorization { username: "jdoe", password: "sassafrass" }) }), target_scheme: "https", user_agent_header: UserAgent("hab/0.9.0/20160815225003 (x86_64-linux; 4.4.15-moby)") }
DEBUG:hyper::net: http scheme
DEBUG:habitat_http_client::net: Proxy 10.200.0.104:8123 "CONNECT willem.habitat.sh:443 HTTP/1.1, Host willem.habitat.sh:443, Proxy-Authorization: Basic amRvZTpzYXNzYWZyYXNz"
DEBUG:habitat_http_client::net: Proxy 10.200.0.104:8123 CONNECT success = 200
DEBUG:hyper::http::h1: request line: Get "/v1/depot/pkgs/core/jq-static/latest" Http11
DEBUG:hyper::http::h1: headers=Headers { Host: willem.habitat.sh, User-Agent: hab/0.9.0/20160815225003 (x86_64-linux; 4.4.15-moby), }
DEBUG:hyper::client::response: version=Http11, status=Ok
DEBUG:hyper::client::response: headers=Headers { Server: nginx/1.8.0, Fastly-Debug-Digest: 964aac9ae9ab1259c7f2577247039b8302e432dedfdeaef2c26adc6f2a97fc2b, ETag: ddae60ebc63f6aea0efdba44d7211d3633d1bfa3c3ab37d4e807817a274d4905, Access-Control-Expose-Headers: content-range, next-range, x-content-range, Date: Mon, 29 Aug 2016 15:56:14 GMT, Cache-Control: public, max-age=31536000, private, no-cache, no-store, Access-Control-Allow-Origin: *, Age: 0, Via: 1.1 varnish1.1 varnish, Content-Type: text/plain, X-Served-By: cache-sea1920-SEA, cache-ord1743-ORD, Strict-Transport-Security: max-age= 7776000; includeSubDomains, Content-Length: 1757, Access-Control-Allow-Headers: authorization, range, Accept-Ranges: bytes, X-Cache-Hits: 0, 0, Vary: Accept-Encoding, Connection: keep-alive, X-Frame-Options: SAMEORIGIN, X-Cache: MISS, MISS, }
✓ Installed core/jq-static/1.5/20160730001211
★ Install of core/jq-static/1.5/20160730001211 complete with 1 new packages installed.

@hiucimon, could I ask you to enter a Studio, as above (in a shell with at least the https_proxy environment variable set, with a hab studio enter) and then try this for me:

env RUST_LOG=habitat_http_client=debug,hyper=debug,warn hab install core/jq-static

There's a really good chance that you'll want to inspect the output for your username and/or password in that output--you can see I used jdoe and sassafrass above which echos back. I'm curious if the authentication step to the http_proxy is failing and if that might be the cause of the issue.

In the meantime, I'll get myself back up to speed and fully digest the thread so far, including that trace zip file.

[juliandunn]

@hiucimon I also suggested to @fnichol that we jump on a GoToMeeting (I understand your employer blocks Zoom which we use) because attempting to do this kind of debugging async over a GitHub issue is kind of 😿 Let me know if you'd be game for that and what your availability is, including timezone (you can e-mail me at [email protected])

[fnichol]

I'm super down for this!

[hiucimon]

I am OOO until later this afternoon, but will check then.

[hiucimon]

I would be willing to schedule something when I get back to the office and have my schedule available. 

Sent from my Verizon, Samsung Galaxy smartphone

[hiucimon]

Setting up a webex will prove to be problematic, because it will not allow me to scrub the results before showing it. As it stands I can insure no sensitive data is seen. It may be slower, but I tink it is what we have to settle for, at least for now.

[juliandunn]

@hiucimon you folks are an existing customer, so we already have a mutual NDA in place. is that sufficient?

[hiucimon]

If you are with Chef, certainly.

[hiucimon]

Just have to be careful with open source support.

[hiucimon]

I could likely be available between 9am and 11am est tomorrow.

[fnichol]

Nice, this does help. It confirms my guess that the proxy server itself is rejecting the CONNECT HTTP request with authentication header and returning a 4xx error code.

[fnichol]

Okay, I was more wrong than right: we are not providing a User-Agent to the proxy server via its CONNECT request. It looks like both wget and curl set a User-Agent header on the CONNECT request although it looks like Go does not

[juliandunn]

@hiucimon no worries, both @fnichol and I are employees of Chef Software Inc. so bound by mutual NDA. I will see if Fletcher still wants to do a Gotomeeting and we can do that tomorrow if he needs more info.

[hiucimon]

That works.

On Aug 29, 2016, at 5:59 PM, Julian C. Dunn [email protected] wrote:

@hiucimon https://github.com/hiucimon no worries, both @fnichol https://github.com/fnichol and I are employees of Chef Software Inc. so bound by mutual NDA. I will see if Fletcher still wants to do a Gotomeeting and we can do that tomorrow if he needs more info.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/habitat-sh/habitat/issues/1180#issuecomment-243270856, or mute the thread https://github.com/notifications/unsubscribe-auth/AMXywh-9sQ9krvs_4wYs9qrD_gT3t4WXks5qk1ZcgaJpZM4JqRGF.

[juliandunn]

@hiucimon ok, let's shoot for 9-10 a.m. tomorrow. you want to send me your e-mail address privately please ([email protected]) and I'll set up an invite in Gotomeeting?

[hiucimon]

In your text you sent me to test with, you left the ':' off after Host. I wrote a PERL program to simulate with, did Wireshark traces of it and CURL, and did byte by byte compares and caught the delta.

[hiucimon]

I just verified that in Habitat, you are missing the ':' after the "Host" as well. Looks like the answer.

[fnichol]

@hiucimon Ha, great eyes! Working up a PR for the fix

[fnichol]

@hiucimon Okay, I have a PR open (above) and a candidate build that you can try if rebuilding the hab binary is too onerous host temporarily on (s3)[https://s3-us-west-2.amazonaws.com/habitat-initial-hab/core-hab-0.10.0-dev-20160830185042-x86_64-linux.hart]. The easiest way to try it out would be to enter a Studio with the https_proxy in the environment, then download and install that version to try with:

# download the package
wget https://s3-us-west-2.amazonaws.com/habitat-initial-hab/core-hab-0.10.0-dev-20160830185042-x86_64-linux.hart

# install the package into the studio
hab install ./core-hab-0.10.0-dev-20160830185042-x86_64-linux.hart

# update the symlink that answers to `hab`
hab pkg binlink -d /hab/bin core/hab hab

# double check that `hab` now returns a version of 0.10.0-dev
hab --version

Then let's try to download that static jq package again with:

env RUST_LOG=habitat_http_client=debug,hyper=debug,warn hab install core/jq-static

Fingers crossed!

[hiucimon]

Issues trying this procedure. Maybe for a Linux not Mac?

[hiucimon]

Wait, I didn't read it as clearly as I should have....

[hiucimon]

No, still fails.

Saving to: 'core-hab-0.10.0-dev-20160830185042-x86_64-linux.hart.1'

core-hab-0.10.0-dev-20160830185042-x86_64-linux.h 100%[=============================================================================================================>] 2.30M 668KB/s in 3.9s

2016-08-30 19:36:51 (604 KB/s) - 'core-hab-0.10.0-dev-20160830185042-x86_64-linux.hart.1' saved [2408073/2408073]

[2][default:/src:0]# hab install ./core-hab-0.10.0-dev-20160830185042-x86_64-linux.hart
✗✗✗
✗✗✗ No such file or directory (os error 2)
✗✗✗

[hiucimon]

I am not sure if it matters, but I run had out of /usr/local/bin/ copied there as root. Do I need to change that?

[fnichol]

@hiucimon Maybe try:

hab install ./core-hab-0.10.0-dev-20160830185042-x86_64-linux.hart.1

It looks like that's what wget saved it off as

[fnichol]

And yes, sorry I wasn't clear: this .hart package is for Linux (will run inside a hab studio enter)

[hiucimon]

That was because I had tried it trice. I tried the install both ways as well, no good.

On Aug 30, 2016, at 4:52 PM, Fletcher Nichol [email protected] wrote:

@hiucimon https://github.com/hiucimon Maybe try:

hab install ./core-hab-0.10.0-dev-20160830185042-x86_64-linux.hart.1
It looks like that's what wget saved it off as

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/habitat-sh/habitat/issues/1180#issuecomment-243576989, or mute the thread https://github.com/notifications/unsubscribe-auth/AMXywlbUe4YzXbXu72sazJE6WvlNy3jXks5qlJgbgaJpZM4JqRGF.

[fnichol]

Could I get the output from (running the installed package inside a hab studio enter):

env RUST_LOG=habitat_http_client=debug,hyper=debug,warn /hab/pkgs/core/hab/0.10.0-dev/20160830185042/bin/hab install core/jq-static

Just want to confirm that the colon is present in the debug output. Thanks!

[hiucimon]

I am stuck on how to test this.

[hiucimon]

OK guys, it was a contorted path but I have your results. I created a virtual machine with Ubuntu, put Habitat 0.9, took some doing, but extracted the file out of the one you created for me and names it habv10, ran the command you asked, and it DOES work behind the proxy and the Host does hae the ':' correct. Way to go.

Now when can I get a Mac build??? :)

[fnichol]

Ah sweeet!

We have this fix in master now and planning on a release tomorrow for that would cut a 0.10.0 across all the software. At that point you should be able to pull a new Mac binary, pull an updated Docker image and we'll be good to go!

Thank you!

[hiucimon]

Verified that the fix works. :)