Habitat: Make working with Corporate SSL Firewalls easier

[bdangit]

As a developer who works behind a firewall where the SSL traffic is inspected, I want hab to work with my own corporate cert.

Under linux environment, I am able to insert my own cert to get into a studio, by locating where the $HAB_STUDIOS/$studio_name/hab/cache/ssl/cert.pem. However, once I attempt to do a build, hab is setting the SSL context to cert.pem that is located in core/cacerts. I could overwrite this file after the first failure, but then this makes it tricky moving forward.

[bdangit]

Diving deep, there are at least 3 spots where I see the cert.pem and cacerts.pem are getting stored.

1. $HAB_STUDIOS/$studio_name/hab/cache/ssl/cert.pem: This is required on first initial building of the studio, and is used to download all the necessary studio dependencies.
2. Within the studio, dependencies use the cert stored from $(pkg_path_for core/cacerts)/ssl/certs/cacert.pem
3. From my base linux host, I have them at /etc/ssl/certs/MYCOMPANY.pem

One would think that your build dependencies should assume what you have setup in your $HAB_STUDIOS but it looks like this feature is only specific to linux. And the commonality between mac and linux would be the use of core/cacerts package. So no more assumptions. It is interesting to see where certs are used all around and that it's literally everywhere.

Workaround

I had to manually overwrite the files inside core/cacerts and hab was happy to continue downloading dependencies.

Longer Term Solution

I wish there was an ability to allow companies to insert their own myorigin/cacerts to be used instead of the core/cacerts. This should probably get introduced when we start to build a studio.

[bdangit]

ah okay something I didn't know, but export SSL_CERT_FILE=/path/to/your/company/cert.pem will allow you to override the default of resorting to core/cacerts within the studio.

[bdangit]

This adds ability to install hab behind a firewall using the components/hab/install.sh script https://github.com/habitat-sh/habitat/pull/2597

[robbkidd]

This issue has come up in another environment.

Some context from this environment:

We're trying to build in a CI job running on a Linux host. This host lives behind a firewall that's performing "transparent" SSL proxying. Nothing using only the public cacerts chain will succeed. Hab bits error, complaining about a self-signed cert (the one presented by the firewall).

🔧 First fix: the host

export SSL_CERT_FILE=/path/to/your/company/certchain/totally-trust-the-corporate-man-in-the-middle.pem

Will allow curl to download the install script and the hab binary.

hab studio enter will succeed in downloading the bits necessary to enter the studio.

😢 Initial Life in the Studio

... but, dear reader, the environment within the studio

1. does not have the SSL_CERT_FILE environment variable set
2. even if it did, /path/to/your/company/certchain/totally-trust-the-corporate-man-in-the-middle.pem is not a file available within the studio

🔧 Second fix: the studio

OK. So, we cp /path/to/your/company/certchain/totally-trust-the-corporate-man-in-the-middle.pem ./ and _then_ enter the studio and _then_ export SSL_CERT_FILE=/src/totally-trust-the-corporate-man-in-the-middle.pem and _then_ the studio environment will trust connections out to the internet. We can download packages and crank out builds. Whoohoo!

😭 CI Build, No Joy

hab studio build . ... "Nope, that's a self-signed cert."

Nuts. Right. The build is a clean environment. No SSL_CERT_FILE environment variable is passed in from the host to the studio. We've still got our copy of the totally-trust-the-corporate-man-in-the-middle.pem in the root of the source directory, so the file is available. But there is no env var set to point to it.

🔨 Third fix: the build

In the end, we had to add this less-than-ideal workaround to the CI script for each of the projects we wanted to build:

# set the trusted cert chain for the host
export SSL_CERT_FILE=/path/to/your/company/certchain/totally-trust-the-corporate-man-in-the-middle.pem

# copy the cert chain into the source directory being built
cp /path/to/your/company/certchain/totally-trust-the-corporate-man-in-the-middle.pem ./

# prepend trusting the copied cert chain to the plan itself, sorry about your version control!
echo -e "export SSL_CERT_FILE=/src/totally-trust-the-corporate-man-in-the-middle.pem\n$(cat plan.sh)" l> plan.sh

# now you can build it, god save your soul
hab studio build .

[bdangit]

haha that almost sounds like what I had to do in my hab wrapper. There should be an easy way.

[robbkidd]

I _think_ I'd like for hab studio * commands to detect the presence of SSL_CERT_FILE being set on the host and if set:

• copy the file at that path to the hab cache that gets mounted into the studio
• set SSL_CERT_FILE inside the studio session to that cached file path

I see hints in http-client/src/lib.rs about caching a certs.pem in the build of the http-client itself. I also see /hab/cache/ssl/cert.pem appear inside a studio. Are there code paths that can be updated to replace that cached cert.pem with a custom certs bundle when creating a studio?

That's a solution for the _build_ environment. If the _packaged services_ should be using that cert-chain, the cert.pem should _also_ get packaged up as muhcompany/ourcerts and packages set SSL_CERT_FILE={{pkgPathFor muhcompany/ourcerts}} in run hooks as appropriate.

[rsertelon]

@robbkidd the package muhcompany/ourcerts could be used as a build dependency to, couldn't it? I mean, I like a clear story for the CERT_SSL_FILE being picked up by the studio, but I don't see why for building it could not be included in the package itself as a dependency?
Nevermind x)

[bdangit]

Sorry cat made me close. :X

[bdangit]

After many of my colleagues kept running into this issue over and over, (granted I also had way too complicated SSL injection scripts), I took the time to start fresh and dig deep into this again.

Habitat is really simple to get setup when dealing in the face of corp certs.

To install hab using the hab install.sh

SSL_CERT_FILE=/path/to/cert install.sh

To enter the hab studio the very first time

SSL_CERT_FILE=/path/to/cert hab studio enter

To make sure you can still build stuff within the studio

export SSL_CERT_FILE=/path/to/cert
build /path/to/plan.sh

or

Place export SSL_CERT_FILE=/path/to/cert in your .studiorc file. Make sure the path you choose is within the perspective of /src.

To build stuff without entering the studio

For this to work, you can't really make use of the hab studio build, since you need to make sure you set the SSL_CERT_FILE environment variable inside the studio context. To get around that, you would do this:

SSL_CERT_FILE=/path/to/cert hab studio run "export SSL_CERT_FILE=/path/within/studio/to/cert && build /path/to/plan.sh"

Alternatively, you could source the .studiorc if you placed your SSL_CERT_FILE export in there, like so:

SSL_CERT_FILE=/path/to/cert hab studio run "source .studiorc && build /path/to/plan.sh"

Feature Request

So all this boils down to, if there is one thing that we can do to make hab great behind corp firewalls would be to implement what @robbkidd suggested:

I think I'd like for hab studio * commands to detect the presence of SSL_CERT_FILE being set on the host and if set:

• copy the file at that path to the hab cache that gets mounted into the studio
• set SSL_CERT_FILE inside the studio session to that cached file path

[Belogix]

We have just hit this issue (two years later) after trying and failing to install Chef Automate v2, which in turn uses Habitat under the covers. Chef Automate must be squarely aimed at the Enterprise and it is surprising to find that we still have to use these work around steps in our process. We are reaching out to Chef to see if this really has not been resolved and there is no better alternative.