Packer: Packer on MacOS installed via Homebrew throws x509: certificate signed by unknown authority

[scross01]

The packer builds for MacOS appear to be different depending whether that are obtained from https://packer.io/download vs installing via Homebrew. The Homebrew installed version is throwing SSL certificate errors x509: certificate signed by unknown authority. The MacOS version downloaded from packer.io/downloads.html works okay.

from homebrew

$ brew install packer
Updating Homebrew...
...
==> Downloading https://homebrew.bintray.com/bottles/packer-1.2.2.high_sierra.bottle.tar.gz
Already downloaded: /Users/scross/Library/Caches/Homebrew/packer-1.2.2.high_sierra.bottle.tar.gz
==> Pouring packer-1.2.2.high_sierra.bottle.tar.gz
==> Caveats
zsh completions have been installed to:
  /usr/local/share/zsh/site-functions
==> Summary
🍺  /usr/local/Cellar/packer/1.2.2: 7 files, 86.8MB

$ packer version
Packer v1.2.2

$ packer build -debug -var-file="ptstest.vars" packer.json 
Debug mode enabled. Builds will not be parallelized.
oracle-classic output will be in this color.

Build 'oracle-classic' errored: Error creating OPC Compute Client: Post https://api-z61.compute.us6.oraclecloud.com/authenticate/: x509: certificate signed by unknown authority

==> Some builds didn't complete successfully and had errors:
--> oracle-classic: Error creating OPC Compute Client: Post https://api-z61.compute.us6.oraclecloud.com/authenticate/: x509: certificate signed by unknown authority

==> Builds finished but no artifacts were created.

from packer.io/downloads.html

$ unzip packer_1.2.2_darwin_amd64.zip
$ ./packer version
Packer v1.2.2

./packer build -debug -var-file="ptstest.vars" packer.json 
Debug mode enabled. Builds will not be parallelized.
oracle-classic output will be in this color.

==> oracle-classic: Creating temporary ssh key for instance...
    oracle-classic: Saving key for debug purposes: oci_classic_oracle-classic.pem
==> oracle-classic: Pausing after run of step 'StepKeyPair'. Press enter to continue. 
...

provisioning continues as expected.

[scross01]

Noted the following differences when running with PACKER_LOG=1 - appears to be built with different versions of Go.

from Homebrew

2018/03/28 09:58:10 Waiting for RPC address for: /usr/local/bin/packer
2018/03/28 09:58:10 packer: 2018/03/28 09:58:10 [INFO] Packer version: 1.2.2
2018/03/28 09:58:10 packer: 2018/03/28 09:58:10 Packer Target OS/Arch: darwin amd64
2018/03/28 09:58:10 packer: 2018/03/28 09:58:10 Built with Go Version: go1.10

from packer.io/downloads.html

2018/03/28 09:54:25 packer: 2018/03/28 09:54:25 [INFO] Packer version: 1.2.2
2018/03/28 09:54:25 packer: 2018/03/28 09:54:25 Packer Target OS/Arch: darwin amd64
2018/03/28 09:54:25 packer: 2018/03/28 09:54:25 Built with Go Version: go1.8.1

[SwampDragons]

I can't reproduce. 😕

I tested with the Packer from our downloads page, and it ran a simple Oracle-classic build with no issues. Then I downloaded the Homebrew Packer, and it also built fine. Homebrew seems to recompile Packer from source rather than using the Packer Downloads page, which is why you got the golang version mismatch (note, this was my fault -- I should have released Packer 1.2.2 compiled with golang 1.10 instead of compiled with golang 1.8.1; and I've since fixed this so that the Packer downloaded from our website is in fact compiled with golang 1.10, so there should be no mismatch between the two anymore), but being unable to reproduce the issue with the Homebrew Packer, I'm not sure where to go from here.

Are you sure there isn't something weird going on with your config?

[scross01]

Hmm, in case it helps I'm running macOS High Sierra 10.13.3

[SwampDragons]

Ah. We've seen some other weirdness with golang 1.10 and osx High Sierra. What happens if you re-download Packer from our downloads page so you're using the one compiled with 1.10? Does it stop working again?

[scross01]

The re-downloaded of version 1.2.2 that is now built with go 1.10 has stopped working for me

[SwampDragons]

Okay, so this sounds like an upstream bug with golang 1.10 and osx High Sierra. Second bug I've seen this week relating to those that is affecting Packer builds. I need to figure out exactly where this bug is and open an issue with golang, if there isn't one already.

[majksner]

I have similar issue that might be related to this one:

==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: RequestError: send request failed
caused by: Post https://ec2.us-east-1.amazonaws.com/: net/http: TLS handshake timeout

[rickard-von-essen]

This might been fixed by https://golang.org/doc/devel/release.html#go1.10.minor

[mwhooker]

This might been fixed by https://golang.org/doc/devel/release.html#go1.10.minor

wow, what timing, lol

[mwhooker]

Here is packer built with go 1.10.1 for darwin. Would appreciate any help in testing if this solves the problem.

packer-1.2.2-go-1.10.1.zip

[scross01]

Nope, still get the same error :-(

[mwhooker]

unfortunately I'm not able to reproduce this error, either. Will continue investigating. Seems like it _might_ have something to do with your environment. Wonder if any type of VPN client might affect your x509 certs. If that's the case then maybe we can add a flag to skip CA checking

[scross01]

The issue does seem to be specific to this environment. I've tested on a separate Mac and it works fine. Doing some further digging I think the following golang fix is related https://github.com/golang/go/issues/16532 - Prior to 1.10 golang was only using the Certificates in the "System Roots" keychain, with go 1.10 it also uses the Certificates "System" keychain. So it seems an invalid Certificate in the "System" keychain of this machine is causing the issue.

Not a Packer bug, closing the issue.

[scross01]

Reopening. There must be something Packer specific going on, because Terraform is using the same Oracle Cloud go SDK library, is also built with go 1.10, and I can provision to the same API endpoint without any issues.

[majksner]

It looks like this issue is only with OSX 10.13.3 and 10.13.4. I can run packer in docker container without any problems.

[mwhooker]

@scross01 I'm comparing how terraform constructs the opc client vs how packer does it, and I don't see too much difference. The only difference that might have an effect is the option to skip certificate validation:

insecure - (Optional) Skips TLS Verification for using self-signed certificates. Should only be used if absolutely needed. Can also via setting the OPC_INSECURE environment variable to true.

Any chance you have that env var set or have the insecure option set in your tf config?

[scross01]

Double-checked, definitely don't have the insecure option set.

[mwhooker]

@scross01 if you've got gdb configured, would you mind telling me the output of

gdb -q -ex "p 'runtime.iscgo'" -ex quit <path to homebrew packer>

[scross01]

$ gdb -q -ex "p 'runtime.iscgo'" -ex quit /usr/local/Cellar/packer/1.2.2/bin/packer
Reading symbols from /usr/local/Cellar/packer/1.2.2/bin/packer...done.
Loading Go Runtime support.
$1 = true

[CyberKoz]

I am having a similar issue and I have tried rebuilding go and packer from source. The issue appears to be with packer using the System keychain. Terraform is able to connect to ec2 just fine, but packer errors out. In the mean time I can work around it by setting the default CA with overriding AWS_CA_BUNDLE and using the CA bundle from https://curl.haxx.se/ca/cacert.pem

[tylert]

I have been using Packer 1.2.2 on macOS 10.13.4 installed via homebrew. I have not seen the errors mentioned here on macOS (or Debian Linux 9.x).

I believe that one of the macOS updates sometime since Jan 2018 (I forget exactly when) replaced OpenSSL with LibreSSL--this info might prove useful to someone here trying to diagnose things.

The output of brew list; brew cask list from a failing machine might be slightly useful. If somebody needs this same list from a working macOS boxen, I can try to assist next time I'm back in front of one.

[tjones2301]

I'm seeing this issue as well on macOS High Sierra (10.13.3). Brew installed, downloaded versions, and from source have the same behavior:

$ ./packer-1.2.4/packer build -var-file us-ashburn-1_vars.json test-image.json 
oracle-oci output will be in this color.

==> oracle-oci: Creating temporary ssh key for instance...
==> oracle-oci: Creating instance...
==> oracle-oci: Problem creating instance: Post https://iaas.us-ashburn-1.oraclecloud.com/20160918/instances: x509: certificate signed by unknown authority
Build 'oracle-oci' errored: Problem creating instance: Post https://iaas.us-ashburn-1.oraclecloud.com/20160918/instances: x509: certificate signed by unknown authority

==> Some builds didn't complete successfully and had errors:
--> oracle-oci: Problem creating instance: Post https://iaas.us-ashburn-1.oraclecloud.com/20160918/instances: x509: certificate signed by unknown authority

==> Builds finished but no artifacts were created.

The cert chain that iaas.us-ashburn-1.oraclecloud.com:443 presents appears correct.

subject= /C=US/ST=California/L=Redwood Shores/O=Oracle Corporation/CN=iaas.us-ashburn-1.oraclecloud.com
issuer= /C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4
serial=5B5D09A5C619F72232811E70BE0769BC
notBefore=Apr 22 00:00:00 2017 GMT
notAfter=Apr 22 23:59:59 2019 GMT

subject= /C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4
issuer= /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
serial=513FB9743870B73440418D30930699FF
notBefore=Oct 31 00:00:00 2013 GMT
notAfter=Oct 30 23:59:59 2023 GMT

Though oddly enough I don't see these errors in uk-london-1 (CA root is digicert).

Is there a clear way to understand what CA Roots are being used?

What's really odd is the dowloaded version of 1.1.3 works fine, however when I built 1.1.3 I see similar issues as newer versions. Is this a regression how crypto/x509 loads system certs?

[rainyday]

Getting this issue on every version downloaded from both packer.io and homebrew.bintray.com. Have tried 1.1.3, 1.2.2 and 1.2.5. Same issue on all of them.

[adamday2]

On OSX 10.13.6, I'm seeing these same issues on 1.3.2, but 1.1.3 works fine.

2018/10/31 13:27:27 [ERR] Checkpoint error: Get https://checkpoint-api.hashicorp.com/v1/check/packer?arch=amd64&os=darwin&signature=<...>&version=1.3.2: x509: certificate signed by unknown authority
2018/10/31 13:27:27 packer: 2018/10/31 13:27:27 [ERR] Checkpoint error: Get https://checkpoint-api.hashicorp.com/v1/check/packer?arch=amd64&os=darwin&signature=<...>&version=1.3.2: x509: certificate signed by unknown authority
2018/10/31 13:27:27 packer: 2018/10/31 13:27:27 [ERR] Checkpoint error: Get https://checkpoint-api.hashicorp.com/v1/check/packer?arch=amd64&os=darwin&signature=<...>&version=1.3.2: x509: certificate signed by unknown authority
...
Build 'azure-arm' errored: Request failed: subscriptions.Client#Get: Failure sending request: StatusCode=0 -- Original Error: Get https://management.usgovcloudapi.net/subscriptions/<...>?api-version=2016-06-01: x509: certificate signed by unknown authority

[SwampDragons]

Is this still an issue for anyone?

[zetas]

I've just been fighting with this for the last 3 hours. I finally fixed It though I'm not sure which of my steps actually did it. This was my error any time I combined a post-processor with a provisioner. If I removed one or the other it worked fine but when I had both while working with digital ocean I got this:

```
digitalocean: output will be in this color.

Build 'digitalocean' errored after 10 seconds 49 milliseconds: DigitalOcean: Unable to get regions, Get https://api.digitalocean.com/v2/regions?page=1&per_page=200: net/http: TLS handshake timeout

==> Wait completed after 10 seconds 49 milliseconds

==> Some builds didn't complete successfully and had errors:
--> digitalocean: DigitalOcean: Unable to get regions, Get https://api.digitalocean.com/v2/regions?page=1&per_page=200: net/http: TLS handshake timeout

==> Builds finished but no artifacts were created.
````

Infuriating. I went thought the process of double checking my config, I even paired down the post processor to nothing more than a shell-local echoing foo and still I got the error, here are the steps I took and which one seemed to actually solve it:

1. Updating go. I had 1.14 so I upgraded to latest which for OS X/homebrew is 1.15 using brew upgrade go
   
   
   
   • still got the same error
   
   
2. Disabled all SSL checking everywhere I could in the packer json just in case
   
   
   
   • error persists
   
   
3. Attempted to upgrade packer using brew upgrade hashicorp/tap/packer. No upgrades available, this makes sense since I literally installed it a few hours ago following these docs but I'm checking the refrigerator for my keys at this point.
   
   
   
   • the error taunts me now
   
   
4. I realized thats a weird path to install something via home-brew and I've never seen it before so I uninstall the hashicorp/tap/packer (which was version 1.6.2) using brew uninstall hashicorp/tap/packer and then install it without the prefix: brew install packer. Still packer version 1.6.2 but now all the sudden IT WORKS.

No idea why that was the case and I'm not sure if it was a cumulative effect of upgrading go and all the other changes as well but it works FFS so maybe try that if you're still having issues. For the record i'm on Catalina 10.15.6

Thank home-brew Jesus for this win.

[SwampDragons]

Wow thanks for the details @zetas

I will look over our docs about installing via homebrew and try to figure out why we have such a weird path.

[SwampDragons]

Ah, the path in our docs is to the hashicorp-maintained tap, not the homebrew-maintained one. There are some differences in build flags. I wonder if that's what's going on here. I can look into some differences and see if I can figure out why one works and the other doesn't. For context on why Hashicorp does this, see here: https://github.com/hashicorp/homebrew-tap