Vault: Vault 0.3.1 does not respect Mac OS X VPN DNS resolution.

[hobbeswalsh]

I'm not sure what flags you used when you built Vault, but it appears that DNS resolution does not work as it did before.

➤ /usr/local/bin/vault version; /usr/local/bin/vault status
Vault v0.3.1
Error checking seal status: Get https://vault.mydomain.com/v1/sys/seal-status: dial tcp: lookup vault.mydomain.com on 192.168.86.1:53: no such host

192.168.86.1 is my home's router.

➤ ./vault-0.2.0 version; ./vault-0.2.0 status
Vault v0.2.0
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0

High-Availability Enabled: true
    Mode: active
    Leader: https://10.128.1.36:8200

What's going on here? Did you use the right netgo flags when 0.3.0 was compiled? Is there any way I can fix this?

Thanks!

[jefferai]

I don't know much about the OSX DNS subsystem, but my guess is that this is related to the fact that 0.2 was built on OSX and incorrectly created a dynamic build rather than a static build. In 0.3.1 (0.3 on all platforms except amd64_linux) we are forcing a static build, which means that cgo is not enabled at build time. This may change resolver behavior as it means that native Go DNS resolution is being used.

From searching around it seems that you can adjust this at runtime using GODEBUG=netdns=cgo. Does that fix the problem?

[hobbeswalsh]

@jefferai Thanks. I tried that and it doesn't fix the problem.

I was able to pull the latest version of the Hashicorp vault repo, check out the v0.3.1 tag, and go install that version, which I can then recommend that the rest of the employees at the company download and use, but this doesn't strike me as a particularly desirable solution.

➤ echo $GODEBUG
netdns=cgo+2
➤ ./vault auth -method=ldap username="Robin Walsh"
Password (will be hidden):
go package net: built with netgo build tag; using Go's DNS resolver
go package net: hostLookupOrder(vault.mydomain.com) = files,dns
Put https://vault.mydomain.com/v1/auth/ldap/login/Robin%20Walsh: dial tcp: lookup vault.mydomain.com on 192.168.86.1:53: no such host

[jefferai]

Hi @hobbeswalsh ,

In GODEBUG you have netdns=cgo+2 -- is that +2 on purpose? Does removing that change anything?

[hobbeswalsh]

You can combine flags, as per https://golang.org/pkg/net/

[jefferai]

Oh, I see -- the plus sign denotes a debugging level (multiple debug flags are joined by commas, so I was confused).

Given that netdns=cgo didn't work, does netdns=go work? That should be what the binary is doing anyways, but it's worth a try.

[jefferai]

I don't know all that much about how Go does name resolution but I'm wondering if the runtime flag they indicate only actually works if the binary was built with cgo enabled in the first place. It's not what the documentation says, but it might just not be clear or correct, if using the cgo resolution method requires being dynamically linked to some host resolver library (and it doesn't pull that into the static build, which it could).

[ProbablyRusty]

I am seeing this issue as well in Vault 0.5.1.

I can confirm that issue does _not_ occur in Vault 0.2.1.

I have not yet tested on versions > 0.2.1 and < 0.3.1, but it is clear the issue was introduced somewhere in one of those versions, if not in 0.3.1 itself.

[ProbablyRusty]

Okay, I just did more testing:

The issue does _not_ exist in Vault v0.2.1-dev (a88fd9236522ae247ed4918998739d2852e2daf2).

The issue does _not_ exist in prior Vault versions (ie: 0.2.0 and earlier).

The issue _does_ exist in Vault v0.3.0-rc (76b22ca24396ab720f4e5029d40872b64cb0d79b), and every version after that which I tested, including 0.5.1.

[jefferai]

It's not a Vault issue, it's a Go issue. The binaries we distribute are built statically, hence it uses Go's internal DNS resolver. We do not plan on changing the build method, however, you can rebuild yourself using dynamic linking to see if that helps.

See the Name Resolution section in https://golang.org/pkg/net/ for more details. You can change this behavior at runtime, but I believe only if the program was built dynamically in the first place.

[thaterikperson]

For anyone on macOS who has DNS issues with vault installed from homebrew, and is Googling to figure out what's wrong, here's your solution as of today.

brew install go
mkdir -p ~/.golang/src/github.com/hashicorp
if [ -z $GOPATH ]; then
  echo "export GOPATH=~/.golang" >> ~/.bash_profile
  echo "export PATH=$GOPATH/bin:\$PATH"  >> ~/.bash_profile
  source ~/.bash_profile
fi
git clone https://github.com/hashicorp/vault.git ~/.golang/src/github.com/hashicorp/vault
WD=`pwd`
cd ~/.golang/src/github.com/hashicorp/vault
# Important part is the `make dev-dynamic`.
# That sets a variable that forces Go to use the cgo DNS resolver.
make bootstrap && make dev-dynamic
cd $WD

I don't know what the standard is for a GOPATH, but that's what I used and now I can finally use vault on my Mac.

[ProbablyRusty]

Or, since it seems you are already using Homebrew to install Go, you could instead simply do this:

brew install vault --with-dynamic

See: https://github.com/Homebrew/homebrew-core/pull/7238

[mindfulmonk]

Sadly --with-dynamic is no longer supported https://github.com/Homebrew/homebrew-core/issues/33507

I've setup a tap for now https://github.com/mindfulmonk/homebrew-tap

[silverbp]

or...

use a wrapper script called 'vault' that is in your path and uses docker..

#!/bin/bash

VAULT_FILE="$HOME/.vault-token"

docker run --rm -it \
    -v $(pwd):/workspace \
    -v $VAULT_FILE:/root/.vault-token \
    -w /workspace \
    -e VAULT_ADDR={your vault address} \
    vault:1.3.0 /bin/vault "$@"

from a devops standpoint, we are just trying to have a bin repository of executable scripts like the above that a new devops person clones down and then installs docker.

once you install docker and it does the initial pull of the image, you don't even know it's running in docker from a usability standpoint and the DNS issues is gone :).

[silverbp]

I updated that bash script to be the following:

#!/usr/bin/env bash

VAULT_FILE="$HOME/.vault-token"

docker_args=" \
    run --rm -it \
    -v $(pwd):/workspace \
    -v $VAULT_FILE:/root/.vault-token \
    -w /workspace \
    -e VAULT_ADDR={your vault address} \
    vault:1.3.0 /bin/vault $@ \
    "

if [ -t 1 ] ; then
    docker $docker_args
else
    docker $docker_args  | tr -d '\r' | cat
fi

This handles scenarios of when you try piping the output to something like jq. Docker returns carriage returns, so it detects if you're piping it and removes the carriage returns and runs it through cat for formatting.

[mshkrebtan]

It is sad, indeed, that Homebrew does not support --with-dynamic.

But we can compile Vault from source code ourselves, right? Luckily, the dev-dynamic target is still there in the Vault's Makefile.

eval "$(go env | grep GOPATH)"
mkdir -p "${GOPATH}/src/github.com/hashicorp" && cd $_
git clone https://github.com/hashicorp/vault.git
cd vault
# Checkout a specific version if you want
git checkout $git_tag
make bootstrap
make dev-dynamic

You will find the Vault binary in the ${GOPATH}/bin or in the $GOBIN folder.
You can make a symlink to the binary:

ln -s "${GOPATH}/bin/vault" /usr/local/bin/vault

Or just add the Go bin folder to your $PATH.