Packer: Packer 1.5.6+ does not use macOS system DNS resolver and cannot connect to host

[VladRassokhin]

Overview of the Issue

Got "Timeout waiting for SSH" due to Packer's inability to correctly resolve host name, details and logs below.
Had to rollback to 1.5.5.

ping -c1 ssh_host correctly resolves DNS and works.

Reproduction Steps

packer-1.5.6 build -var ssh_host=HOST.DOMAIN config.json

Packer version

Packer v1.5.6
Packer v1.6.1

Simplified Packer Buildfile

{
  "builders": [
    {
      "type": "null",
      "communicator": "ssh",
      "ssh_host": "{{user `ssh_host`}}",
      "ssh_username": "Administrator",
      "ssh_timeout": "15s",
      "ssh_agent_auth": true
    }
  ]
}

Operating system and Environment details

Mac OS X 10.14.6 18G6020
Env:

• GODEBUG="netdns=1"
• PACKER_LOG=1

Computer is connected to corporate network using VPN.
192.168.0.1 is IP address of WiFi router.

/etc/resolv.conf content

#
# macOS Notice
#
# This file is not consulted for DNS hostname resolution, address
# resolution, or the DNS query routing mechanism used by most
# processes on this system.
#
# To view the DNS configuration used by this system, use:
#   scutil --dns
#
# SEE ALSO
#   dns-sd(1), scutil(8)
#
# This file is automatically generated.
#
nameserver 192.168.0.1

scutil --dns

DNS configuration

resolver #1
  search domain[0] : DOMAIN
  search domain[1] : consul
  search domain[2] : test
  search domain[3] : DOMAIN_2
  nameserver[0] : 192.168.0.1
  if_index : 10 (en0)
  flags    : Request A records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #2
  domain   : consul
  nameserver[0] : CORPORATE_NAMESERVER_1
  nameserver[1] : CORPORATE_NAMESERVER_2
  if_index : 19 (ipsec0)
  flags    : Supplemental, Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 101001

resolver #3
  domain   : DOMAIN_2
  nameserver[0] : CORPORATE_NAMESERVER_1
  nameserver[1] : CORPORATE_NAMESERVER_2
  if_index : 19 (ipsec0)
  flags    : Supplemental, Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 101003

resolver #4
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000

resolver #5
  domain   : DOMAIN
  nameserver[0] : CORPORATE_NAMESERVER_1
  nameserver[1] : CORPORATE_NAMESERVER_2
  if_index : 19 (ipsec0)
  flags    : Supplemental, Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 101000

resolver #6
  domain   : test
  nameserver[0] : CORPORATE_NAMESERVER_1
  nameserver[1] : CORPORATE_NAMESERVER_2
  if_index : 19 (ipsec0)
  flags    : Supplemental, Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 101002

resolver #7
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300200

resolver #8
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300400

resolver #9
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300600

resolver #10
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300800

resolver #11
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  nameserver[0] : 192.168.0.1
  if_index : 10 (en0)
  flags    : Scoped, Request A records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #2
  search domain[0] : DOMAIN
  search domain[1] : DOMAIN_2
  nameserver[0] : CORPORATE_NAMESERVER_1
  nameserver[1] : CORPORATE_NAMESERVER_2
  if_index : 19 (ipsec0)
  flags    : Scoped, Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)

Log Fragments and crash.log files

Packer 1.5.5 log

==> null: Waiting for SSH to become available...
2020/08/05 12:36:28 packer-1.5.5 plugin: go package net: using cgo DNS resolver
2020/08/05 12:36:31 packer-1.5.5 plugin: [INFO] Attempting SSH connection to HOST.DOMAIN:22...
2020/08/05 12:36:31 packer-1.5.5 plugin: [DEBUG] reconnecting to TCP connection for SSH

Packer 1.5.6 log

==> null: Waiting for SSH to become available...
2020/08/05 12:38:52 packer-1.5.6 plugin: go package net: built with netgo build tag; using Go's DNS resolver
2020/08/05 12:36:52 packer-1.5.6 plugin: [DEBUG] TCP connection to SSH ip/port failed: dial tcp: lookup HOST.DOMAIN on 192.168.0.1:53: no such host
2020/08/05 12:36:57 packer-1.5.6 plugin: [DEBUG] TCP connection to SSH ip/port failed: dial tcp: lookup HOST.DOMAIN on 192.168.0.1:53: no such host

[SwampDragons]

Hi thanks for reaching out. Based on timing and on those debug lines from the net library, I'm going to guess this was caused by https://github.com/hashicorp/packer/commit/f12269f1241d9774dbc08509e92047afe3bd0e0f

We disabled cgo compilation because it started messing up static linking once we started building our release builds on linux; we didn't realize it was going to break this but it sounds like the golang net library isn't quite up to par with how cgo was for darwin.

For now, the workaround is to rebuild packer yourself without the cgo disabling. You can do this by cloning the repo, making sure https://github.com/hashicorp/packer/blob/de5c5873a227a8c268fe3846e18d28b6b08fa9ad/version/version.go#L17 has the "dev" tag set rather than an empty string, and calling make dev from Packer's root directory. This will add a new packer binary to bin/packer inside your packer root.

I'll try to figure out how to change this for future releases.

[TheRedBull]

@VladRassokhin I can confirm that building packer (latest (1.6.2)) from the repo fixes this as I attempted it last night. Incidentally, the 1.6.2 build seems to have also fixed a like hanging Ubuntu install issue I was having in my Windows lab

[VladRassokhin]

Personally I'm happy with 1.5.5 for now, though I'd like to see that somehow fixed in newest official releases. I do understand that test for such case is quite complicated.

[marco-m]

@TheRedBull for my understanding, when you say:

I can confirm that building packer (latest (1.6.2)) from the repo fixes this as I attempted it last night.

are you building from macOS ? If yes, with what command-line?

For me, macOS build, at least using make, is broken, since scripts/build.sh wants realpath that doesn't exist on macoS. Or did you brew install coreutils to get realpath ?

[azr]

@marco-m, you can try with the Nightly build: https://github.com/hashicorp/packer/releases/tag/nightly

[nywilken]

@marco-m the default make target is setup to use the Packer build script which is meant for releasing. I'll see about changing that for the future. In the meantime, the correct command to run is make dev which should compile using the standard Go build tools.

@azr I think the latest nightly releases are also using Linux with CGO disabled so DNS may still be an issue. But @marco-m you can give it a try. Otherwise the path forward, for now, is to build locally on your Mac using make dev.

[marco-m]

Thanks both @azr and @nywilken, I will try the suggestions. I don't want to hijack this thread, I might open another ticket: .github/CONTRIBUTING.md is severely out of date and gives the impression that you can build on macOS (or Windows for that matter) with a plain make, while it is not the case.

[SwampDragons]

@marco-m no need for another issue. I'll make a Pr to change the default make target to not require that script; we don't release using that command anyway.