Terraform: Terraform fails to use SSH agent on Mac OS X

[joschi]

Terraform Version

Terraform v0.11.1
+ provider.docker v0.1.1

Terraform Configuration Files

locals {
  # If using Docker for Mac
  host = "127.0.0.1"
  # If using Docker on Linux
  #host = "${docker_container.test.ip_address}"
  port = "2222"
  pubkeys = ["${file("id_rsa.pub")}", "${file("id_rsa_encrypted.pub")}"]
}

resource "docker_container" "test" {
  # Other declarations […]

  # Add SSH public keys to authorized keys
  upload {
    file = "/root/.ssh/authorized_keys"
    content = "${join("", local.pubkeys)}"
  }

  provisioner "remote-exec" {
    connection {
      type  = "ssh"
      host = "${local.host}"
      port = "${local.port}"
      user  = "root"
      agent = "true"
    }

    inline = [
      "echo 'Hello world!' > /root/readme.txt"
    ]
  }
}

For completeness, the output of ssh-add -L:

$ ssh-add -L
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDMmGshVJkhWskSnk2v2KvPRXh53Yqh7C7pcbEWywnTDz1U92a7GbV8nwTcGSXQPP918qvsb/8LRR6gYSqD4wtUaFSJTmk6FOguJiBCUh7Pt5fNVJZOPkjC3LOeO69aoB9x7IhesSxTDtCInusUlWKXoCbvL9GFC2MUKMcK2Yw3zoY/M8rJJyw5GfJ+X1OXA0Wd2StlP/Enwl+tUeb06ZNMPG37Tx6/Puh+sy7y5uy6i8cOMxkK7yAYOL0gyI61+J+ebY58pwdP3hPO2PhKjR/xnn4Ha4i8VXlRUE+kTNPCI69KxbA9H064pMcyLN5zHepGyeHZP14EptBBUjWWcBz id_rsa
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDToUTwvcvpZ3AzmmKA1SG4Gn/0kIQpWMGK6BJLv35XApTE7cSAPLgVGTZKlSi8BgfPuhfcHLaS/dpqpiZ0aJi2hMPK9gGQGghHHEi3i24ouo25gBUr5Xd88YhsPBt5HTytulvg+C13d6o5+e0P+01DRZh2x9R4h/Lbv1SQF2t2ZOhY4/YD18zwh3EBMHQ7uYYy/28zxmTDCOwb3qZ+K9DmZPpvIui4EsL/wv72cv1C0C+InqsU0Ps3zhUbmBl7/QsHiisezkCcwJ4haH98vgIUjp8sa9uTT8crBMAnYT4VE5PF6iiBEhUA2a4h/SBLhNRRxUa3wPRvktX3OP0C+bWN id_rsa_encrypted

Debug Output

https://gist.github.com/joschi/b031fe7d37119bc7ddc2a719219b9717#file-terraform-log

Expected Behavior

The remote-exec provisioner should be able to log into the created Docker container via SSH with the keys provided by the running (and otherwise working) SSH agent.

Actual Behavior

The remote-exec provisioner fails to log into the Docker container until Terraform times out and aborts.

Manually logging into the Docker container (ssh -l root -p 2222 -i id_rsa_encrypted 127.0.0.1) works.
Deactivating the use of the SSH agent (remote-exec.connection.agent = "false") and explicitly using a private key instead ( remote-exec.connection.private_key = "${file("/path/to/id_rsa")}") also works (but doesn't support encrypted private keys and needs to know the location of the private key).

Steps to Reproduce

1. Start Docker daemon
2. ssh-add id_rsa id_rsa_encrypted
3. terraform init
4. terraform apply

Basically see https://gist.github.com/joschi/b031fe7d37119bc7ddc2a719219b9717#file-readme-md

References

• https://gist.github.com/joschi/b031fe7d37119bc7ddc2a719219b9717#file-readme-md

[joschi]

FWIW, this isn't provider-specific and also fails with the Google Cloud provider. Docker just seemed simpler to reproduce the issue.

[jbardin]

Hi @joschi,

Thanks for the very complete set of steps here. I know that in most cases the remote-exec ssh connection works correctly with the macOS ssh-agent, and your configuration works as expected on my system as well.

If the agent weren't being used, I would expect the failure to be ssh: unable to authenticate, attempted methods [none]. In your case the failure is a disconnect with too many authentication failures.

I suspect that you have more than the 2 keys loaded in your agent than what you've shown here, and that the id you want is at the end of a longer list. Can you confirm how many keys are shown from ssh-add -l?

[joschi]

I suspect that you have more than the 2 keys loaded in your agent than what you've shown here, and that the id you want is at the end of a longer list.

@jbardin Yes, that's correct.

$ ssh-add -l|wc -l
11
$ ssh-add -l|tail -n 2
2048 SHA256:NzXGAaH4UJtVrXr2BULdqUy7+F1ejN7EA9uqBaX9w7o id_rsa (RSA)
2048 SHA256:69tX3qCePqqRwjOfg0IMBbQ07FXoQdzvhFLIZnbzJ90 id_rsa_encrypted (RSA)

The last 2 listed keys are the ones used for the demo.

[jbardin]

Thanks for the update.

The default sshd_config setting for openssh has a MaxAuthTries of 6. Any time the agent has more than 6 keys, there's a possibility that you may run out of chances to authenticate before you get to try the key that works. This is the same behavior as the openssh cli client, though it may try keys in a slightly different order.

[joschi]

@jbardin Makes sense.

Would it help to add some "hints" to the ~/.ssh/config file, e. g. a Host block which lists the correct IdentityFile for the respective hosts?
In other words, does Terraform use the SSH client installed on the machine so that changes to the OpenSSH configuration have an effect in the Terraform remote-exec provisioner?

[jbardin]

No, terraform doesn't use the local ssh binaries at all. I'll mark this as a feature request to see if we can support something equivalent to IdentityFile.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.