Terraform-provider-digitalocean: [FreeBSD] DigitalOcean SSH Keys Not Attached to New Droplets

[mkeen]

Hi,

I'm unable to attach an SSH key to a new DO droplet using Terraform. I've done some digging and there does appear to be a discrepancy between DO API documentation, and their own console implementation. I am able to attach my key just fine from the DO console, but cannot do it at all through Terraform. See below details.

Terraform Version

```[user@machine terraform]$ terraform -v
Terraform v0.12.0

• provider.digitalocean v1.3.0
• provider.local v1.2.2
• provider.null v2.1.2
• provider.template v2.1.2

### Affected Resource(s)
- digitalocean_droplet

If this issue appears to affect multiple resources, it may be an issue with Terraform's core, so please mention this.

### Terraform Configuration Files
```hcl
resource "digitalocean_droplet" "salt_master" {
  private_networking = true
  backups = true
  region = "${var.region}"
  image = "${var.image}"
  name = "${var.name}"
  size = "${var.size}"
  ssh_keys = ["24688717"]
  ipv6 = false

  connection {
    host = "${self.ipv4_address}"
    user = "root"
    type = "ssh"
    private_key = "${file("/home/user/.ssh/id_rsa")}"
    timeout = "5m"
  }

  provisioner "remote-exec" {
    inline = [
      "env ASSUME_ALWAYS_YES=YES pkg install git",
      "env ASSUME_ALWAYS_YES=YES pkg install devel/py-gitpython"
    ]

  }

}

Debug Output

019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4: ---[ REQUEST ]---------------------------------------
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4: POST /v2/droplets HTTP/1.1
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4: Host: api.digitalocean.com
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4: User-Agent: Terraform/0.12.0-rc1 godo/1.10.0
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4: Content-Length: 182
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4: Accept: application/json
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4: Content-Type: application/json
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4: Accept-Encoding: gzip
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4:
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4: {
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4:  "name": "saltm",
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4:  "region": "nyc1",
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4:  "size": "512mb",
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4:  "image": "freebsd-12-x64-zfs",
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4:  "ssh_keys": [
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4:   24688717
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4:  ],
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4:  "backups": true,
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4:  "ipv6": false,
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4:  "private_networking": true,
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4:  "monitoring": false,
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4:  "tags": []
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4: }
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4:
2019-05-28T16:26:10.465-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4: -----------------------------------------------------
2019-05-28T16:26:11.679-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4: 2019/05/28 16:26:11 [DEBUG] DigitalOcean API Response Details:
2019-05-28T16:26:11.679-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4: ---[ RESPONSE ]--------------------------------------
2019-05-28T16:26:11.679-0400 [DEBUG] plugin.terraform-provider-digitalocean_v1.3.0_x4: HTTP/2.0 202 Accepted

Expected Behavior

The SSH Key specified should have been added to the new droplet, or the API should produce an error.

Actual Behavior

They SSH key is not added to the new droplet. It is successfully created, but I am unable to access the instance using SSH at any point.

I looked into this further, and while the DO API docs specify that the ssh_keys attribute should be used, and should work with either ids or fingerprints in a list, the Digital Ocean Console itself sends a different attribute, ssh_key_ids. The request that it sends looks like this:

{"droplet":{"name":"freebsd-s-4vcpu-8gb-ams3-01","size_id":110,"region_id":9,"image_id":47424258,"ssh_key_ids":[24688717],"volumes":[],"tags":[],"fleet_uuid":"f2339fec-5311-467c-91db-442db1c5b946","vpc_uuid":null}}

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply

References

• GH-159 Seems to talk about this, but was closed due to possible misconfiguration.

[eddiezane]

@mkeen thanks for the report and apologies for the inconvenience. I am able to reproduce but not sure what it's related to at the moment. About to board a plane so I'll dig in later this week.

edit:

Droplet created with doctl seems to set the ssh key fine. Next step is to check via the virtual console via the Cloud Dashboard to confirm it's not there.

doctl compute droplet create freebsdtest --region nyc3 --size s-1vcpu-1gb --image freebsd-12-x64-zfs --ssh-keys 21374091

[mkeen]

@eddiezane Thanks for the quick reply. I was able to successfully create the droplet with an attached ssl key via doctl (same command you used). A trace of the successful request results in the following being sent to DO:

"POST /v2/droplets HTTP/1.1
Host: api.digitalocean.com
User-Agent: doctl/1.0.0-dev godo/1.1.0
Content-Length: 197
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

{
"name":"freebsdtest2",
"region":"nyc3",
"size":"s-1vcpu-1gb",
"image":"freebsd-12-x64-zfs",
"ssh_keys": [24688717],
"backups":false,
"ipv6":false,
"private_networking":false,
"monitoring":false,
"tags":[]
}

Here's the terraform output again for comparison:

POST /v2/droplets HTTP/1.1
Host: api.digitalocean.com
User-Agent: Terraform/0.12.0-rc1 godo/1.10.0
Content-Length: 182
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

{
"name": "saltm",
"region": "nyc1",
"size": "512mb",
"image": "freebsd-12-x64-zfs",
"ssh_keys": [
24688717
],
"backups": true,
"ipv6": false,
"private_networking": true,
"monitoring": false,
"tags": []
}

At a glance, it appears that the only differences are the user-agent, the region (I also tried with nyc3 and got the same result, so I don't think that's significant), and that terraform is sending newlines in the ssh_keys attribute value. I can't imagine that would cause the issue, but I've seen more surprising things.

Have a safe flight ✈

[andrewsomething]

@mkeen I can tag in for @eddiezane since he's in transit. Let's try and rule a few things out. First off, the ssh_keys vs ssh_key_ids difference is just an implementation detail from the control panel. ssh_keys is indeed the correct attribute.

I notice from the debug output that you've provided that you are creating a FreeBSD Droplet. For FreeBSD, the default user name is freebsd, not root as seen in you config. I am able to successfully connect to the Droplet and run a provisioner using this connection block:

  connection {
    host = "${self.ipv4_address}"
    user = "freebsd"
    type = "ssh"
    timeout = "5m"
  }

Does that work for you?

[mkeen]

@eddiezane Actually, this appears related to Private Networking being enabled! When I disable it in my Terraform config, the ssh_key is attached successfully.

@andrewsomething both root and freebsd users work when the key is successfully attached, FYI.

The following config works just fine:

resource "digitalocean_droplet" "salt_master" {
  private_networking = false
  backups = true
  region = "${var.region}"
  image = "${var.image}"
  name = "${var.name}"
  size = "${var.size}"
  ssh_keys = [24688717]
  ipv6 = false

  connection {
    host = "${self.ipv4_address}"
    user = "root"
    type = "ssh"
    private_key = "${file("/home/user/.ssh/id_rsa")}"
    timeout = "5m"
  }

  provisioner "remote-exec" {
    inline = [
      "env ASSUME_ALWAYS_YES=YES pkg install git",
      "env ASSUME_ALWAYS_YES=YES pkg install devel/py-gitpython"
    ]

  }

}

edit: I also confirmed that the keys are not added when creating the droplet from the DO console when Private Networking is enabled.

[mkeen]

FYI this impacts all FreeBSD images.

[andrewsomething]

Thanks @mkeen This has been confirmed to be an issue with the FreeBSD images themselves. We've escalated this to the team at DigitalOcean responsible for maintaining them.

[mkeen]

Thanks guys!

[mkeen]

@andrewsomething Any way to get some visibility into what the timeline looks like upstream? We've got a nontrivial rollout we are hoping to use DO for, and this is a show stopper for us.

[mkeen]

@eddiezane @andrewsomething Any update? 👻

[eddiezane]

@mkeen apologies for the delay here.

The issue actually appears to have been present for some time but has gone unreported. We've got it reported internally and are tracking updates as the team resolves—unfortunately no ETA at the moment. Appears to be related to cloud-config itself. Trying to get it resolved with :fire:

[wadenick]

Hi @mkeen we caught your comment above and I'm just popping in for an update - recognizing we're late here - we did find a workaround a few days ago and missed the connection from our internal SEV to this Github thread. So, our apologies for that.

While we continue work on diagnosing the root cause, here's the workaround from the team in case that's still useful for you (I believe the new Droplet needs to be rebooted once before snapshot creation):

Creating a FreeBSD droplet with an ssh key and then creating a snapshot of that droplet will result in an image that does not exhibit this issue. This is the case whether the ssh key (authorized_keys) and /var/lib/cloud/intances is deleted or not.

We'll continue to work this issue to try and find the root cause but in the meantime the best workaround for VIPs and other users would be to create a snapshot from a FreeBSD droplet and use that for creates.

[mkeen]

Thanks @wadenick. I appreciate the workaround. We will try that, but still hoping for a fix soon.

[wadenick]

Hi @mkeen, just popping in with an update. An extended team in DO was just able to track down the upstream bug we are dealing with this morning:

https://github.com/cloud-init/cloud-init/commit/fff37e7dc6849fd16db504b0d338fae20a7beb39

The FreeBSD repos have not updated their packages with this fix. We've tried to contact FreeBSD team members to see when that'll be pulled into FreeBSD. We're also working with the cloud-init version directly, and have offered assistance to both. We hope to have both a workaround and a long term fix soon.

BTW, I did want to check in, has the near-term workaround we suggested worked for you?

[wadenick]

Hi @mkeen, OK we've pulled in a fixed version of the affected components and rebuilt FreeBSD. These fixes are released to production for DigitalOcean 11.3 and 12.0 FreeBSD base images. When FreeBSD gets around to fixing these and releasing officially, we'll also rev our base images again then.

If you wouldn't mind letting me know if this is working for you now, I know the whole team would appreciate it – five to six people worked on this one off and on over a number of weeks here.

And again, sorry for the delays in finding, fixing, and communicating on this one. It was a tricky one to be sure, but it did take us too long to update you and we strive to do better. I hope we haven't disappointed you thoroughly. Best, -nick

[mkeen]

@wadenick We are going to reprovision everything tonight. I'll let you know how it goes. Thank you very much for the detailed info. Always frustrating when upstream issues cause things like this. Cheers to your team for working it out. I'll confirm when we have completed reprovision.

[mkeen]

@wadenick This is working now. Thank you again.