Terraform-provider-google: Google Cloud Instance Startup Scripts & Provisioning with Compute Templates

[hashibot]

_This issue was originally opened by @jwmarshall as hashicorp/terraform#10685. It was migrated here as part of the provider split. The original body of the issue is below._

---

So I'm familiar with Terraform and have built environments on AWS with it. I have a new project on google cloud and I'm having an issue (or misunderstanding) on how to get my provisioning scripts/steps into an autoscaling group.

Terraform Version

Terraform v0.7.13

Affected Resource(s)

• google_compute_instance_template

Terraform Configuration Files

resource "google_compute_instance_template" "events_service_template" {
  name = "${var.environment}-events-service-template"
  machine_type = "${var.machine_type}"
  can_ip_forward = false

  tags = []

  disk {
    source_image = "ubuntu-os-cloud/ubuntu-1404-lts"
    disk_type = "pd-ssd"
    disk_size_gb = "30"
    auto_delete = true
    boot = true
  }

  network_interface {
    network = "default"
  }

  metadata {
    ssh-keys = "root:${file("${var.public_key_path}")}"
  }

  service_account {
    scopes = ["https://www.googleapis.com/auth/compute.readonly"]
  }

  provisioner "file" {
    source = "scripts/startup.sh"
    destination = "/root/startup.sh"

    connection {
      type = "ssh"
      user = "root"
      private_key = "${file("${var.private_key_path}")}"
      agent = false
    }
  }

  provisioner "remote-exec" {
    connection {
      type = "ssh"
      user = "root"
      private_key = "${file("${var.private_key_path}")}"
      agent = false
    }

    inline = [
      "chmod +x /root/startup.sh",
      "/root/startup.sh"
    ]
  }
}

Output

Error applying plan:

1 error(s) occurred:

* ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

References

I used a modified version of the google two-tier example.

[meangrape]

Hi, there!
I'm interested in the same functionality...I can make templates just fine. But I need to provision the instantiations separately -- for instance, I pull my consul configuration from a remote state (generated by the consul servers instantiation).

Do I need to try and shoe-horn everything into the google_compute_instance_template startup scripts using terraform templates?

[danawillow]

This is a fairly old issue but I thought I'd take a look and see what I could come up with.

As mentioned in the comments of the originally-filed issue, the sample configuration in question is trying to remote-exec on an instance template. Instance templates themselves don't have any sort of ssh ability; it's the instances created from those templates that do. Hypothetically I could see an FR to add this type of support to instance_group_manager, since that creates instances. However, that doesn't sit great with me- you aren't really sshing into the manager, what you want is to ssh into the instances. Also, Terraform only allows setting one "connection info" in state per-resource, so it's unclear which instance of the provisioned would be the one that gets connected to.

Because of this, I'm going to go ahead and close this issue. However, if anyone feels strongly about it please speak up and I'm happy to reopen it.

cc @jayed, @jwmarshall

[dkpat12]

This feature would be extremely useful to me. I need to add a file (not a startup script, which I can use the metadata tag for) to all the instances in the managed group for their use during the startup script.

[nmaludy]

Ran across this too. Need to ssh in to perform some post-boot customizations to the image.

[southwel]

+1 this would be lovely.

[ndmckinley]

I've been thinking about how to build this. I think the technique here would be to create an instance from the template, run the provisioner on that, then create an image from that instance, then create a second template from that image. Does that seem right? If so, I don't think there's work on the provider-side to enable that - anything we did on our end to reduce the complexity in configuration there would have to perform roughly those same startup steps behind the scenes, and it seems to me that explicit is better than implicit.

An alternative would be to construct a way to run the provisioners on every instance created by the template. You might be able to rig this up with wait_for_instances = true, and an array of null resources which depends_on each of a target pool's instances. I'm not sure that that would work, but I think it would.

[paddycarver]

I'm pretty against adding this functionality. The docs are pretty clear that you should not be relying on the state of disks:

Because of the stateless nature of managed instance groups, you should design or retrofit your application so that it does not depend on specific instance properties that will not persist, such as an IP address or in-memory data. Likewise, the default behavior for boot persistent disks is to delete them when the corresponding VM instance is deleted so you should not rely on boot disks as persistent data in a managed instance group.

If you need to customize an image after boot, the appropriate place to do that is in the startup script. remote-exec is pretty much never appropriate with an instance group manager, because the instance group manager is managing the instances, not Terraform. That means instances may be autoscaled, deleted, added, or restarted without Terraform ever knowing about it, and remote-exec can't run in those situations.

[meangrape]

Thanks for your comments and see how it probably would end up being an ugly kludge.

[ndmckinley]

Thanks! I'll go ahead and close the issue.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!