Terraform: Delayed execution of reading for file variables

[lusis]

In an attempt to build some really reusable modules, we've broken a few things out. One thing we need is the ability to generate an AWS keypair for each environment we stand up. I discovered the null resource and thought it would help by wrapping the ssh-keygen process up into a resource that could then be used for aws_key_pair. Evidently this doesn't appear to work as ${file("foo")} contents are required to be read before work can begin:

resource "null_resource" "generate_keypair" {
    provisioner "local-exec" {
        command = "ssh-keygen -b 4096 -f ${path.cwd}/${var.spath_orgname}-deployer-key"
    }
}

resource "aws_key_pair" "deployer" {
  key_name = "${var.spath_orgname}-deployer-key"
  public_key = "${file("${path.cwd}/${var.spath_orgname}-deployer-key.pub")}"
  depends_on = ["null_resource.generate_keypair",]
}

terraform plan and terraform apply both fail as the file does not yet exist (apparently though I've not dug into the code yet) for reading. This is a normally safe sanity check so I totally understand WHY.

Is there an option for lazy evaluation or is there a workaround that involves a dance around generating the keypair files locally and then some how cat'ing them out into a variable?

[apparentlymart]

A possible hack around this would be to use the template_file resource with depends_on your null_resource, treating the file as a template that happens to have no interpolations in it.

Having a resource for reading files is probably the only way to make this delayed read work within the current config mechanism, since functions can't express dependencies.

[phinze]

Yep @apparentlymart's suggestion is how I'd do it - use template_file to get your file treated as a node on the dependency graph.

[afiune]

@lusis @apparentlymart @phinze Hey guys... I happened to find this topic very useful. I tried the suggestion myself but I might be doing something wrong. :smile:

Having this code:

# Generate key
resource "null_resource" "generate_key" {
  provisioner "local-exec" {
    command = "ssh-keygen -t rsa -N '' -b 2048 -f my_key -y"
  }
}

# Template to delay action of reading the generated key
resource "template_file" "my_key" {
  depends_on = ["null_resource.generate_key"]
  template = "${my_key}"
  vars {
    my_key = "${file("my_key")}"
  }
}

I would imagine from here I can just use "${template_file.my_key.rendered} but I am still having an error trying to run plan:

$ terraform plan
There are warnings and/or errors related to your configuration. Please
fix these before continuing.

Errors:

  * file: open my_key: no such file or directory in:

${file("my_key")}

Am I doing something wrong? (using Terraform version v0.6.9)

[HX-Rd]

@afiune Yeah having the same "no such file or directory in" problem. Looks like there is a check done on the paths in the variable interpolation in a phase before the actual run. If you create a empty "my_key" before the terraform apply, that key will be overwritten and used. Ugly hack but the only one that I have come up with to solve this.

[afiune]

@HX-Rd It is exactly what I did. But it is very ugly indeed. Thanks for the answer though! 👍

[rneu31]

@apparentlymart @afiune @HX-Rd @phinze Does anyone know if this "hack" is still functional in the newest Terraform? I'm having quite the time trying to do something like this.

I am trying to have an md5 generated for a directory each time terraform runs, save the hash to a file, and use that file's contents to determine if provisioning needs to happen. If this is a far off use case feel free to let me know.

My problem seems to be the policy-hash-file gets refreshed first. If I run twice in a row, the second time will pick up the necessary changes. Verified this by inspecting the state file directly.

resource "null_resource" "hash-policy" {
  provisioner "local-exec" {
    command = "find ../../../../policy -type f -exec md5sum {} ; | sort -k 2 | md5sum > policy-hash.txt"
  }

  triggers {
    uuid = "${uuid()}" # trigger always
  }
}

data "template_file" "policy-hash-file" {
  depends_on = ["null_resource.hash-policy"]
  template = "${file("${path.root}/policy-hash.txt")}"
}

resource "null_resource" "run-chef-client" {   
  provisioner "remote-exec" {
    # run chef client
  }

  triggers {
    random_file =  "${data.template_file.policy-hash-file.rendered}"
  }
}

This is all an elaborate scheme to run Chef when contents of a specific folder changes.

Thanks.

[apparentlymart]

In order for the hack to work it is necessary to use the older (now deprecated) form of template_file that takes a filename directly as an argument, rather than using the file function.

In the future I'd like to be able to address this sort of problem using the mechanism added by #8768.

[irvingpop]

So I got this working in Terraform 0.9.10 with some slight tweaks on the above technique. Namely by using a combination of a computed resource for the filename and by depending on an intermediary resource for data.

It looks like this:

resource "null_resource" "generate_chef_keypair" {
  provisioner "local-exec" {
    command = "ssh-keygen -t rsa -N '' -f .chef/delivery-validator-${random_id.automate_instance_id.hex}.pem"
  }
}

resource "aws_instance" "chef_server" {
  # resources to use the pre-created key
  depends_on = ["null_resource.generate_chef_keypair"]
}

# template to delay reading of validator key
data "template_file" "delivery_validator" {
  template = "${file(".chef/delivery-validator-${random_id.automate_instance_id.hex}.pem")}"

  vars {
    hacky_thing_to_delay_evaluation = "${aws_instance.chef_server.private_ip}"
  }
  depends_on = ["aws_instance.chef_server"]
}

# other resources that use ${template_file.delivery_validator.rendered}

of course this will surely break in 0.10.x :)

[irvingpop]

Fellow travelers who land here, an update.

As expected the hacky thing above (left for posterity) did in fact break with 0.10.x, but the External Data Source provider is awesome (once you get the hang of it).

I've built an updated example here: https://gist.github.com/irvingpop/968464132ded25a206ced835d50afa6b

[berzoidberg]

Just wanted to drop in and mention that I've accomplished this using the TLS provider.

Relevant docs:
https://www.terraform.io/docs/providers/tls/
https://www.terraform.io/docs/providers/tls/r/private_key.html#

[mukund1989]

@ZoidBB That provider stores the key in statefile

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.