Terraform: Feature Request: create temporary files

[natefaerber]

I recently came upon terraform-providers/terraform-provider-consul#5 that led me to imagine a new function that "wraps a text block in a file". This basically amounts to a tempfile function where terraform manages ephemeral temporary files for use in a single run. The idea is that I could pass a variable or attribute to a resource argument that normally expects a filename using this tempfile function.

The example I provided in terraform-providers/terraform-provider-consul#5 is:

provider "consul" {
  address = "${aws_instance.master.public_ip}:8080"
  datacenter = "${coalesce(var.cluster_name, var.subnet_id)}"
  scheme = "https"

  # Use of templates is a nasty hack to force dependency ordering
  ca_file = "${tempfile(tls_self_signed_cert.consul_ca.cert_pem)}"
  cert_file = "${tempfile(tls_locally_signed_cert.consul_node.cert_pem)}"
  key_file = "${tempfile(tls_private_key.consul_node.cert_pem)}"
}

tempfile returns the path of the temporary file that Terraform will clean out at the end of its run.

This avoids the use of local_file to manage the file locally and puts the burden of managing temporary files on Terraform instead of the user. Just a thought but I wanted to follow-up my comment with this feature request to gauge the likelihood of this happening.

Regards.

[apparentlymart]

Hi @natefaerber! Thanks for this suggestion.

We've generally tried to avoid the need for temporary files by passing direct values rather than file paths where possible. I assume that isn't true here due to limitations of the underlying Consul client library, which perhaps expects to read keys from files rather than in-memory strings. (I didn't actually check, so this assumption may be wrong.)

In an ideal world I'd prefer to add to the consul provider new settings ca_cert_pem, cert_pem and key_pem that just take the strings directly, taking external files out of the equation altogether. This is more consistent with our usual approach and avoids the risk of leaking potentially-sensitive data in the temporary directory in cases where cleanup is not possible for some reason.

What do you think? Is there another reason why passing this data through temporary files is desirable, or would passing them directly in memory meet your needs here?

[natefaerber]

Only reason I suggested temporary files is because I looked at the code and Condul client can't accept a string so I was trying to avoid changes to two projects. Also, I made an assumption that this wasn't the only case where an underlying client library would force a provider to pass a file rather than in-memory strings.

By the way, I could totally do the work for adding these settings. I have many outstanding pull-requests on the consul provider. How do I get someone with authority to review them? Also, I made many comments to the issues in that project that could allow closing the tickets. Sorry to hijack the discussion.

[BernhardBln]

I'd need this for the archive_file data source - when zipping a js file for an AWS lambda.

I'd love if it would be possible to use that without output_path, and then just refer to data.archive_file.lambdazipped.zippedcontent, but that's not possible...

Now the zip file ends up in my source folder :-/

[whyman]

:+1:

[apparentlymart]

Hi @natefaerber! Sorry for the silence here, and on your issues in the Consul provider.

The Consul provider is now maintained by the Consul team themselves, to allow it to better keep up with new Consul features. Looking over there I see that in the mean time your PRs have been merged; I expect the delay you saw was prior to the transition of responsibility to the Consul team, where the Consul provider was unfortunately competing for attention with many of the other providers that our provider development team looks after.

As for the new settings we were talking about: I assume it would first need a change to the Consul client library, but at least coordinating such a thing should be easier with the same team looking after both codebases. That'd be a couple changes over in the Consul repository and the Consul provider repository rather than here, I assume.

As for temporary files _in general_: I think a function isn't the right way to implement that because we'd have no way to actually clean it up afterwards (functions don't have a "lifecycle" to implement a cleanup step in) but we'll use this issue as a marker for the general use-case of creating temporary files, or perhaps alternatively for creating temporary _directories_ that you could then write files into in other ways (provisioners, etc). This will probably require a new concept in Terraform altogether, of something that lives only for a single run, which might also generalize to include the use-case described in #8367 but not sure yet about that.

We won't be able to look into this right now due to attention being elsewhere, but I'll mark it as needing further design effort and we'll return to it when we're able.

(In the mean time, if you want to "vote" for this to influence priority, please do so by leaving a :+1: _reaction_ on the original comment rather than leaving new comments, since we can report on which issues have the most thumbs-up reactions while comments just create noise for anyone following the issue.)

[apparentlymart]

Ahh, I recently opened #21308 and I knew when I was writing it that there was another issue for this out there somewhere but I couldn't find it at the time.

For the moment I'm just going to connect the two with a link, because there are lots of votes on this one but there's a recent design proposal on the other one. We're not planning to take any immediate action on this due to priorities being elsewhere, but once we do I expect we'll close one of these issues to consolidate. For now, my issue #21308 can be thought of as a specific technical proposal to meet the use-case described in this issue.

[jemc]

My use case is a bit different from the OP, so it may add some justification.

Because Terraform's Kubernetes provider doesn't support arbitrary manifests, the most common approach in the community seems to be hacking it in with a local-exec provider, like this:

locals {
  my_kubernetes_yaml = templatefile("my_kubernetes.yaml", {
    # ...
  })
}

resource "null_resource" "kubectl_apply_my_kubernetes_yaml" {
  triggers = {
    my_kubernetes_yaml = "${local.my_kubernetes_yaml}"
  }

  provisioner "local-exec" {
    command = "kubectl apply -f - <<'EOF'\n${local.my_kubernetes_yaml}\nEOF"
  }
}

The issue is that for a certain application (that expects a very large config file in a configmap manifest), the string local.my_kubernetes_yaml ends up huge (141301 bytes), which results in a shell error that the argument list to the shell is too long:

fork/exec /bin/sh: argument list too long.

Because the large config file in question needs to be one contiguous file, there is no way to split it up among multiple invocations. In such a situation, it seems I am forced into using a temporary file because I have no way to get the manifest into the kubectl apply invocation.