Terraform: Allow plugins to export custom functions

[thegedge]

It would be interesting to allow the set of functions allowed in interpolations to be an extension point for plugins. For example, one could write a plugin that exports a function to encode a string into base64, split a root CIDR block into a list of smaller CIDR blocks that covers the root block, or read a string from a URL.

[thegedge]

I should note that you can create a plugin that exports resources to compute things for you right now, but I think functions would be more appropriate than read-only + compute-once resources in many/most cases.

[phinze]

Yep totally - this is something I've thought about as well. The use case is absolutely clear, but the implementation is very important to get right so we don't shoot our future selves in the foot with a difficult-to-support ecosystem of configs. I will join @mitchellh in the thinking tag here. :grinning:

[thegedge]

Just glad to hear that there's interest in this. Looking forward to what you come up with :)

[steve-jansen]

+1 on CIDR functions

[apparentlymart]

Heh... this is the first time I came across this issue, but funny that two of your three example use-cases have been implemented in the mean time:

• #3325 implemented Base64 encoding
• #3127 will implement CIDR manipulation once it's merged

No URL parsing yet, but I think that's suitably generic that it could be in core too.

I think continuing to add things to core Terraform will be sufficient at least until a function comes along that is truly specific to one particular set of resources, e.g. a function for parsing AWS IAM policy documents.

[polvoazul]

Why not support a "sh" function in variable interpolation, and then the user can run anything he wants?

[theist]

This is also something I'll like to have. My use case is to remove account section and region section of AWS ARNs to store them in policy documents. This is provider specific and can not be a generic function, and I want this because this allows to reuse policies with resources across regions.

For example if I use in the resource section "${aws_sqs_queue.my_queue.arn}" I get a computed ARN like 
arn:aws:sqs:us-west-1:000111223334455:my-queue and I want something to transform it to arn:aws:sqs:::my-queue. This fine string manipulation looks good for a plugin (imho)

[jhoos]

As another example usecase: we have a standard set of tags that we like to apply across all of our AWS resources, and the values of some of the tags vary per-resource (usually on the names of things). Being able to create a function that returned a map of those tags would be an effective way to make sure we're tagging everything consistently.

For example, something like mytags("myenv", "myService") which could return { environment = "${arg.env}", name = "${arg.service}", selector="${arg.env}-${arg.service}" } so that for each AWS resource we could simply add tags = mytags("myenv", "myService") to each resource and be done with it. Doing this with a data source or module would be unwieldy.

[apparentlymart]

It seems that this issue has grown to represent a couple different use-cases. That's fine (we may choose to split them later if one gets addressed first, but we'll see), but I just wanted to write them down here for future reference:

Functions as part of a provider plugin

A few times now we've seen situations where it would be handy to have a function that does something provider-specific, like parsing an AWS ARN. It's felt weird to add such functions to Terraform Core, but they could potentially be at home in the aws provider itself.

A possible design we considered for this is to extend HCL with a "function namespace" syntax, allowing extension functions to be placed in a namespace named after the provider itself. For example, if we choose :: as the function namespace delimiter then we might see a function aws::parsearn for parsing AWS ARNs.

This is not something we plan to do in the very near future because it may require some changes to HCL and so we want to make sure those changes feel right for HCL in general (since HCL is used by more than just Terraform) before moving forward with it. But we do see how it would be useful, and would like to do something like it eventually.

User-defined functions in configuration

With the above implemented, you could potentially write a provider plugin that _only_ includes functions and use that to deal with user-defined functions, which are truly specific to a particular configuration or set of configurations and make no sense to share with others.

However, we've also thought about offering some syntax for this inside the configuration language itself, allowing local functions to be written as well as local values.

Some challenges/questions down this path are:

• Once user-defined functions are possible, it seems inevitable that we'll want to enable sharing of them between modules and between configurations, so we need to figure out how to define them to enable such re-use. Perhaps they are exported as part of a module? or perhaps there's some other new reusability construct for functions that doesn't also imply the possibility of creating infrastructure?
• Are functions written in HCL itself, or in some other language? (Keeping in mind the general restrictions in the section below.)
• Do user-defined functions participate in the dependency graph? Functions being in the dependency graph is not part of the Terraform model today -- they run only as part of configuration evaluation. In order to avoid a drastic redesign of how evaluation works in Terraform (which may not actually be tractable in practice) we may have to limit functions to operating only on their own arguments, and not anything else in the configuration.

General constraints on extension functions

However they are implemented, there are some constraints that must hold for functions in Terraform's current model. While some of these may be able to change slightly, I think for initial design we should assume they are fixed design constraints:

• All functions in Terraform must behave as pure functions. If this constraint is not followed, Terraform cannot keep its promise that the actions taken during apply will match the actions planned (unless there is an error). Terraform has consistency checks that are likely to detect impure function behavior, but they are not very user-friendly so Terraform's existing functions are designed to behave as pure, at least under normal operating conditions.
• Functions must be local-only and not make any external API calls. Terraform evaluates functions at various points during its work and expects function evaluation to be relatively cheap in time and money.
• Functions cannot participate in the dependency graph, as described in the previous section. As a consequence, it's likely that we'd restrict them to produce results based only on their own arguments, and not on anything else that might normally be interpolatable from elsewhere in the configuration.

As a consequence of the above restrictions, I expect that functions-in-providers support would be implemented by running the functions in an unconfigured provider context, constrained similarly to the context used for validation, and that user-defined functions would be implemented in HCL itself or some other language that can guarantee "pure function" behavior.

---

As was implied in other earlier comments above, this is one of those issues where the _design_ of it is the hardest part -- making sure whatever is added is reliable, sustainable, and interacts well with other features -- with the subsequent implementation then probably _relatively_ straightforward. Therefore we're going to keep "thinking" on this for now, representing that there's still some more design work to do. We've been focused on configuration language work for a long time now, so I expect in the very near future we're going to cast our attention into some other areas that have been somewhat neglected during development of v0.12.0. We do still intend to return to this problem of extensible functions eventually, though.

[pll]

@jhoos I've accomplished this by doing something like the following:

tags = "${merge(var.tags, map("Name", "my unique name"))}"

[cc-stjm]

A use case I have would be for some string operations. A common pattern we have for resource names is "(current-git-branch)-(resource-specific-suffix)". These can get quite long, and sometimes long branch names will mean it goes over the limit of a given resource name (and of course each resource name has different limits).

What I'd love to be able to define is a function that would combine these, and truncate the git branch name as needed (possibly with a sha of the git branch name as a suffix so that 2 git branches differing in only the last character don't clash). This would be wonderful as a pure string manipulation function that I could define once and use everywhere, but is a nightmare if I have to do it in every single resource.

(Even better would be if somehow the function could know the resource name limits for the resource in which it was being invoked, but I imagine that's a step too far)

[bmaltais]

Indeed. I would like to create a function that would essentially do this and be able to call it to generate a compliant name for all the resources that need it by simply passign the right arguments:

locals {
  azurecaf_naming_convention-Project-law-replace = replace("${var.env}CLD-${var.group}-${var.project}", "_", "-")
  azurecaf_naming_convention-Project-law-regex   = regex("[0-9A-Za-z-]+", local.azurecaf_naming_convention-Project-law-replace)
  azurecaf_naming_convention-Project-law-54      = substr(local.azurecaf_naming_convention-Project-law-regex, 0, 54)
  azurecaf_naming_convention-Project-law-59      = substr("${local.azurecaf_naming_convention-Project-law-54}-${local.unique_Logs}", 0, 59)
  azurecaf_naming_convention-Project-law-result  = "${local.azurecaf_naming_convention-Project-law-59}-law"
}

resource "azurerm_log_analytics_workspace" "Project-law" {
  # name                = azurecaf_naming_convention.Project-law.result
  name                = local.azurecaf_naming_convention-Project-law-result
  location            = azurerm_resource_group.Logs-rg.location
  resource_group_name = azurerm_resource_group.Logs-rg.name
  sku                 = "PerGB2018"
  tags                = var.tags
}

That way I could create different functions for different Azure resources with different weird name rules instead of having to declare a local for every resource and repeat the code all over the place.

[Ardalan-Saberi]

How about extending hcl by a function that can pass a query into external data resources inline... something like the external_lookup function below:

data "external" "my_function" {
    program = ["python", "${path.module}/my_function.py"]
}

resource "instance" "foo" {
    name = external_lookup("my_function", {query="foo"}).result.instance_name
    ...
}

[ocervell]

I think the main use case here for enterprises is to simplify naming conventions and extracting things from them, to simplify end-user configs like shown in comment above. Custom functions would help greatly with that !