Terraform: Template variables and bash substitution/array

[smotti]

Currently how template variables are defined in templates interfere with bash's substitution feature and also array access.

For example, we like to append the instance id to the hostname of a new instance during bootstrap, for this we use the user_data attribute provided by the aws_instance resource.
The user_data will use as value a rendered template_file, i.e.:

resource "template_file" "user_data" {
  filename = "user_data.txt"
}

resource "aws_instance" "worker" {
  ami = "ami-ffeeddcc"
  instance_type = "t1.micro"
  user_data = "${template_file.user_data.rendered}"
}

When using AWS EC2 one can get the instance-id via a meta-data http api:

curl -s http://169.254.169.254/latest/meta-data/instance-id
i-ffeeddcc

But we don't need the 'i-' thus we use the following bash substitution:

#!/bin/bash -xe
ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
echo worker-${ID%i-} > /etc/hostname
/etc/init.d/hostname start

Unfortunately this results in a 'parse error: syntax error' when terraform wants to render the template.

We also tried to escape the '{' and '}' but this will result in the hostname:

worker-$\{ID%i-\}

[radeksimko]

I agree that there should be a way to express ${} without any interpolation, it may also be necessary for IAM policy variables:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": ["s3:ListBucket"],
      "Effect": "Allow",
      "Resource": ["arn:aws:s3:::mybucket"],
      "Condition": {"StringLike": {"s3:prefix": ["${aws:username}/*"]}}
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Effect": "Allow",
      "Resource": ["arn:aws:s3:::mybucket/${aws:username}/*"]
    }
  ]
}

probably do not want ${aws:username} to be interpolated here.

[mitchellh]

You can escape with double dollar signs: $${foo}. See the user data example here in the Vault repo for an example: https://github.com/hashicorp/vault/blob/master/terraform/aws/scripts/install.sh.tpl#L43

I'm going to update the docs now to show this.

[radeksimko]

Docs updated: 0c9e95a32fc22b9f86246aad8e23a27b961d34d9

[jacoelho]

Even escaping like in docs:

resource "aws_iam_group_policy" "human_users_policy" {
  name  = "test"
  group = "${aws_iam_group.human_users.id}"

  policy = <<EOF
{
    "Statement": [
        {
            "Resource": "arn:aws:iam::742736341824:user/$${aws:username}",
            "Action": [
                "iam:*LoginProfile",
                "iam:*AccessKey*",
                "iam:*SigningCertificate*"
            ],
            "Sid": "AllowUsersAllActionsForCredentials",
            "Effect": "Allow"
        },
    ],
    "Version": "2012-10-17"
}
EOF
}

Aws policy console:

 "Statement": [
        {
            "Resource": "arn:aws:iam::742736341824:user/$${aws:username}",
            "Action": [
                "iam:*LoginProfile",
                "iam:*AccessKey*",
                "iam:*SigningCertificate*"
            ],
            "Sid": "AllowUsersAllActionsForCredentials",
            "Effect": "Allow"
        }

[opokhvalit]

there is a mistypo in docs
You should escaping NOT just double $ sign, but construction like: $${foo} (backslash,dollar,dollar,variable_name)

it will be cool to fix documentation section on site

[nicerobot]

No combination of escaping and doubling seems to work in 0.7

I get

failed to render : 29:12: unknown variable accessed: IP

for:

• SUBNET=$${IP%.*}
• SUBNET=\$${IP%.*}

and parse failure for:

• SUBNET=${IP%.*}
• SUBNET=\${IP%.*}

[adamdecaf]

@nicerobot I'm running into this issue too on terraform 0.7.3. Instead though I'm seeing:

* data.template_file.mutate_users: failed to render : parse error: syntax error

On a data resource pointing to a template.

data "template_file" "mutate_users" {
  template = "${file("${path.module}/usr-local-sbin/users.sh")}"
}

The good news for us is we don't need to interpolate anything and we're using it for a trigger.

[matschaffer]

Just wanted to confirm that $${bash_var} works for inserting ${bash_var} into the rendered template as of v0.7.4.

\$${VAULT_FLAGS} in the comment above is also inside a heredoc, so the backslash is there to prevent interpolation during script execution and produce ${VAULT_FLAGS} within /etc/init/vault.conf

[inthecloud247]

I'm not sure if this is the place to mention this, but the choice of brace quotes like ${var_name} for terraform templating is a bad technical decision, and I think it would be a meaningful improvement to the project if future terraform versions allow a different templating format.

Well-written Bash scripts have frequent use of brace quotes like ${var_name} for referencing variables. When starting a terraform project, I've usually found that I'm at some point integrating existing Bash scripts with terraform. At the moment, doing this requires changing the scripts to reference Bash variables with $var_name, which often isn't secure, and isn't technically possible in all circumstances... or moving to $${var_name}, which makes it very difficult to test/debug scripts during development since that isn't a valid Bash syntax.

Both situations require making a bad technical or security tradeoff in the name of compatibility with terraform, and it isn't clear at all why the terraform parser can't use a different syntax ( e.g. double braces {{ var_name }} ) common to other golang-based projects that doesn't conflict with Bash.

I've found that I agree with the vast majority of the project's technical decisions, and find them to meet a consistently high standard.. which makes this particular issue stick out even more. As a devops-centric tool, Terraform shouldn't have this technical incompatibility issue with Bash.

[Binternet]

+1 with @inthecloud247

terraform parser should use a different syntax for placing variables into a template
file such as {{ var_name }} to avoid conflicts with bash scripts.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.