Terraform: `jsonencode` should not flatten lists of lists

[austinbyers]

The new locals feature combined with the improved jsonencode is very powerful, because it lets you build arbitrary Terraform objects _with interpolation_ and then convert them to JSON. :tada:

However, in order for this to work, jsonencode needs to honor nested list structures and not automatically flatten them.

Terraform Version: 0.10.3

Example: aws_cloudwatch_dashboard

terraform {
  required_version = "~> 0.10.3"
}

variable "aws_region" {
  default = "us-east-1"
}

provider "aws" {
  version = "~> 0.1.4"
  region  = "${var.aws_region}"
}

# Much more readable than using a raw JSON string.
# Also, comments and interpolation FTW!
locals {
  dashboard_body = {
    widgets = [
      {
        type = "metric"
        properties = {
          title  = "Invocations/Errors"
          region = "${var.aws_region}"
          stat   = "Sum"
          period = 300
          metrics = [
            ["AWS/Lambda", "Invocations"],
            ["AWS/Lambda", "Errors"]
          ]
        }
      }
    ]
  }
}

resource "aws_cloudwatch_dashboard" "lambda" {
  dashboard_name = "Dashboard"
  dashboard_body = "${jsonencode(local.dashboard_body)}"
}

Note the metrics list of lists. A terraform plan shows the following auto-generated JSON:

{'widgets': [{'properties': {'metrics': ['AWS/Lambda',
                                         'Invocations',
                                         'AWS/Lambda',
                                         'Errors'],
                             'period': '300',
                             'region': 'us-east-1',
                             'stat': 'Sum',
                             'title': 'Invocations/Errors'},
              'type': 'metric'}]}

See how the metrics have been flattened into a single list? This will fail if you try to apply:

* aws_cloudwatch_dashboard.lambda: 1 error(s) occurred:

* aws_cloudwatch_dashboard.lambda: Putting dashboard failed: InvalidParameterInput: The dashboard body is invalid:
[
  {
    "message": "Field \"metrics\" has to be an array of array of strings..."
  }
]

The problem persists even if you only have one nested list element.

[apparentlymart]

Hi @austinbyers! Thanks for reporting this and sorry for the weirdness.

My suspicion is that this is actually an issue with the configuration parser rather than with the jsonencode function itself. The HCL parser takes some liberties in its parsing to support things like repeated blocks with the same name, and this leads to some weirdness which we've got plans to address as part of a suite of forthcoming language changes.

In the mean time, it's usually possible to work around these parsing oddities by generating structures dynamically in the interpolation language, using the list and map functions, since then the parser doesn't deal with these directly and these unwanted simplifications don't get applied.

In this case I think you could get away with applying this rewriting only to the metrics part:

metrics = "${list(list("AWS/Lambda", "Invocations"), list("AWS/Lambda", "Errors"))}"

The config language revamp is, unfortunately, quite a big project since we need to manage various small compatibility breaks that result from removing this sort of ambiguity, so we're currently planning the best way to get it done. I hope the above workaround helps in the mean time while we work through this.

[austinbyers]

The explicit list construction works really well, thanks so much! That's a much better workaround than the string hacking I was trying to do :)

[holmesb]

@austinbyers, is your code working when substituting in @apparentlymart's list of lists? We're getting an error because jsonencode is converting the period integer into a string:

* aws_cloudwatch_dashboard.lambda: Putting dashboard failed: InvalidParameterInput: The dashboard body is invalid, there are 2 validation errors:
[
  {
    "message": "Should be integer",
* module.cloudwatch.aws_cloudwatch_dashboard.lambda: 1 error(s) occurred:
    "dataPath": "/widgets/0/properties/period"
  },
   {
    "message": "Should match exactly one schema in oneOf",
   "dataPath": "/widgets/0/properties/period"
}

Output shows the json-encoded dashboard_body has 300 as a string:
dashboard_body: "" => "{\"widgets\":[{\"properties\":{\"metrics\":[[\"AWS/Lambda\",\"Invocations\"],[\"AWS/Lambda\",\"Errors\"]],\"period\":\"300\",\"region\":\"eu-west-1\",\"stat\":\"Sum\",\"title\":\"Invocations/Errors\"},\"type\":\"metric\"}]}"

Is there any way to make jsonencode respect integers and not convert them? Or should this be opened as a new issue?

[austinbyers]

@holmesb Terraform does not (yet) have a distinction between numbers and strings. So a simple string replace that strips the quotes will work:

dashboard_body = "${replace(jsonencode(local.dashboard_body), "/\"(\\d+)\"/", "$1")}"

[apparentlymart]

Hi all! Sorry for the long silence here.

I'm pleased to report that I tried this in Terraform v0.12.0-alpha1 today and verified that it worked as I expected:

variable "aws_region" {
  default = "us-west-1"
}

locals {
  dashboard_body = {
    widgets = [
      {
        type = "metric"
        properties = {
          title  = "Invocations/Errors"
          region = "${var.aws_region}"
          stat   = "Sum"
          period = 300
          metrics = [
            ["AWS/Lambda", "Invocations"],
            ["AWS/Lambda", "Errors"]
          ]
        }
      }
    ]
  }
}

output "example" {
  value = jsonencode(local.dashboard_body)
}

I beautified the output JSON so it was easier to review:

{
    "widgets": [{
        "properties": {
            "metrics": [
                ["AWS/Lambda", "Invocations"],
                ["AWS/Lambda", "Errors"]
            ],
            "period": 300,
            "region": "us-west-1",
            "stat": "Sum",
            "title": "Invocations/Errors"
        },
        "type": "metric"
    }]
}

Notice also that jsonencode will now correctly serialize numbers as JSON numbers, rather than turning them into strings as before.

Since these fixes are already in master I'm going to close this out. It'll be included in the final v0.12.0 release too. Thanks for reporting this, and for your patience while we laid the groundwork to fix it.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.