Terraform: jsonencode doesn't preserve numbers when encoding maps

[vincer]

Terraform Version

Terraform v0.11.1

Expected Behavior

> jsonencode(map("foo",3))
{"foo":3}

Actual Behavior

When using jsonencode on a map with any integers/floats, those will be cast to strings in the resulting JSON.

> jsonencode(map("foo",3))
{"foo":"3"}

Steps to Reproduce

Terraform console makes it easy to repro.

Additional Context

Also wrong/interesting behavior:

> jsonencode(3)
jsonencode: unknown type for JSON encoding: int in:
> list(3)
list: unexpected type int for argument 0 in list in:

Using floats instead of ints exhibits the same behavior.

[vincer]

One workaround is to post-process the jsonencode with a regex, to remove all quotes around any integers/floats:

> replace(jsonencode(map("foo",3)), "/\"([0-9]+\\.?[0-9]*)\"/", "$1")
{"foo":3}
> replace(jsonencode(map("foo",3.1)), "/\"([0-9]+\\.?[0-9]*)\"/", "$1")
{"foo":3.1}

[apparentlymart]

Hi @vincer! Sorry for this unexpected behavior.

This is a known limitation of the current interpolation language, which converts all primitive values to strings to interact with Terraform core.

We are currently in the process of integrating a new version of the configuration language that addresses this limitation, which will then allow us to use a new jsonencode implementation that is able to faithfully preserve all of the input value types.

We're planning to release an opt-in experimental version of this new language implementation soon to gather feedback and identify any issues before we release it as the main implementation.

[Lasering]

Neither does it encode booleans and floats:

> jsonencode(true)
jsonencode: unknown type for JSON encoding: bool in:

${jsonencode(true)}
> jsonencode(5.5)
jsonencode: unknown type for JSON encoding: float64 in:

${jsonencode(5.5)}

[Jaff]

This might be helpful for a problem I am experiencing currently with doing a string-replace in a cloud-init template file. Recent changes in our cloud-init to enable upgrade of etcd2 to etcd3 in CoreOS has resulted in a section of the template-file that contains "variable-like" strings which are NOT supposed to be replaced; Trying to code a work-around has yielded a solution that I expected to work, but consistently fails. I will work on sample code to exhibit the failure which I can upload.

[ploh]

@apparentlymart

This is a known limitation of the current interpolation language, which converts all primitive values to strings to interact with Terraform core.

But if I then assign the encoded json, I get an error message:

resource "aws_batch_job_definition" "this" {
  name = "my_batch_job"
  type = "container"
  container_properties = "${jsonencode(local.container_properties)}"
}

where local.container_properties is defined as Terraform map containing

    "memory" = 7600,

This leads to the following error during terraform plan:

* module.batch.aws_batch_job_definition.this:
AWS Batch Job container_properties is invalid:
Error decoding JSON: json: cannot unmarshal string into
Go struct field ContainerProperties.Memory of type int64

So it seems the ContainerProperties.Memory does not work with a _string_ but with an _int64_. Since you said that it is all strings in Terraform core, this seems like a bug in there, right?

[dgoetsch]

@ploh you can get around it by using a template file, though it is quite a bit more verbose. Container properties is a string representation of the json, but jsonencode is incorrectly converting the number field to a string field. If you use a properly formatted Json string for container properties, it will work (I presume, I have a similar issue with ecs services). The bug here is in how jsonencode writes values.

[ploh]

@dgoetsch I am getting around it by using a heredoc string, i.e.

container_properties = <<CONTAINER_PROPERTIES
...
CONTAINER_PROPERTIES

I just think it is funny that jsonencode will make a string out of my int and then the AWS provider apparently cannot use that (because of ContainerProperties.Memory being int64) - although terraform core in the end expects a string anyway.

I guess I will just have to wait for the _new version of the configuration language_ to do this properly.

[cmacrae]

Just a quick tip for what I'm using at the moment to get around this for AWS ECS container definitions.
I've put in the following hack so I can dictate what I want to be parsed as a string/an int:

resource "aws_ecs_task_definition" "ecs_service" {
  family        = "${var.service_name}-${var.environment}"
  task_role_arn = "${aws_iam_role.ecs_service.arn}"
  // FIXME: The use of 'replace' here is because of a bug in 'jsonencode':
  //        https://github.com/hashicorp/terraform/issues/17033
  container_definitions = "${replace(replace(jsonencode(var.container_definitions), "/\"([0-9]+\\.?[0-9]*)\"/", "$1"), "string:", "")}"
}

The inner replace (as seen further up in the comments) is to strip any quotations around a number value. The outer replace allows you to forego the inner replace by putting string: in front of your value.

This is useful in the case that your container definition(s) might include some environment parameters which are numbers, but you still want them left as strings.
Here's an example:

  container_definitions = [{
    name = "${local.service}-${local.env}",
    image = "${var.circle_docker_image}"
    cpu = 10
    memoryReservation = 200
    portMappings = [{
      containerPort = 3000,
      hostPort = 0}]
    logConfiguration = {
      logDriver = "awslogs"
      options = {
    awslogs-group = "${local.service}-${local.env}"
    awslogs-region = "${local.region}"
      }
    }
    environment = [
      {
    name = "MY_INT_STRING"
    value = "string:14"
      }
    ]
  ]}

When this is evaluated, any integer values will be honoured as such, but any string:<INT> values will have string: stripped, but be left as a string: "<INT>"

Hope this helps others. It's not the nicest approach, but it works for now.

[tanmng]

@cmacrae and @vincer

The replace trick was really nice and I did ended up using that a lot. However, I think we should expand the regex a little bit to fit any potential negative number. In the case of ECS task definition, ulimits might be negative too. So I think the example by @cmacrae should be

resource "aws_ecs_task_definition" "ecs_service" {
  family        = "${var.service_name}-${var.environment}"
  task_role_arn = "${aws_iam_role.ecs_service.arn}"
  // FIXME: The use of 'replace' here is because of a bug in 'jsonencode':
  //        https://github.com/hashicorp/terraform/issues/17033
  container_definitions = "${replace(replace(jsonencode(var.container_definitions), "/\"(-?[0-9]+\\.?[0-9]*)\"/", "$1"), "string:", "")}"
}

In fact, I'm using that exact RegEx for my code

[thefotios-enigma]

This also appears to be a problem with boolean values. Terraform appears to be coercing them into 1 or 0. Requiring different types in the environment block versus everywhere else complicates things even further. Using the string: marker method from https://github.com/hashicorp/terraform/issues/17033#issuecomment-399908596 and a slightly modified regex here are some examples:

/**
 * This shows the input types and output types (the comment at the end of the line)
 *
 * - In the `environment` block, you need to ensure your value is cast to a Str
 * - Everywhere else, you need to ensure your value is cast to it's correct type
 */

locals {
  container_definition = {
    boolean = {
      string = "true"        # Bool
      raw    = true          # Int (This is probably never what you want)
      marker = "string:true" # String
    }

    integers = {
      raw    = 1          # Int
      string = "1"        # Int
      marker = "string:1" # String
    }

    float = {
      raw    = 0.25          # String
      string = "0.25"        # String
      marker = "string:0.25" # String
    }
  }
}

output "output" {
  value = "${replace(
    replace(
      "${jsonencode(local.container_definition)}",
      "/\"(true|false|[[:digit:]]+)\"/", "$1"
    ), "string:", ""
  )}"
}

/**
{
  "boolean": {
    "marker": "true",
    "raw": 1,
    "string": true
  },
  "float": {
    "marker": "0.25",
    "raw": "0.25",
    "string": "0.25"
  },
  "integers": {
    "marker": "1",
    "raw": 1,
    "string": 1
  }
}
*/

[hflamboauto1]

👎 please make jsonencode work as expected or remove it from terraform. This is not acceptable.

[zoe-alexander]

Normally I would find @hflamboauto1's comment a little rude, but in this case I think it's appropriate. The current state of jsonencode() is extremely confusing, and effectively worse than nothing.

Looking forward to seeing how the introduction of real types in the next release will impact jsonencode(). Would be especially great to get some clarification on that from Hashicorp, since between JSON and YAML, we have something close to the lingua franca of passing around cloud service configs (and since YAML is a superset of JSON, jsonencode() can essentially meet both needs if it gets fixed).

[apparentlymart]

This will be fixed in the 0.12 release indeed. You can see the draft updated docs to see how the new implementation (which is already written) will behave.

(As usual, links to the markdown content from the website on GitHub do not show with working links due to the difference in structure on the website vs. in the repository; the "Terraform language values" link there will ultimately lead to the documentation on the types supported by the language. Both of these pages may be further revised before release, but likely to just be copyediting and minor tweaks of details at this point.

[apparentlymart]

Hi all!

I'm pleased to report that the fix here is now merged into master ready for inclusion in the v0.12.0 release, and I have verified it in the v0.12.0-alpha1 prerelease build:

$ terraform console
> jsonencode(map("foo",3))
{"foo":3}
> jsonencode({"foo" = 3})
{"foo":3}
> jsonencode(3)
3

This new behavior and the associated docs will both go out with the v0.12.0 release.

[PaulHammondZoyo]

Hi All,

Thanks for this!

The 'replace' method also works to json-sanitise rendered container definition output when defining elements such as cpu/memory via input variables (which are string based).

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.