Terraform: json: cannot unmarshal object into Go value of type string // Show Inputs and Outputs of External Data Command

[Miserlou]

Terraform Version

Terraform v0.9.3

Affected Resource(s)

• data.external

Terraform Configuration Files

data "external" "template" {
  program =  ["zappa", "template",  "-r", "'${aws_iam_role.iam_for_my_lambda.arn}'", "-l", "'${aws_lambda_function.my_lambda.arn}'", "--json"]
  depends_on = ["aws_lambda_function.my_lambda"]
}

Debug Output

* data.external.template: data.external.template: command "zappa" produced invalid JSON: json: cannot unmarshal object into Go value of type string

Expected Behavior

The error shows the command which was executed and the output of that command.

Actual Behavior

I receive no useful information about the arguments of the executed command, or the output which failed to be parsed to JSON.

I don't now if this is a "Terraform-can't-parse-valid-JSON" bug, or a "My-program-pooped-the-bed" bug, because the output has no useful information in it, which makes it a "Terraform should be more helpful here in the UX department" bug.

Steps to Reproduce

1. Create a complex command.
2. terraform apply

Important Factoids

Running the command manually in my terminal produces valid JSON.

[apparentlymart]

Agreed that this data source should not be just returning the raw error message from the Go JSON library here. Thanks for pointing this out!

In this case, based on that error message I suspect that the JSON being generated has a nested object. Due to limitations of Terraform's type system we can currently allow only string values in the returned JSON object. This will hopefully get better in future after we do some work on Terraform's internals to support mixed types.

[Miserlou]

Wow, that's a pretty heavy requirement. Why not just store the whole command string output separately as "result" or something like that? It's UNIX, the whole point is that strings can be passed around standard inputs and outputs, not that they speak a specialized _subset_ of a web serialization format..

Also, please don't just cover up this error with another error string. Actually show the output of the command that caused the problem.

[apparentlymart]

In the case of this resource, "UNIX-ness" was not the goal but rather being able to expose multiple values in a way that would work in a similar way to a first-class Terraform resource.

Possibly we will eventually have a separate data source that just executes a command and captures its output. The result of such a thing would likely be harder to use in Terraform in the general case since it lacks good primitives for string parsing, but I can see that it would be useful in simple cases where the entire result is used verbatim as a single value.

However, let's just consider this issue to represent producing a more actionable error for this data source, and we can consider the other requirement separately.

[BrianGallew]

1.5 years and we still can neither unmarshal non-trivial JSON nor can we update the error message to include the invalid JSON. What needs to be done to have some traction on this?

[ghost]

It's very difficult to use terraform for more complicated infrastructure generation with dependencies, that require the import of (sometimes) nested JSON files. This really needs to be implemented.

[tomdavidson]

@BrianGallew

I invested to much in my external script and couldn't give up. Here's a workaround:

In my bash script encode my JSON array and add it to some new JSON as a single string:

 printf '{"base64_encoded":"%s"}\n' $(echo "${json_array}" | base64 -w 0)

Then in my terraform plan I can base64decode - and also json encode:

data "external" "find_files" {
  program = ["bash", "${path.module}/find.sh"]

  query = {
    filter = "*.yaml"
  }
}

output "files" {
  value = jsondecode(base64decode(data.external.find_files.result["base64_encoded"]))
}

Of course, I'm still burned about all my zero dependency (no jq) JSON parsing and building ... when I am just going to return a delimited list of files and not a JSON array.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.