Terraform: Chef provisioner attributes_json cannot parse rendered template_file

[b-dean]

Terraform Version

0.8.1

Affected Resource(s)

• null_resouce
• template_file (data)

Terraform Configuration Files

data "template_file" "json" {
  template = <<EOF
    {
      "abc": "def"
    }
EOF
}

resource "null_resource" "chef" {
  triggers { json = "${data.template_file.json.rendered}" }

  provisioner "chef" {
    node_name = "chef"
    server_url = "http://chef.chef/chef"
    user_name = "chef"
    user_key = "chef"
    run_list = []
    attributes_json = "${data.template_file.json.rendered}"
  }
}

Debug Output

https://gist.github.com/b-dean/5e1b28f9f03a48823c2aa8bb515d5f36

Expected Behavior

plan has the following output (this output is from running terraform plan with 0.7.13):

+ null_resource.chef
    triggers.%:    "1"
    triggers.json: "    {\n      \"abc\": \"def\"\n    }\n"

Actual Behavior

plan fails:

There are warnings and/or errors related to your configuration. Please
fix these before continuing.

Errors:

  * null_resource.chef: Error parsing attributes_json: invalid character 'D' after top-level value

Steps to Reproduce

1. terraform plan

[mengesb]

I'm bit by this too. Took a look at the provider, and I didn't see anything obvious so I'm at a loss for what's different with builtin/provisioners/chef... guessing it's not there.

[mengesb]

Any chance someone could point me in the right direction? I could take a stab at a PR if I know where to look... I've not been able to figure it out yet

[ChiefAlexander]

I have spent sometime poking around at this, it affects all of the 0.8.x versions. It also appears to only affect Providers.
For example using the same info as above you can get a clean run with:

variable "json" {
  default = <<-EOF
  {
    "abc": "test"
  }
  EOF
}

resource "null_resource" "chef" {
  provisioner "chef" {
    node_name = "chef"
    server_url = "http://chef.chef/chef"
    user_name = "chef"
    user_key = "chef"
    run_list = []
    attributes_json = "${var.json}"
  }
}

or even:

variable "test" {
  default = "abc"
}

resource "null_resource" "chef" {
  provisioner "chef" {
    node_name = "chef"
    server_url = "http://chef.chef/chef"
    user_name = "chef"
    user_key = "chef"
    run_list = []
    attributes_json = <<-EOF
    {
      "abc": "${var.test}"
    }
    EOF
  }
}

However including anything else such as using a module to render the template results in the same error.
Module:

data "template_file" "json" {
  template = <<EOF
    {
      "abc": "def"
    }
EOF
}

output "test" {
  value = "${data.template_file.json.rendered}"
}

Main:

module "templete" {
  source = "./template"
}

resource "null_resource" "chef" {
  provisioner "chef" {
    node_name = "chef"
    server_url = "http://chef.chef/chef"
    user_name = "chef"
    user_key = "chef"
    run_list = []
    attributes_json = "${module.templete.test}"
  }
}

Or using self to get the error

resource "null_resource" "chef" {
  count = 1

  provisioner "chef" {
    node_name = "chef"
    server_url = "http://chef.chef/chef"
    user_name = "chef"
    user_key = "chef"
    run_list = []
    attributes_json = <<EOF
    {
      "abc": "${self.count}"
    }
    EOF
  }
}

I have not been able to gather anything helpful from debugging the code. We should be able to look at the commits made to v0.8.0-beta1 and find what has changed between versions.

[ChiefAlexander]

I was able to narrow down the commit to b979d8927e3cf2a91c6f9fb0840531a4f7341553
Line: https://github.com/hashicorp/terraform/blob/master/config/interpolate_walk.go#L176
The value of attributes_json is being marked as a UnknownVariableValue and set to the constant 74D93920-ED26-11E3-AC10-0800200C9A66
Thus our error appears as Error parsing attributes_json: invalid character 'D' after top-level value

I am still lost on how it is being set as a UnknownVariableValue. Hoping that one of the contributors can chime in.

[mengesb]

I'm a bit currious how it is that you traced it down to interpolate_walk.go... could you elaborate @ChiefAlexander ?

[ChiefAlexander]

After discovering that this only affected 0.8.x I built the tag 0.8.0-beta1 and found that i was still getting the error. Then I just went back in time building on the commits in 0.8.0-beta1 until I narrowed down which commit was causing our issue. Not the most elegant debugging but the error message wasn't much help. Then I did some prints to see what was going through that line of code and found that we are passing in the constant I listed above.

[sarkis]

Looking at the comment: https://github.com/hashicorp/terraform/blob/master/config/interpolate_walk.go#L156-L157

"if the result contains any "UnknownVariableValue" which is set if it is computed"

wouldn't ${data.template_file.json.rendered} be marked as computed during the plan? Just trying to figure out why it's being set as an UnknownVariableValue..

[luckymike]

This bit me too, but only on a few of our modules so I looked to see what was different about them. It appears that this is triggered on variables nested more than one level deep, e.g. this is fine:

{
  "env": "${var.env}"
}

but this fails:

{
  "env": "${var.env}"
  "namespace": {
    "foo": "${var.bar}"
  }
}

Interestingly, double escaping ($${var...}) the deeply nested variables seems to resolve the error, but I haven't actually attempted an apply that way, so I don't know if it actually interpolates correctly.

[luckymike]

Update on my previous comment. I attempted an apply with the double escaped vars, and I ended up with the single escaped vars in my first_run.json. So, possibly interesting data point, but not a workaround.

[gburson]

I'm affected by this as well. Just learning Terraform, just hooked it up to my chef provisioner but it fails at this point. Interestingly luckymike says that avoiding nesting is a workaround, but I still get a failure.

...
attributes_json = <<-EOF
{
"database_endpoint": "${var.database_endpoint}",
"database_port": "${var.database_port}"
}
EOF

• aws_instance.tag: Error parsing attributes_json: invalid character 'D' after top-level value

[gburson]

BTW I'm on a slightly later version (just did a default install of Terraform)

Terraform v0.8.2

[thebrianlopez]

+1

[gburson]

Really surprised more people aren't being affected by this issue. What it means (at least for me) is that I can't create an RDS instance via Terraform, and pass the details to my application nodes for Chef provisioning. I guess my last resort is to downgrade to < v0.8.0 in the short term.

[ChiefAlexander]

To add a little more context of the issue. Since the commit b979d8927e3cf2a91c6f9fb0840531a4f7341553 removed the removeCurrent function, our values that need to be computed are set to the constant UnknownVariableValue. This UnknownVariableValue gets passed back to the Chef Provisioner which then uses its decodeConfig function to parse what should be a json string https://github.com/hashicorp/terraform/blob/master/builtin/provisioners/chef/resource_provisioner.go#L356. However we are no longer passing back a json string anymore since the removeCurrent function was removed. I am unsure of what needs to change within the Chef Provisioner to return functionality however.

[gburson]

Thanks Chief. I've dropped an email to the person who did the commit on the off chance he can take a look. It's a little outside my area of expertise!

[pgporada]

I encountered a variant of this today in the bitbucket_repository resource.

invalid character 'R' looking for beginning of value

The curious thing was that if I waited an extra ~10 minutes between applies, I would get different amounts of fail. I was attempting to run about 600 changes against bitbucket so it seemed like an API rate limit to me.

[gburson]

I got a response from the developer. He says they are a bit inundated after the Christmas break but he's kindly said he'll try to get a fix included in a patch release they have next week.

[triasity]

I tested this today from v0.8.0 up to v0.8.4 and still getting the error. Have rolled back to v0.7.13 until the issue is fixed.

aws_instance.tag: Error parsing attributes_json: invalid character 'D' after top-level value

[lajoie]

I ran in to this as well but was able to work around it.

In my case, I was trying to inject the FQDN of a Route53 A record. So I had this:
${aws_route53_record.db_a_record.fqdn}

However, within that resource, that value is really just computed from some other fields. So, I replaced the Route53 FQDN reference with the same string interpolation that I fed into the Route53 resource. Obviously that won't work for everyone, but something to look at if you can do it.

[gburson]

Not so good for me but thanks in any case. I'm trying to bring up a complete infrastructure including RDS. My app (in an EC2 instance) needs the name of the endpoint for the RDS cluster, which is only created during the Terraform run and provisioned with Chef.

I'm thinking I can work around this by giving my application a static DNS name (in Route53) then update the DNS entry with a CNAME record that points to the actual RDS endpoint. Painful but should work.

BTW I dropped the developer another email asking if he'd managed to take a look yet. If I hear anything I'll post back.

[brainrape]

I guess this kind of friction is to be expected when implementing a lazy
functional language (where stuff is computed only when needed, like tf
resource dependencies) in a strict imperative one.

On Fri, Jan 13, 2017, 08:09 gburson [e-mail] wrote:

Not so good for me but thanks in any case. I'm trying to bring up a
complete infrastructure including RDS. My app (in an EC2 instance) needs
the name of the endpoint for the RDS cluster, which is only created during
the Terraform run and provisioned with Chef.

I'm thinking I can work around this by giving my application a static DNS
name (in Route53) then update the DNS entry with a CNAME record that points
to the actual RDS endpoint. Painful but should work.

BTW I dropped the developer another email asking if he'd managed to take a
look yet. If I hear anything I'll post back.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/hashicorp/terraform/issues/10788#issuecomment-272376394,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AASdXSWpHy7FFtXV5mmNYv7A29g_OzyBks5rRyNFgaJpZM4LPUAj
.

[felka]

The most problematic case for people who is managing their state remotely.... There is no way back form 8 to 7 version without restoring from version 7 backup.

[sjwl]

Error parsing attributes_json: invalid character 'D' after top-level value is an error from json.Unmarshal() when the value of attributes_json is 74D93920-ED26-11E3-AC10-0800200C9A66 (also known as UnknownVariableValue). Which happens when attributes_json has any interpolation of computed values.

I attempted a fix that worked for me, and submitted a pull request #11348. One downside of this fix is that any JSON syntax errors of attributes_json with computed interpolation won't be caught during terraform plan.

[mengesb]

Where are we on evaluating #11348 ? Currently there's no way to bootstrap chef using terraform's templating language to provide an attributes set.

[luckymike]

This has been open 6 weeks. We're Atlas users (i.e. we pay for each instance managed via Terraform) and this bug has two or three of our environments totally wedged. Our other environments are being held back on 0.7.x because of it. We've opened support tickets and simply been told someone will look into it. I realize that TF is a big OSS project, and maintaining it is no small effort, but this regression upends a fundamental function of the tool, and of a service like Atlas built around it.

[mitchellh]

I'm taking a look at this now. Sorry for the delay!

[mitchellh]

Alright sorry I missed this. I caught up on all the comments here and the PR opened (thanks!).

I've determined the "root cause" of this. Others above have rightly determined it is to do with the value being UnknownVariableValue, but I didn't know at first why this suddenly became an issue. Now its clear. I'll explain below for those curious.

For those not interested in knowing the "why", I'll have a fix PRed today, we'll merge it first thing this week and we'll get it into a 0-8-stable release so you won't have to wait too much longer. I'm very sorry.

The background: provisioners directly access c.Config. Prior to 0.8, c.Config would delete any computed keys. This was suboptimal and caused issues because it would delete _too much_: if one key in a map was computed for example, the entire map would get removed. Our fix was to keep the values in but set the more granular value to "UnknownVariableVar". The actual fix was a lot more involved then that but that's the high level.

We didn't think this would cause problems because we didn't realize anyone accessed c.Config directly. Providers for example go through the helper/schema framework which tests IsComputed prior to access, so its not an issue. Provisioners in 0.9 also use a framework, so its also not an issue. But in the interim, we have provisioners that access the configs directly, therefore making them think a value exists when its really just a UUID...

The fix is that we must guard any Config access with IsComputed checks in provisioners for now.

But in 0.9 the provisioner writing framework will do this automatically.

[mitchellh]

PR opened.

[robomon1]

Did this make it into an 0.8.x release? I didn't see it noted anywhere on the changelog page.

[b-dean]

@robomon1, it's in 0.8.6. The changelog references the PR #11502

[gsaslis]

@robomon1 yep - I have tested this and it works for me in 0.8.6

[mengesb]

Not sure this is an artifact of the changes here, but I wonder if #13696 (DIFF resulting in a NULL) happens because of this change?

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.