Packer: Variables can't tolerate arrays

[tb3088]

There are a multitude of Provision and Builder objects that are constructed from Arrays. But the Variables object is coded to only support strings:

Error compiling user variables: json: cannot unmarshal array into Go value of type string

Please fix so that Arrays are handled natively, or if that would be entirely too complicated, maybe as a back-door hack, leverage https://golang.org/src/encoding/json/decode.go which can suss out arrays from within strings. So that in variable stanzas one can use:

{ "variables": [
"key": "[ "value", "value2", ... ]",
...
]
}

[mitchellh]

If you comma separate the string, it'll turn into an array.

[tb3088]

And where did you DOCUMENT that little hack? Which doesn't work BTW. And why deviate from standard JSON? The parser manages to handle the rest of the document just fine.

Quite simply the "Variable" data structure is deficient! If you're going to cripple it by only supporting strings, then when it's time to evaluate the variable and substitute it into the in-memory version of the Packer template, it needs to be properly decoded.

{
"variables": {
"recipe": "recipe[my-cereal-box], recipe[foo::jenkins]",
"databag_secret_path": "",
"build_number": "{{env BUILD_NUMBER}}",
"git_commit": "{{env GIT_COMMIT}}"
},
...
"provisioners": [
{
"type": "chef-solo",
"cookbook_paths": ["cookbooks"],
"encrypted_data_bag_secret_path": "{{user databag_secret_path}}",
"environments_path": "../chef/environments",
"chef_environment": "{{user environment}}",
"run_list": [ "{{user recipe}}" ],
"install_command" : "curl -LO https://www.opscode.com/chef/install.sh && {{ if .Sudo}}sudo{{end}} bash ./install.sh -v 11.12.8 && rm install.sh"
}
]

amazon-ebs: Starting Chef Client, version 11.12.8
amazon-ebs: [2015-06-19T09:08:07-04:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
amazon-ebs: Chef Client failed. 0 resources updated in 2.583636316 seconds
amazon-ebs: [2015-06-19T09:08:07-04:00] ERROR: Unable to create Chef::RunList::RunListItem from String:"recipe[my-cereal-box] , recipe[foo::jenkins]": must be recipe[] or role[]
amazon-ebs: [2015-06-19T09:08:07-04:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

Furthermore, stripping the quotes off this line doesn't work either:
Failed to parse template: Error in line 83, char 21: invalid character '{' looking for beginning of object key string
"run_list": [ {{user recipe}} ],

So either support arrays in "Variables" or at least intelligently handle the form: "[ "element1", "element 2 with spaces or embedded special characters", "element 3" ... ]"

I can see why people give up in disgust dealing with packer's parser and just feed it JSON generated from something else that has more smarts.

[drichards-jut]

I am also having the same issues. I feel like I have tried everything to dynamically pass the run list.

[alovak]

@tb3088 basically has the same issue now :( and going to generate whole json template because I can't pass run_list in any way...

[tewner]

Hmmm. I'm also having this issue - I can't use -var-file to configure things like blockDeviceMaps for the AMI Builders, even using the samples included in the documentation (https://www.packer.io/docs/builders/amazon-ebs.html#availability_zone) :

"ami_block_device_mappings": [
{
  "device_name": "/dev/sdb",
  "virtual_name": "ephemeral0"
},
{
  "device_name": "/dev/sdc",
  "virtual_name": "ephemeral1"
}

]

gives me:
invalid value "/tmp/my.json" for flag -var-file: Error reading variables in '/tmp/my.json': json: cannot unmarshal array into Go value of type string

If we have the option configure the template variables using JSON files, it seems reasonable to expect JSON Arrays and Objects to work...

[kpettijohn]

@drichards-jut @alovak along with anyone attempting to pass a chef run_list to Packer I was able to get things working by passing a comma separated string as @mitchellh suggested.

"chef_run_list": "recipe[apt],recipe[blacksmith],recipe[rsyslog],recipe[rsyslog::client]"

Packer Variables

{
  "aws_region": "us-west-2",
  "aws_account_id": "0123456789",
  "aws_iam_instance_profile": "my_profile",
  "aws_security_group_ids": "sg-012345",
  "aws_instance_type": "m3.xlarge",
  "aws_ssh_username": "user",
  "ami_source": "ami-012345",
  "ami_name": "my-ami",
  "ami_user_data_file": "user-data",
  "ami_tool_x509_key_path": "my_aws.key.pem",
  "ami_tool_x509_cert_path": "my_aws.csr.pem",
  "ami_s3_bucket": "ami-storage",
  "chef_dir": "/var/chef",
  "chef_run_list": "recipe[apt],recipe[blacksmith],recipe[rsyslog],recipe[rsyslog::client]"
}

chef-client

    {
      "type": "chef-client",
      "server_url": "http://localhost:8889",
      "config_template": "templates/client.rb.tpl",
      "skip_clean_node": true,
      "skip_clean_client": true,
      "skip_install": true,
      "run_list": "{{user `chef_run_list`}}"
    }

[alovak]

@kpettijohn I confirm that your solution works!

[rossedman]

@kpettijohn @alovak You guys just saved my codebase from so much duplication!

[b-ryan]

Can this please be documented? I really would expect this to be in https://www.packer.io/docs/templates/user-variables.html and if it had, it would have saved me several hours of work. I just happened to finally google the right thing to end up here.

[kpettijohn]

@b-ryan I struggled finding the solution as well but to be fair its not necessarily generic enough to be under user-variables in my opinion.

The run_list parameter does mention it should be a array of strings but its not necessarily clear.

Its a bit buried (bottom of the page) on the Chef Client Provisioner page but it does include an example of using the Chef Client in local-mode.

https://www.packer.io/docs/provisioners/chef-client.html

[daaru00]

I have same problem on ami_users property in amazon-ebs builder.

I've try to do something like this:

"ami_users": "[ {{user `aws_share_ami`}} ]"

and in variables file

"aws_share_ami": "00000000000,00000000000",

but I receiving a runtime error

Error modify AMI attributes: InvalidAMIAttributeItemValue: Invalid attribute item value "[ 00000000000 " for userId item type.

[dragon788]

You are nesting the array. If the field already supports the list syntax,
just supply the user variable syntax { user varname } and it will unpack
it as an array if you quote each individual element inside the user
variable.

On Feb 14, 2017 5:27 AM, "Fabio Gollinucci" [e-mail]
wrote:

I have same problem on ami_users property in amazon-ebs builder
https://www.packer.io/docs/builders/amazon-ebs.html.

I've try to do something like this:

"ami_users": "[ {{user aws_share_ami}} ]"

and in variables file

"aws_share_ami": "00000000000,00000000000",

but I receiving a runtime error

Error modify AMI attributes: InvalidAMIAttributeItemValue: Invalid
attribute item value "[ 00000000000 " for userId item type.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/mitchellh/packer/issues/2278#issuecomment-279682378,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAdxXmiPfVs3fXcRnSKl8mKhRLuBrOR3ks5rcY-JgaJpZM4FGslB
.

[daaru00]

Thank you. It works, can you add on the documentation which properties support this feature? Somethings like this:

ami_users (array of strings or string with comma-separated values)

[rickard-von-essen]

@daaru00 all do.

[ikari7789]

Why is this being accepted as a valid way to handle arrays in variables? What if you have a rather large array of values or an actual need for a comma in a variable? In no way is this a logical way to handle variablized arrays. Why is it not possible for Packer to parse a JSON array into a variable?

[tb3088]

because it's a "design choice" of Hashicorp-authored software projects to pretend the only datatype end-users are allowed to supply is String. And I haven't yet gone in and rewritten the input processor. It's present in Terreform too. I was writing support for CloudHSM using ExternalDatasource and despite every tool on earth using JSON, it was written to only accept String Object so you had to mangle the input JSON into a string which it then happily unpacked. Huh?
When I learned programming it was self-evident that you accepted all reasonable datatypes as inputs and if the internal implementation was brain-dead it was the tool author's responsibility to mangle the input and not the caller's responsibility to do so. It's inscrutable to me that if Go can throw an unmarshal exception why it's not caught and the obvious conversion done transparently.

[SwampDragons]

Hey @tb3088. I'm sorry that this is frustrating for you. However, the tone of your comment is pretty hurtful. I'd like you to read over the HashiCorp community guidelines, since you have a history of making abrasive comments both in this issue and in other issues and PRs. We are all here because we want to make Packer better. What is inscrutable to you may have historical or architectural reasons you haven't considered. Or it may just have not been well thought-through, but even in that situation, being hostile or implying that Packer's developers are somehow delinquent is not going to make anyone resolve any issues any faster; instead it is a distraction that detracts from your actual concerns. We are all trying to make a good tool that works well for as many people as possible; please try to remember that positive intent when you are interacting with our community. https://www.hashicorp.com/community-guidelines

[dilchenko]

This might not be obvious to some, as it was not for me, but it seems like a comma-separated string in question cannot contain whitespaces between commas.

E.g. based on the example above(https://github.com/hashicorp/packer/issues/2278#issuecomment-135585464), this:

"chef_run_list": "recipe[apt],recipe[blacksmith],recipe[rsyslog]"

would result in an array as expected, with all recipes, but this:

"chef_run_list": "recipe[apt], recipe[blacksmith], recipe[rsyslog]"

would result in an array with only first recipe in it:

[ "recipe[apt]" ]

[tb3088]

That's a totally avoidable defect in the parser @dilchenko. Open it as a separate bug.

[rickard-von-essen]

Let's not use four year old (closed) issue as a bulletin board. Open new issues or use the user mailing list.