Packer: Use HCL

[delitescere]

It would be great to have YAML (or the HCL) for less cruft and also allow comments. (See https://github.com/mitchellh/packer/issues/283)

If JSON must be the configuration format, then it would be nice to allow comment-like key/values (perhaps where the names are prefixed with a comment-like glyph). For example:

{
  "builders": [
...
  ],
  "provisioners": [
      {
      "#TODO": "turn this into a puppet manifest",
      "type": "shell",
      "script": "setup.sh"
    },
    {
      "type": "shell",
      "execute_command": "sudo -E /bin/bash '{{ .Path }}'",
      "#": "Run as sudo and remove all the things we don't need",
      "inline": ["DEBIAN_FRONTEND=noninteractive apt-get -y clean autoremove"]
    }
  ]
}

[delitescere]

cc @pandacalculus

[sethvargo]

HCL is coming soon :smile:

[nchammas]

@delitescere Is this issue not a duplicate of #283? I noticed you linked to it in the initial post, but it's not clear what the distinction between the two issues is.

[delitescere]

Here unknown keys would be handled differently to how they are now. This proposes "comment-like" keys which are ignored. The comment-ness is indicated by a generally known self-closing comment prefix (viz: #, //, ;)

It was an alternate suggestion given the larger code change to support comments in non-standard JSON, or JSON5 or YAML or HCL.

@sethvargo's comment about the move to support HCL resolves the issue.

[VladRassokhin]

@sethvargo

HCL is coming soon :smile:

Sorry to bother you, any updates on this?

[sethvargo]

It's still coming :smile:

[mitchellh]

We decided to delay this a bit, since we don't have an HCL writer yet and we'd like to maintain packer fix as an option for the next couple versions.

[chiefy]

I have drank the HCL kool-aid. :+1:

[jevonearth]

Now that https://github.com/hashicorp/hcl exisits, can we get an alternative to json? :)

[sethvargo]

@jevonearth that has existed for quite some time :smile:. HCL is still coming!

[xahare]

Since its still coming, is this a good time to feature request include files? see https://github.com/mkheironimus/packergen

@jevonearth heres an alternative till add hcl

[j1n6]

still coming soon?

[misham]

@mitchellh @sethvargo is this the HCL writer you mentioned: https://github.com/hashicorp/hcl/tree/master/hcl/printer

[Tatsh]

Waiting for YAML. Not really into HCL

[rickard-von-essen]

It's kind of trivial to wrap packer with a bash function and convert yaml to json.

To get you started:

ruby -r json -r yaml -e "yaml = YAML.load(File.read(\"$1\")); print yaml.to_json"

(and the reverse)

ruby -r json -r yaml -e "json = JSON.parse(File.read(\"$1\")); print YAML::dump(json)"

[Tatsh]

I already made that. It would be nice to not have to. That's why this ticket exists for HCL

On Jan 16, 2017, at 12:31 PM, Rickard von Essen [e-mail] wrote:

It's kind of trivial to wrap packer with a bash function and convert yaml to json.

To get you started:

ruby -r json -r yaml -e "yaml = YAML.load(File.read(\"$1\")); print yaml.to_json"
(and the reverse)

ruby -r json -r yaml -e "json = JSON.parse(File.read(\"$1\")); print YAML::dump(json)"
―
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[RichardBronosky]

HCL

HashiCorp needs to show a little faith in itself and use HCL everywhere it can. Either HCL was a good idea and they should run with it, or HCL was a mistake and Terraform is a disaster. Clearly the latter isn't true. So, as a new evaluator/adopter, I have to say that it's cause for concern that these 2 products feel like "two intelligence agencies that don't talk to one another."

I'm with @sethvargo. Let's don our direwolf sigil and proclaim that "HCL is coming". (Not only that, also make it happen.)

JSON

There is no need to make any changes to shoehorn comments into JSON. The solution was given to us by Douglas Crockford and exemplified by Nicholas Chammas in The Notorious Two Eight Three https://github.com/mitchellh/packer/issues/283#issuecomment-60039068

YAML

Ruby miners love them some YAML and gave us a solution to using it from its inception. This was wonderfully exemplified by Rickard von Essen 2 comments ago https://github.com/mitchellh/packer/issues/1768#issuecomment-272921740 (though I'd favor using ARGF.read as I do below)

Et cetera

I love that HashiCorp makes it easy to use as many or as few of their products as you need. It fits well with the unix philosophy (as summarized by Peter H. Salus in A Quarter-Century of Unix (1994)):

• Write programs that do one thing and do it well.
• Write programs to work together.
• Write programs to handle text streams, because that is a universal interface.

We should be fine with:

JSON

$ cat > example.json <<'EOF'
// from: https://www.packer.io/intro/getting-started/vagrant.html
{
    // can't we take this out?
    "variables": {
        "aws_access_key": "", // why is this blank?
        "aws_secret_key": ""  // ditto
    },
    "builders": [{
        "type": "amazon-ebs",
        "access_key": "{{user `aws_access_key`}}",
        "secret_key": "{{user `aws_secret_key`}}",
        "region": "us-west-2",
        "source_ami": "ami-b7a114d7",
        "instance_type": "t2.micro",
        "ssh_username": "ubuntu",
        "ami_name": "packer-example {{timestamp}}",
        "tags": {
            "created_by": "Packer.io/example"
        }
    }],
    "provisioners": [{
        "type": "shell",
        "inline": [
            "sleep 30",
            "sudo apt-get update",
            "sudo apt-get install -y redis-server"
        ]
    }],
    "post-processors": ["vagrant"]
}
EOF

$ packer validate <(jsmin < example.json)
Template validated successfully.

YAML

$ cat > example.yaml <<'EOF'
---
# from: https://www.packer.io/intro/getting-started/vagrant.html

# can't we take this out?
variables:
  aws_access_key: '' # why is this blank?
  aws_secret_key: '' # ditto

builders:
- type: amazon-ebs
  access_key: '{{user `aws_access_key`}}'
  secret_key: '{{user `aws_secret_key`}}'
  region: us-west-2
  source_ami: ami-b7a114d7
  instance_type: t2.micro
  ssh_username: ubuntu
  ami_name: packer-example {{timestamp}}
  tags:
    created_by: Packer.io/example

provisioners:
- type: shell
  inline:
  - sleep 30
  - sudo apt-get update
  - sudo apt-get install -y redis-server

post-processors:
- vagrant
EOF

$ packer validate <(ruby -r json -r yaml -e "print YAML.load(ARGF.read).to_json" < example.yaml)
Template validated successfully.

[vtolstov]

i'm create small pr for yaml templates, for my limited testing it works fine for me.

[mwhooker]

Definitely something we're planning to investigate. For now, though, since this not something we're targeting for 1.0, I'm going to close this. Please see the mailing list for more details on 1.0.

[VladRassokhin]

It's time to reopen issue back. Definitely HCL should be default configuration language in Packer.

PS: Still has no idea why almost all issues were closed, there's projects, milestones and tags to manage issues.

[mwhooker]

This is something we're planning to do. Since it's such a massive overhaul, we want to make sure we do it well. The next step for us is to go through our RFC process, which would be on a separate issue.

[darkn3rd]

I welcome JSON5, YAML, and HCL. Pure JSON is painful.

[sean-]

@darkn3rd We feel your pain and it is painful enough that we, Joyent, maintain a JSON5 branch that tracks master because the current situation is untenable for anyone who actually uses and maintains any real quantity of packer templates: https://github.com/joyent/packer/tree/f-json5

/me makes a mental note to update the branch this week

[duckie]

We are near 3 years of "coming soon"... I am a beginner with packer and I am astonished, to say the least, that it doesnt support HCL. Cant wait to test it.

[mkjmdski]

I just said my friend "hey lat's migrate that awful json to HCL, it's a hashicorp tool, right?". How wrong I was...

[alanbchristie]

If it's of interest yacker (a Python utility) wraps Packer in a YAML-to-JSON interpreter. It's nothing Earth-shattering - it just provides you with a YAML-wrapper that converts template and variable files temporarily to JSON before passing control to Packer.

https://yacker.readthedocs.io/en/latest/

[vtolstov]

Packer team? Why you still not provide HCL? I'm create small patch for YAML support almost more then year ago, that works fine and you reject it with reason, that HCL is coming. But after year nothing changed!
What you say?

[rickard-von-essen]

Still coming :wink:

[SwampDragons]

@vtolstov see #7023

[SwampDragons]

Internal product folks at HashiCorp have decided that we're going to skip straight over HCL1 and implement HCL2 instead; sorry this never got implemented but I promise we're actively working on HCL2 support right now.

[RichardBronosky]

Thank you for the update. We haven't given up on this.

[tdmalone]

Thank you @SwampDragons! Given this is actively being worked on, should this issue be reopened?

[SwampDragons]

Good point; it was accidentally closed when we closed the associated PR without merging.

[ndobbs]

Really excited for HCL2 support with packer!

[NasAmin]

@SwampDragons Any news on HCL please?

[SwampDragons]

We're hoping (but not promising) to get it out it with the 1.5.0 release. The team has a working prototype in a branch but it's not polished or hardened yet and with @azr about to be on parental leave we may end up having to slip it a bit later. Goal is by the end of the year, preferably closer to September than December.

[NickLarsenNZ]

I came back after some years and was surprised this was still open.
Glad it's on it's way in 1.5.

@SwampDragons Do you have any rough timeframes for when 1.5.0 will be available (even as a beta)?

[alanbchristie]

If the change is complicated (and there may be very good reasons for that) then isn't it a much lower development risk and cost to leave a perfectly good tool alone and instead, rather than changing the core definition format, provide _adapters_ that convert format-X to JSON?

Just a thought.

I don't want to trivialise the problem and you may have really important reasons to change the core format but, considering the time and effort that's being spent on this, if it's just about how users define their files then, as a project, is there a need for _language adapters_ like yacker?

[SwampDragons]

@alanbchristie To be fair, it's not been actively worked on this whole time; it's been a back burner "we should think about this" issue for a very long time, most recently because we wanted to watch and see how the HCL2 rollout went with Terraform 0.12 so we could learn from their process before trying to do it again ourselves. I don't think it will be nearly as arduous a process as it was for Terraform, but I can't speak to the details because like I said it's a project @azr was working on and he's off experiencing parental bliss at the moment. ;) When he gets back I'll ask him if he can write up a little progress update for y'all.

[SwampDragons]

@NickLarsenNZ The roadmap I show my boss says sometime around October or November. If you want, I can ping you once we have a tryable build.

[ruckc]

I'm new to packer... but i'll be honest, the fact this has been an unaddressed issue for 5-6 years is not only fairly disheartening, but extremely frustrating.

Using jq to strip comments while a workaround, means there are probably 99 comments that went unwritten for every 1 comment written with jq to strip.

I could go on for days how json, as implemented by packer, is not suitable to this task, but the fact it doesn't allow comments and encourages code/config duplication (why is it i feel like i need to write a packer json pseudo-compiler to enable config reuse).

[SwampDragons]

Locked to reduce noise; @azr is back and I'll have him give an update on this issue, including providing beta builds, when he has a chance.

[azr]

Hello everyone ! I have some good news 🎉 😃

I recently took quite some time to make the HCL2 move happen and you can find a PR implementing it here: #8423. Some binaries can be found here.
We are doing our best to test a lot of different cases to avoid bugs but there are a lot of builders/provisioners/post-processors and we are a small team so any help will be super welcome for testing this out ! If you do try this out and find a bug; please open a new issue stating that it's an HCL bug.

The current (old) way of setting up Packer will remain working and we currently plan on fully supporting both versions for a while until we feel confident enough to deprecate the old one.

Example hcl file

build {
    sources = [
        "source.amazon-ebs.first",
    ]

    provisioner "shell" {
        inline = [
            "sleep 5"
        ]
    }

    post-processor "shell-local" {
        inline = [
            "sleep 5"
        ]
    }
}

source "amazon-ebs" "first" {
    // access_key = "{{user `aws_access_key`}}"
    // secret_key = "{{user `aws_secret_key`}}"

    ami_name = "hcl2-test"
    region = "us-east-1"
    instance_type = "t2.micro"

    kms_key_id = "c729958f-c6ba-44cd-ab39-35ab68ce0a6c"
    encrypt_boot = true
    source_ami_filter {
        filters {
          virtualization-type = "hvm"
          name =  "amzn-ami-hvm-????.??.?.????????-x86_64-gp2"
          root-device-type = "ebs"
        }
        most_recent = true
        owners = ["amazon"]
    }
    launch_block_device_mappings {
        device_name = "/dev/xvda"
        volume_size = 20
        volume_type = "gp2"
        delete_on_termination = "true"
    }
    launch_block_device_mappings {
        device_name = "/dev/xvdf"
        volume_size = 500
        volume_type = "gp2"
        delete_on_termination = true
        encrypted = true
    }

    ami_regions = ["eu-central-1"]
    run_tags {
        Name = "packer-solr-something"
        stack-name = "DevOps Tools"
    }

    communicator = "ssh"
    ssh_pty = true
    ssh_username = "ec2-user"
    associate_public_ip_address = true
}

PS: docs and variables handling are coming !

[NickLarsenNZ]

This is great news @azr, thanks to you and the team.

[mexisme]

WRT: deprecating JSON.
Is it right that you plan to leverage HCL / HCL2's built-in JSON support?

[darkn3rd]

Is there a json2hcl converter? I have well made packer scripts from Netflix OSS that I would love to convert to HCL.

[azr]

Hey everyone, I'm excited about this change !

@darkn3rd I plan on doing a packer specific converter so users can quickly try it without too much copy paste & edit work.

@mexisme The HCL library fully understand JSON but the format is changed a little so there is going to need some edits. I will try to have the converter convert a file to JSON too ( or something ).

---

⚠️ my first example has changed and looks like terraform a bit more.

Example hcl file

build {
    sources = [
        "source.amazon-ebs.first",
    ]

    provisioner "shell" {
        inline = [
            "sleep 5"
        ]
    }

    post-processor "shell-local" {
        inline = [
            "sleep 5"
        ]
    }
}

source "amazon-ebs" "first" {
    // access_key = "{{user `aws_access_key`}}"
    // secret_key = "{{user `aws_secret_key`}}"

    ami_name = "hcl2-test"
    region = "us-east-1"
    instance_type = "t2.micro"

    kms_key_id = "c729958f-c6ba-44cd-ab39-35ab68ce0a6c"
    encrypt_boot = true
    source_ami_filter {
        filters {
          virtualization-type = "hvm"
          name =  "amzn-ami-hvm-????.??.?.????????-x86_64-gp2"
          root-device-type = "ebs"
        }
        most_recent = true
        owners = ["amazon"]
    }
    launch_block_device_mappings {
        device_name = "/dev/xvda"
        volume_size = 20
        volume_type = "gp2"
        delete_on_termination = "true"
    }
    launch_block_device_mappings {
        device_name = "/dev/xvdf"
        volume_size = 500
        volume_type = "gp2"
        delete_on_termination = true
        encrypted = true
    }

    ami_regions = ["eu-central-1"]
    run_tags {
        Name = "packer-solr-something"
        stack-name = "DevOps Tools"
    }

    communicator = "ssh"
    ssh_pty = true
    ssh_username = "ec2-user"
    associate_public_ip_address = true
}

New binaries can be found here

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.