Packer: HCL2: implementation list

[azr]

HCL2 support is a beta work in progress and features are added one by one. As a result it can be hard for end users to know where we are at. While the full list HCL2 issues can be found using the HCL2 label:

• open HCL2 issues
• closed HCL2 issues

Here is a high level view of the todos and dones:

• [x] packer build
  
  
  
  • [x] -only -except
  
  
• [x] packer validate #9346
• [x] packer console #9359
• [x] packer inspect #9468
• packer fix ( will only come when a breaking HCL2 change will happen and this is needed)
• [x] packer fmt #10225
• [x] JSON to HCL2 transpiler #9659
• [x] Add a build.source block that allows to override specific builder fields #9167
• [x] HCL2 variables, terraform like ( input vars and locals )
• [x] HCL2 dynamic stanza
• [x] min_packer_version becomes packer.required_version https://github.com/hashicorp/packer/issues/9284
• Setting common fields from config:
  
  
  
  • [x] build { name = "xxx" can be set and seen in the logs #9245
  
  
  • [x] can be used as ${build.name} #10114
  
  
  • [x] source ... { name = "xxx" can be set and used as ${source.name} https://github.com/hashicorp/packer/pull/8956
  
  
  • [x] source "type" "...." type can be set and used as ${source.type} https://github.com/hashicorp/packer/pull/8956
  
  
  • [x] provisioner.pause_before
  
  
  • [x] provisioner.name
  
  
  • [x] provisioner.max_retries
  
  
  • [x] provisioner.timeout
  
  
• Templating/interpolation {{...}}:
  ⚠️ We recommend using HCL2 functions (docs) and variables (docs) if you can. The support for templating is going to stay for backwards compatibility but I would like to get rid of it. In HCL2 the interpolation is done at the HCL2 layer, whereas currently templating needs to be enabled for each string/value causing it to not work everywhere; also the HCL2 templating often comes first and the go templating comes second, mixing them looks and behaves confusingly. ⚠️
  
  
  
  • functions.
    
    
    
    
    
    • [x] {{timestamp}}; HCL2 equivalent: timestamp
    
    
    
    • [x] {{isotime}}; HCL2 equivalent: formatdate
    
    
    
    • [x] {{packer_version}} becomes packer.version #10117
    
    
    
    • [x] {{uuid}}; HCL2 equivalent : uuidv4, uuidv5
    
    
    
    • [x] {{lower}}; HCL2 equivalent lower
    
    
    
    • [x] {{replace}} - {{replace_all}}; replace and regexreplace
    
    
    
    • [x] {{split}}; HCL2 split
    
    
    
    • [x] {{upper}}; HCL2 upper
    
    
    
    • [x] {{clean_resource_name}}; This will not be a 1:1 replacement; instead we will do variable validation and will error at build time with a recommended regex_replace to fix variable names if they do not match the specific cloud's needs. #10148 https://github.com/hashicorp/packer/pull/10206
    
    
    
  
  
  • pwd functions
    
    
    
    
    
    • ~{{template_dir}}~ use path.root
    
    
    
    • ~{{pwd}}~ use path.pwd
    
    
    
  
  
  • Variables
    
    
    
    
    
    • ~{{user ...}}~ use var.name instead
    
    
    
    • ~{{env}}~ use var.name instead
    
    
    
  
  
  • integrations
    
    
    
    
    
    • [x] {{consul_key ...}} https://github.com/hashicorp/packer/pull/10119
    
    
    
    • [x] {{vault ...}} https://github.com/hashicorp/packer/pull/9746
    
    
    
    • [x] {{aws_secretsmanager ...}} #10124
    
    
    
  
  
  • Builder variables https://github.com/hashicorp/packer/pull/9444
    
    
    
    
    
    • [x] {{build ...}}
    
    
    
    • [x] {{build_name}}
    
    
    
    • [x] {{build_type}}
    
    
    
  
  

As a reminder, contributions are always welcome 🙂.

[marcosdiez]

Hi! Could you please add packer fmt, which works just like terraform fmt ? It's really useful. Currently I am making symlinks from myfile.pck.hcl to myfile.tf to have them formatted

[azr]

Hello @marcosdiez ! That's a great idea ! I initially wanted to put this under a packer validate --inplace command but calling it packer fmt will allow to be more consistent with Terraform.

[deric4]

i've been using the hclfmt command from the hashicorp/hcl repo https://github.com/hashicorp/hcl/tree/hcl2/cmd/hclfmt.

its not perfect but running hclfmt -w *.hcl has been working fairly well so far.

https://hub.docker.com/r/deric4/hclfmt

[mschuchard]

Is there a way for me to subscribe specifically to packer validate/fmt/fix, whether an issue, WIP PR, and/or project task? It would be helpful for me to update the Jenkins Pipeline and Atom Editor integrations with Packer.
Thanks!

[SwampDragons]

Unfortunately, we don't have granular updates like that. The best you can do is keep an eye on the CHANGELOG.

[azr]

@mschuchard I just wanted to add that the validate command is in the 1.6.0 version, fix is not a feature of HCL2 yet and it will probably come when HCL2 is stable and then we want to introduce a breaking change.

Lastly (as stated by @/mschuchard ) hclfmt seems like a good replacement for packer fmt for now. 🙂 I hope that helps.

[drewcarmichael-okta]

where are the vault and consul integrations in the road map ? seems that both of those should be easy given the existing TF implementation of both of those /

[SwampDragons]

I'm hoping to have the vault and consul integrations by 1.6.1

[upodroid]

For clean resource name, i have been doing ${lower(regex_replace(timestamp(), ":", "-"))} which works nicely. Took me a while to figure it out

[azr]

Hey @upodroid ! Thanks for sharing this, packer console could help a little to visually sort out regexes quicker here. 🙂

[upodroid]

@azr it would be great if packer had a time function that returns unix time. because we could do

image_name = "${var.image_name_prefix}-${unix_time()}"

instead of:

image_name = "${var.image_name_prefix}-${lower(regex_replace(timestamp(), ":", "-"))}"

[azr]

@upodroid, you could also use: formatdate("YYYYMMDDhhmmss", timestamp()) or something like:

locals {
  timestamp = "${timestamp()}"
  timestamp_sanitized = "${regex_replace("${local.timestamp}", "[- TZ:]", "")}"
}

timestamp returns an RFC3339 date, just like Terraform so for parity with Terraform I think it would be better to just keep it that way ( and not add unix_time ) but at the same time Packer JSON always had unit_time, so I'm hesitant. Please open a new issue if you think this is important 🙂 !

[universalvishwa]

Can you please address #7915

[olfway]

It seems that aws_secretsmanager is not supported in hcl2 too?

[cbednarski]

This is awesome, Packer team! Thanks for all the hard work on HCL2. It's soooo nice to use white space and comments in my packer templates. 😁 👍

[azr]

Hey @cbednarski thanks ! 😁 I (we) like it too !!! Hopefully we can drop the beta flag soon and start adding bigger features 👍 . :shipit:

[tlindsay42]

Any plans for an equivalent for the sensitive-variables feature?

[joshuaspence]

@tlindsay42 I think you can do that with the variable block:

variable "wifi_password" {
  type      = string
  default   = ""
  sensitive = true
}

[tlindsay42]

Didn't work for me, but that's how I'd like it to work.

[azr]

Hey @tlindsay42 ! Thanks for reporting; that is most likely something to fix, I'm opening a new issue.

Edit: link here: https://github.com/hashicorp/packer/issues/9939

[nitrocode]

Nice work! Once this is implemented, would we be able to create terraform modules that can be reused? We have a set of golden AMIs and many packer files in different repos that reuse our golden AMIs but use their own packer configurations so sometimes tagging and other items are inconsistent. I'd love to set a bunch of defaults in hcl with our base golden amis that child packer configurations can inherit and overwrite.

Also, I was waiting for this feature to be fully implemented before I try it, but then it dawned on me that there's this popular repo for terraform ecs task definitions which dumps out json.

https://github.com/cloudposse/terraform-aws-ecs-container-definition/blob/master/main.tf#L90

What's the advantage of using packer hcl vs a terraform module that dumps out json and then running packer build on the output ?

[azr]

Hey @nitrocode

We have a set of golden AMIs and many packer files in different repos that reuse our golden AMIs but use their own packer configurations so sometimes tagging and other items are inconsistent.

I'd love to set a bunch of defaults in hcl with our base golden amis that child packer configurations can inherit and overwrite.

This could be solved by two enhancements we have in mind here:

• create base templates that we can 'inherit' from or import somewhere and optionally overload. In a stateless sense.
• support some sort of state after a build and build images on top of other artifacts.

These are kinda different but I think they are good ideas :). We first would like to deprecate Packer Classical-JSON before starting to think about adding nice features like these !

What's the advantage of using packer hcl vs a terraform module that dumps out json and then running packer build on the output ?

The manifest post-processor will dump a JSON file containing was Packer built as well; so I think both should work.

[nitrocode]

This could be solved by two enhancements we have in mind here:

Cool, looking forward to it. In the past, we also used racker and doing something packer native would be better.

The manifest post-processor will dump a JSON file containing was Packer built as well; so I think both should work.

The manifest post-processor and the reference to the terraform module that outputs json are two different types of json processors. The former is dumping the json outputted by the post-processor (post packer build) while the latter would dump out the json that would be used by the packer build command.

[melck]

hello, I'm glad I can use a more flexible language than JSON.

Until now I have been using yaml. I am starting to use HCL and Packer (I have more training with Terraform).

Do you plan to add this kind of feature :

• for_each like Terraform for sources ?

• I want a way to merge a map with default variable in a source (like vmware-iso), ex :

  locals {
      default_vmware_iso = {
          ssh_username = "root" 
          ssh_password = "root" 
          ssh_timeout = 15 
      }
  }
  
  source "vmware-iso" "vagrant" {
      default_vmware_iso...
  }
  

[azr]

Hello @melck ! I'm happy to hear you picked HCL2 over YAML !! Yes we plan on supporting the for_each tag on specific blocks 👍🏼 . But before adding more features to HCL2 we want to tag HCL2 as "stable" and so reach feature parity with JSON. Hang in there 🙂 . For a list of blocks that will support the for_each statement, this is not 100% sure but I was thinking source and the source block in a build for now.

[zmingxie]

Hi @azr, I reported this HCL2 issue in the discussion forum, but I can't seem to find an active GitHub issue for it. Could you confirm if this is being looked after? https://discuss.hashicorp.com/t/getting-the-source-ami-id-when-using-hcl-configuration/9911/7?u=zmingxie

[jparr93]

Anyone had any joy implementing HCL packer within a devops pipeline at all? Doesn't seem to want to work

[azr]

@zmingxie, this issue was fixed by Megan here: https://github.com/hashicorp/packer/pull/10224. You can already try it using packer's latest nightly build.

Hey @jparr93, sorry to hear it was not a joy for you, if you have any specific pain points you want to talk about or feedback: please open a new issue, we're really eager to hear your thoughts on this as it can help us improve the process, docs, etc. 🙂 .

[s11k]

@azr:

For a list of blocks that will support the for_each statement, this is not 100% sure but I was thinking source and the source block in a build for now.

I just started using Packer + HCL2 for the first time in the last few days and came across the need for for_each. Glad to hear that you're thinking about it!

Thanks for all the hard work going into adding support for HCL2!

[thomaswors]

Anyone had any joy implementing HCL packer within a devops pipeline at all? Doesn't seem to want to work

Groovy / Jenkins:

pipeline {
    parameters {
        string( name: "packerVersion",
                description: "Version of Packer to use. Needs to be uploaded to GS bucket",
                defaultValue: "packer_1.6.4"  /* update this if needed */
                )
    }
    environment {
        imageVersion = "0-0-1-${BUILD_NUMBER}"
        packerFile = "test-linux.pkr.hcl"
    }
    agent { label 'npe' }
    stages {
        stage('Bake Image') {
            steps {
                git branch: 'test-branch-2',
                credentialsId: 'github',
                url: "https://github.com/coolrepo.git"

                script {
                    sh "gsutil cp gs://mr-bucket/installers/${packerVersion} packer"
                    sh 'chmod +x packer'
                    sh "./packer build ${packerFile}"
                }
            }
        }
    }
    post {
        cleanup {
            deleteDir()
            dir("${workspace}@tmp") {
                deleteDir()
            }
        }
    }
}

[thomaswors]

Issue running HCL; their docs do not match up with the HCL expectations. Also tend to only mention .json examples
see : https://www.packer.io/docs/builders/googlecompute#image_storage_locations

Error:
Inappropriate value for attribute "source_image_project_id": list of string
required.

According to https://www.packer.io/docs/builders/googlecompute#source_image_project_id , you should just be able to define it as:

source "googlecompute" "test-packer" {
  project_id = "redacted"
  zone = "redacted"
  source_image_project_id = "redacted"
truncated the rest including other required things
}

but this does not work, leads to error above

[thomaswors]

Issue running HCL; their docs do not match up with the HCL expectations. Also tend to only mention .json examples

Error:
Inappropriate value for attribute "source_image_project_id": list of string
required.

According to https://www.packer.io/docs/builders/googlecompute#source_image_project_id , you should just be able to define it as:

source "googlecompute" "test-packer" {
  project_id = "redacted"
  zone = "redacted"
  source_image_project_id = "redacted"
truncated the rest including other required things
}

but this does not work, leads to error above

Fixed it by adding the values in [ ], however the document says all the values are strings, so why does one string need it and others don't? or just update the docs

source "googlecompute" "test-packer" {
  project_id = "x"
  zone = "x"
  source_image_project_id = [
    "x"
    ]
  source_image_family = "centos-7"
  image_storage_locations = [
    "x"
    ]

[azr]

@thomaswors 👋🏼 🙂

source_image_project_id ([]string) - A list of project IDs to search for the source image. Packer will search the first project ID in the list first, and fall back to the next in the list, until it finds the source image.

While Legacy Packer JSON allows you to put a string in a field that is actually an array of strings, this is not possible in HCL2, the language is more "strongly typed". As the docs states, source_image_project_id is a list of strings.

I think the docs are correct here but my guess is that you used the hcl2upgrade command that simply transforms configs in-place without checking any types and this stayed a string instead of being changed to an array; and that's definitely an issue to me. I'm opening an issue to see if we can fix this one.

[thomaswors]

@azr Thanks for the reply, and i completely over looked the []String requirement in the documentation 😭 .

Also, i DID NOT run the hcl2upgrade tool, i just happened to create my own issues by not reading it close enough.
My complaint about HCL examples still stands however 😄

[azr]

My complaint about HCL examples still stands however 😄

Yes, that's something we are working on improving ! But it will take more time.
We used code generation to move very quickly to get HCL2 to where it is, but the docs are just manual currently. I have this idea in the back of my mind to also auto-generate examples with all fields set from the code... So that one could just copy paste what's necessary.

[queglay]

Will it be possible for packer to utilise modules and dependencies with this transition? By modules, I mean it would be excellent to:

• easily run a subset of builds (say multiple distros in a single module)
• build all images from many modules in parrallel (fastest, but perhaps sometimes wasteful)
• build images with dependencies on other images (efficient, but slower in some circumstances, however better for consistency)

[melck]

I didn't found anything to join path (like a function). I needed it to support Windows and Linux pathing.

[azr]

@melck In the HCL2 world every path separator is a slash (/) so that you don't have to do this, but maybe we will have to fix some things for this statement to be true; can you please open a new bug report for that one ? Thanks 🙂

[danpilch]

Is there any support for conditionals? for example (pseudo code) I'd like to include variables to a post processor if var.vars_enabled { do something } is this possible?