Packer: "environment_vars" support for the shell-local provisioner

[josh-padnick]

I'm using the amazon-chroot builder to create an AMI with multiple partitions. Because of this, I make heavy of the shell-local provisioner. My issue is that I want to pass a secret from my local environment into the shell-local provisioner, but there's no explicit support for the environment_vars property.

Therefore, my code winds up looking like this:

{
     "variables": {
          "github_auth_token": "{{env `GITHUB_OAUTH_TOKEN`}}",
     }
}
...
{
      "type": "shell-local",
      "command": "GITHUB_OAUTH_TOKEN={{user `github_auth_token`}}; some_command
},

This executes as expected, but Packer dutifully outputs the secret value in the logs:

==> amazon-chroot: Executing local command: GITHUB_OAUTH_TOKEN=<redacted>; some_command

I'd prefer to solve this problem with the "environment_vars" property being added to the shell-local provisioner. Any thought on how difficult this would be to add?

[mwhooker]

Could you set the variables outside of packer, and then just reference them directly from the shell-local scripts?

MY_VAR=foo packer build ...?

FOO=bar packer build shell-local.json
docker output will be in this color.

==> docker: Creating a temporary directory for sharing data...
==> docker: Pulling Docker image: ubuntu:xenial
    docker: xenial: Pulling from library/ubuntu
    docker: Digest: sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950
    docker: Status: Image is up to date for ubuntu:xenial
==> docker: Starting docker container...
    docker: Run command: docker run -v /Users/mwhooker/.packer.d/tmp/packer-docker840343318:/packer-files -d -i -t ubuntu:xenial /bin/bash
    docker: Container ID: 1bff55e69d5c545dfc9ae32b203d400f4bd1f9158ae82a44ba87d631ee72f05d
==> docker: Executing local command: echo $FOO
    docker: bar
==> docker: Killing the container: 1bff55e69d5c545dfc9ae32b203d400f4bd1f9158ae82a44ba87d631ee72f05d
Build 'docker' finished.

==> Builds finished. The artifacts of successful builds are:
--> docker: Exported Docker file:
~/dev/packertest ᐅ cat shell-local.json
{
  "builders": [
    {
      "type": "docker",
      "image": "ubuntu:xenial",
      "discard": true
    }
  ],
  "provisioners": [
    {
      "type": "shell-local",
      "command": "echo $FOO"
    }
  ]
}

[kwilczynski]

@josh-padnick it would not be hard to add (it would need to support Windows too), but as per @mwhooker, the environment variables (by default) should be inherited from the shell you used to start Packer from.

I rely on this (which is a default behaviour for parent-child interaction on most *nix platforms) to pass details about Jenkins build into the resulting image (as I store some details about the build environmnet metadata inside the image itself).

Would that be acceptable?

[josh-padnick]

@mwhooker, @kwilczynski: Thanks for your input! My main issue with the proposed solution is that I won't get any validation from Packer saying that a variable I was supposed to set is empty. Where I work we're building infra as code to be used by other software teams, so these kinds of protections are important for my use case.

My current workaround looks like this:

{
      "type": "shell",
      "environment_vars": [ "GITHUB_OAUTH_TOKEN={{user `github_oauth_token`}}" ],
      "inline": [
        "echo 'Writing GITHUB_OAUTH_TOKEN to /tmp/env-vars in chroot environment to allow shell-local provisioner to read env vars without outputting their value to Packer logs.'",
        "echo \"GITHUB_OAUTH_TOKEN=$GITHUB_OAUTH_TOKEN\" > /tmp/env-vars"
      ]
}

[mwhooker]

@josh-padnick that's a great point. (on another note, if/when we switch to HCL as a configuration language, we'll be able to offer finer-tuned control over which variables are allowed to be printed in debug logs).

As I think I've mentioned elsewhere, there is an ongoing debate internally about the shell-local provisioner. It doesn't exactly fit packer's stated use-case, and it's caused us lots of trouble, but this definitely seems like something we'd need if it stays around.

Off topic slightly, but for your use-case, is there anything crucial packer provides with the shell-local provisioner that you couldn't get by distributing a script, instead? ease of use counts!

[jamesrgrinter]

I'm relatively new to Packer.

What I get, so far, is that the template remains the sole source of knowledge about what steps are involved in the build process. Scripts to invoke packer itself can remain merely convenience (with optimisations for repeating one step over and over whilst testing.)

I was surprised to just find that the shell-local provisioner does not have the same environnent_vars functionality as the shell provisioner, and the shell-local post processor! I had already used it with shell and assumed it would work the same way! Which is why I ended up here.

[josh-padnick]

@mwhooker

if/when we switch to HCL as a configuration language ...

This would be a welcome change! I miss the lack of commenting in JSON and would love to be able to organize my Packer templates similarly to Terraform templates (e.g. vars.tf, main.tf)

For your use-case, is there anything crucial packer provides with the shell-local provisioner that you couldn't get by distributing a script, instead? ease of use counts!

I see two major use cases for shell-local Amazon-Chroot:

1. Builds are faster because you avoid the launch/destroy overhead.
2. You can manage disk partitions for an AMI since the volume your partitioning is on a different sever than the one running Packer.

My use case falls under number 2 above, though frankly I would prefer to use the EBS Surrogate builder if I could since I had to write a Terraform template just to launch an EC2 Instance from which Packer is launched, as well as a wrapper bash script to allow Terraform apply, run Packer, and destroy all in one operation.

If EBS Surrogate achieved parity with amazon-chroot in terms of the options, that would meet my use case number 2 above. Maybe that could meet use case number 1 as well if you just add an option to EBS Surrogate like "just launch an EC2 Instance but don't terminate it."

Update: Ah, sorry, I'm confusing shell-local and the Amazon Chroot builders. My bad. Anyway, my point is that, when using Amazon-Chroot, I have to use shell-local so that I can perform disk operations from the Host EC2 Instance to the mounted EBS Volume. See https://www.youtube.com/watch?v=8h_Y-L1Q8xI& for additional details.

[dhs-rec]

Workaround available or not, is there any chance to get this fixed? It's just annoying that this seems to be the only shell-based provider/post-processor that works differently than all the others.

[kwilczynski]

@dhs-rec I would be happy to add support on Linux et al, and try to do it on Windows, but someone would have to test it on Windows, as I am not a Windows person.

[SwampDragons]

Just as a matter of practicality, if there's a workaround available, then the priority for fixing an issue is lower. We'll absolutely review PRs for it but are unlikely to create any of our own in the near future.

[dhs-rec]

@kwilczynski, while I am a Linux person, too, I have some Windows systems at hand where I could test this.

[mrmaxsteel]

+1 this would be very useful - I'm using packer for both ssh provisioners and API driven provisioners (helm for kubernetes, for example). The null builder (with communicator 'none') is really useful for the latter. 'environment_vars' is available for the 'shell' provisioner, why not make it available for 'shell_local' (along with all the configuration options available to the 'shell' provisioner), to help packer become a first class tool for api driven provisioners as well?

[mrmaxsteel]

Acutally, looks like this was resolved recently with #5956

[SwampDragons]

Good point!

[tamsky]

Thanks and kudos to @SwampDragons !

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.