Packer: Passing variables to shell provisioner environment variables

Created on 18 Oct 2017  ·  14Comments  ·  Source: hashicorp/packer

I would like to be able to pass my local environment variables through the environment variables of the shell. Here is my current json setup:

{
  "variables": {
    "aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}",
    "aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}"
  },
  "builders": [{
    "type": "amazon-ebs",
    "access_key": "{{user `aws_access_key`}}",
    "secret_key": "{{user `aws_secret_key`}}",
    "region": "us-east-1",
    "source_ami": "ami-e392eef5",
    "instance_type": "t2.micro",
    "ssh_username": "ec2-user",
    "security_group_id": "sg-065e6d7c",
    "subnet_id": "subnet-a70c3a8f",
    "ami_name": "packer-example {{timestamp}}"
  }],
  "provisioners": [{
  "type": "shell",
  "execute_command": "{{.Vars}} sudo -E -S bash '{{.Path}}'",
  "scripts": [
    "0_Provisioner_AWS_Setup.sh",
    "1_Provisioner_Initialize_AMI.sh",
    "2_Provisioner_Chapter1_Host_Configuration.sh",
    "3_Provisioner_Chapter2_Host_Configuration.sh",
    "4_Provisioner_Benchmark_Setup.sh",
    "5_Provisioner_Splunk.sh",
    "6_Provisioner_Qualys.sh",
    "7_Provisioner_System_Restart.sh",
    "8_Processor_Run_Benchmarks.sh"
  ],
  "environment_vars": [
    "DOCKER_CONTENT_TRUST=1",
    "access_key={{env `AWS_ACCESS_KEY_ID`}}",
    "secret_key={{env `AWS_SECRET_ACCESS_KEY`}}"
  ]
  }]
}

I am needing to pass through the environment variables because I need to be able to run the AWS CLI from within the EC2 Instance and I need to pass through my credentials so I can set up the CLI for the first time.

Thanks everyone!

provisioneshell question
Source
picture LuckyStubs

Most helpful comment

I had the same exact problem today. When I was validating that the environment variables were existing, they were. An echo $MY_VAR was outputing the actual value of the variable but packer was unable to read the value from the environment and pass it along the user variables scope.

The way I got it fixed was by creating a file named ~/.packer then inside the file I couldn't just type MY_VAR=VALUE and source it normally. I had to force the export of the environment variable within the file.

export MY_VAR=VALUE

Then when I source this file followed by the packer command line, it worked out.

I got this issue in a Ubuntu shell from a Windows Subsystem for Linux (WSL). I guess that the injection of the environment variables doesn't work like a traditional bash on a native ubuntu shell.

picture drivard on 13 May 2019
👍2

All 14 comments

Here is the output. You can see that the two env variables are blank..

[1.0.10] I, [2017-10-18T20:44:04.703331+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: Env Variables!
[1.0.10] I, [2017-10-18T20:44:04.705830+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: secret_key=
[1.0.10] I, [2017-10-18T20:44:04.709895+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: LESS_TERMCAP_mb=ESC[01;31m
[1.0.10] I, [2017-10-18T20:44:04.711694+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: LESS_TERMCAP_md=ESC[01;38;5;208m
[1.0.10] I, [2017-10-18T20:44:04.714705+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: LESS_TERMCAP_me=ESC[0m
[1.0.10] I, [2017-10-18T20:44:04.715611+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: TERM=unknown
[1.0.10] I, [2017-10-18T20:44:04.717091+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: SHELL=/bin/bash
[1.0.10] I, [2017-10-18T20:44:04.718771+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: SSH_CLIENT=10.229.230.129 55696 22
[1.0.10] I, [2017-10-18T20:44:04.720308+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: EC2_AMITOOL_HOME=/opt/aws/amitools/ec2
[1.0.10] I, [2017-10-18T20:44:04.721929+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: LESS_TERMCAP_ue=ESC[0m
[1.0.10] I, [2017-10-18T20:44:04.723411+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: USER=root
[1.0.10] I, [2017-10-18T20:44:04.724762+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: SUDO_USER=ec2-user
[1.0.10] I, [2017-10-18T20:44:04.726351+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: SUDO_UID=500
[1.0.10] I, [2017-10-18T20:44:04.728406+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: EC2_HOME=/opt/aws/apitools/ec2
[1.0.10] I, [2017-10-18T20:44:04.729926+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: USERNAME=root
[1.0.10] I, [2017-10-18T20:44:04.732020+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: LESS_TERMCAP_us=ESC[04;38;5;111m
[1.0.10] I, [2017-10-18T20:44:04.733588+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: PATH=/sbin:/bin:/usr/sbin:/usr/bin
[1.0.10] I, [2017-10-18T20:44:04.735365+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: MAIL=/var/mail/ec2-user
[1.0.10] I, [2017-10-18T20:44:04.736953+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: PACKER_BUILD_NAME=amazon-ebs
[1.0.10] I, [2017-10-18T20:44:04.744290+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: PWD=/home/ec2-user
[1.0.10] I, [2017-10-18T20:44:04.744944+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: PACKER_BUILDER_TYPE=amazon-ebs
[1.0.10] I, [2017-10-18T20:44:04.746074+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: JAVA_HOME=/usr/lib/jvm/jre
[1.0.10] I, [2017-10-18T20:44:04.748527+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: AWS_CLOUDWATCH_HOME=/opt/aws/apitools/mon
[1.0.10] I, [2017-10-18T20:44:04.752470+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: LANG=en_US.UTF-8
[1.0.10] I, [2017-10-18T20:44:04.755394+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: SUDO_COMMAND=/bin/bash /tmp/script_7079.sh
[1.0.10] I, [2017-10-18T20:44:04.756231+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: HOME=/root
[1.0.10] I, [2017-10-18T20:44:04.757497+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: SHLVL=2
[1.0.10] I, [2017-10-18T20:44:04.757854+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: AWS_PATH=/opt/aws
[1.0.10] I, [2017-10-18T20:44:04.763093+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: AWS_AUTO_SCALING_HOME=/opt/aws/apitools/as
[1.0.10] I, [2017-10-18T20:44:04.763450+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: LOGNAME=root
[1.0.10] I, [2017-10-18T20:44:04.764610+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: SSH_CONNECTION=10.229.230.129 55696 10.229.228.29 22
[1.0.10] I, [2017-10-18T20:44:04.768203+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: AWS_ELB_HOME=/opt/aws/apitools/elb
[1.0.10] I, [2017-10-18T20:44:04.768535+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: DOCKER_CONTENT_TRUST=1
[1.0.10] I, [2017-10-18T20:44:04.768946+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: LESSOPEN=||/usr/bin/lesspipe.sh %s
[1.0.10] I, [2017-10-18T20:44:04.769564+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: SUDO_GID=500
[1.0.10] I, [2017-10-18T20:44:04.769822+0000#23164]  INFO -- Packer: 1508359444,,ui,message,    amazon-ebs: access_key=
LuckyStubs picture LuckyStubs on 18 Oct 2017

Environment variables are only available when you're declaring a variable, not inside a provisioner etc - see https://www.packer.io/docs/templates/user-variables.html#environment-variables.

if you change to:

"environment_vars": [
    "DOCKER_CONTENT_TRUST=1",
    "access_key={{user `aws_access_key`}}",
    "secret_key={{user `aws_secret_key`}}"
  ]

you should be good.

matt-richardson picture matt-richardson on 18 Oct 2017

I have used your suggestion @matt-richardson and I found that I am not having any issue when connecting to the user variables. My issue is actually my env variable.

When I declare my variables it is for some reason not including my environment variable
Here a snippet of the code:

  "variables": {
    "aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}",
    "aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}"
  },

And here is a snippet of my printenv

AWS_PROFILE=secret
AWS_SECRET_ACCESS_KEY=secret
AWS_ACCESS_KEY_ID=secret

When I hardcode my variables, it does work. Only when I bring in via environment variables do I get the issue.

LuckyStubs picture LuckyStubs on 19 Oct 2017

You aren't referencing those defined variables elsewhere in your config -- in your shell provisioner you're trying to access the env variables again instead of using the user variables that you defined at the top of your config from the env variables. Which is why they're blank. Like Matt said above, if you change the variables inside the provisioner to use user variables, you should be okay.

SwampDragons picture SwampDragons on 19 Oct 2017

if it's still failing even though you did what he asked, can we see the whole json again?

SwampDragons picture SwampDragons on 19 Oct 2017

This still does not work

{
  "variables": {
    "aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}",
    "aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}",
    "aws_profile": "{{env `AWS_PROFILE`}}"
  },
  "builders": [{
    "type": "amazon-ebs",
    "profile": "{{ user `aws_profile` }}",
    "region": "us-east-1",
    "run_tags": {
      "UAI":"uai2007116",
      "Name": "uai2007116-GEDockerBeanstalk",
      "env": "lab"
    },
    "run_volume_tags": {
      "UAI":"uai2007116",
      "Name": "uai2007116-GEDockerBeanstalk",
      "env": "lab"
    },
    "snapshot_tags": {
      "UAI":"uai2007116",
      "Name": "uai2007116-GEDockerBeanstalk",
      "env": "lab"
    },
    "tags": {
      "UAI":"uai2007116",
      "Name": "uai2007116-GEDockerBeanstalk",
      "env": "lab"
      },
    "source_ami": "ami-e392eef5",
    "instance_type": "t2.micro",
    "ssh_username": "ec2-user",
    "security_group_id": "sg-065e6d7c",
    "subnet_id": "subnet-a70c3a8f",
    "ami_name": "packer-example {{timestamp}}"
  }],
  "provisioners": [{
  "type": "shell",
  "execute_command": "{{.Vars}} sudo -E -S bash '{{.Path}}'",
  "scripts": [
    "0_Provisioner_AWS_Setup.sh",
    "1_Provisioner_Initialize_AMI.sh",
    "2_Provisioner_Chapter1_Host_Configuration.sh",
    "3_Provisioner_Chapter2_Host_Configuration.sh",
    "4_Provisioner_Benchmark_Setup.sh",
    "5_Provisioner_Splunk.sh",
    "6_Provisioner_Qualys.sh",
    "7_Provisioner_System_Restart.sh",
    "8_Processor_Run_Benchmarks.sh"
  ],
  "environment_vars": [
    "DOCKER_CONTENT_TRUST=1",
    "access_key={{ user `aws_access_key` }}",
    "secret_key={{ user `aws_secret_key` }}"
    ]
  }]
}
LuckyStubs picture LuckyStubs on 19 Oct 2017

I'm sorry, but I'm unable to reproduce this issue. I would make sure that the env var is actually set.

I'm going to close this unless you can send me something that reproduces the issue exactly. Something like a docker image, or a zip of the project and an invocation script.

~/dev/packertest ᐅ BAR=xyz packer build 5480.json
amazon-ebs output will be in this color.

==> amazon-ebs: Prevalidating AMI Name: packer-qs-1508778222
    amazon-ebs: Found Image ID: ami-bcdc16c6
==> amazon-ebs: Creating temporary keypair: packer_59ee20ee-dbc8-3d72-cc2d-c3477648dcc3
==> amazon-ebs: Creating temporary security group for this instance: packer_59ee20f3-9abb-b5ee-e68a-ad4c5a9d581b
==> amazon-ebs: Authorizing access to port 22 from 0.0.0.0/0 in the temporary security group...
==> amazon-ebs: Launching a source AWS instance...
==> amazon-ebs: Adding tags to source instance
    amazon-ebs: Adding tag: "Name": "Packer Builder"
    amazon-ebs: Instance ID: i-05c58d92958546a7b
==> amazon-ebs: Waiting for instance (i-05c58d92958546a7b) to become ready...
==> amazon-ebs: Waiting for SSH to become available...
==> amazon-ebs: Connected to SSH!
==> amazon-ebs: Provisioning with shell script: ./5480.sh
    amazon-ebs: foob=xyz
==> amazon-ebs: Stopping the source instance...
    amazon-ebs: Stopping instance, attempt 1
==> amazon-ebs: Waiting for the instance to stop...
==> amazon-ebs: Creating the AMI: packer-qs-1508778222
    amazon-ebs: AMI: ami-6ac11a10
==> amazon-ebs: Waiting for AMI to become ready...
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Cleaning up any extra volumes...
==> amazon-ebs: No volumes to clean up, skipping
==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary keypair...
Build 'amazon-ebs' finished.

==> Builds finished. The artifacts of successful builds are:
--> amazon-ebs: AMIs were created:
us-east-1: ami-6ac11a10

~/dev/packertest ᐅ cat 5480.json
{
    "builders": [
        {
            "access_key": "{{user `aws_access_key`}}",
            "ami_name": "packer-qs-{{timestamp}}",
            "instance_type": "t2.micro",
            "region": "us-east-1",
            "secret_key": "{{user `aws_secret_key`}}",
            "source_ami_filter": {
                "filters": {
                    "name": "*ubuntu-xenial-16.04-amd64-server-*",
                    "root-device-type": "ebs",
                    "virtualization-type": "hvm"
                },
                "most_recent": true,
                "owners": [
                    "099720109477"
                ]
            },
            "ssh_username": "ubuntu",
            "type": "amazon-ebs"
        }
    ],
    "provisioners": [
        {
            "environment_vars": [
                "DOCKER_CONTENT_TRUST=1",
                "foob={{user `foo`}}"
            ],
            "execute_command": "{{.Vars}} sudo -E -S bash '{{.Path}}'",
            "scripts": [
                "./5480.sh"
            ],
            "type": "shell"
        }
    ],
    "variables": {
        "foo": "{{env `BAR`}}"
    }
}
~/dev/packertest ᐅ cat 5480.sh
#!/bin/bash

echo foob=${foob}
mwhooker picture mwhooker on 23 Oct 2017

@mwhooker I know this is closed, but I do not know what else to do to show that this is not working. I am running everything properly, but when I run the template it is not picking up the environment variables.

Can you clarify for me that when I use 

  "variables": {
    "aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}",
    "aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}",
    "aws_profile": "{{env `AWS_PROFILE`}}",
    "temp": "{{env `SHLVL`}}",
  },

That it is connecting to my local environment variables and pulling the data from my local environment variables as variables to be used within the build.

LuckyStubs picture LuckyStubs on 30 Oct 2017

yes that is correct.

like I said, if you want more help I need to be able to reproduce your issue. Please try emailing the mailing list for more help using packer.

mwhooker picture mwhooker on 30 Oct 2017

@zach5410

To use the the default configs in .aws/credentials and .aws/config use the following:

"variables": {
"aws_access_key": "",
"aws_secret_key": "",
"aws_profile": "",
},

To use the environment variables you should try like so:

"variables": {
"aws_access_key": "{{user AWS_ACCESS_KEY_ID}}",
"aws_secret_key": "{{user AWS_SECRET_ACCESS_KEY}}",
"aws_profile": "{{user AWS_PROFILE}}",
},

lnozot-vt picture lnozot-vt on 14 Mar 2018

For anyone else with the same pain, the environment variables may not be available if "execute_command" is not set correctly. Either don't use it at all, or see the examples here: https://www.packer.io/docs/provisioners/shell.html#execute-command-example

andrewfraley picture andrewfraley on 6 Jun 2018

I'm having a similar issue with the QEMU provisioner without libvirt

This works, but obviously it shows up in ps -ef
sudo IMAGE_PASSWORD=blah ... packer ...

This does not work

export IMAGE_PASSWORD
sudo -E printenv | grep IMAGE_PASSWORD # Shows the password is exported properly
sudo -E ... packer ... # Doesn't pass the password through

packer.json

{
  "variables": {
    "password": "{{env `IMAGE_PASSWORD`}}",
    ...
  },

  "builders":
  [
    {
      "type": "qemu",
    ...
      "ssh_password": "{{user `password`}}",
    ...
    }
  ],
}

I tried editing sudoers with env_keep and setenv but testing with sudo -E seems to indicate that isn't the problem. I'll see if I can allocate some time to make a test case.

LucidOne picture LucidOne on 19 Jul 2018

I had the same exact problem today. When I was validating that the environment variables were existing, they were. An echo $MY_VAR was outputing the actual value of the variable but packer was unable to read the value from the environment and pass it along the user variables scope.

The way I got it fixed was by creating a file named ~/.packer then inside the file I couldn't just type MY_VAR=VALUE and source it normally. I had to force the export of the environment variable within the file.

export MY_VAR=VALUE

Then when I source this file followed by the packer command line, it worked out.

I got this issue in a Ubuntu shell from a Windows Subsystem for Linux (WSL). I guess that the injection of the environment variables doesn't work like a traditional bash on a native ubuntu shell.

drivard picture drivard on 13 May 2019
👍2

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashibot[bot] picture hashibot[bot] on 28 Mar 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

tleyden picture tleyden  ·  3Comments
mushon4 picture mushon4  ·  3Comments
mwhooker picture mwhooker  ·  3Comments
znerd picture znerd  ·  3Comments
brettswift picture brettswift  ·  3Comments