Packer: builder/amazon: Docs for creadential chain is wrong and missing details

[rickard-von-essen]

The docs states:

Automatic Lookup

If no AWS credentials are found in a packer template, we proceed on to the following steps:

• Lookup via environment variables.
  First AWS_ACCESS_KEY_ID, then AWS_ACCESS_KEY
  First AWS_SECRET_ACCESS_KEY, then AWS_SECRET_KEY

  • Look for local AWS configuration files
    First ~/.aws/credentials
    Next based on AWS_PROFILE

  • Lookup an IAM role for the current EC2 instance (if you're running in EC2)

1. The order of ~/.aws/credentials and AWS_PROFILE is incorrect and AWS_PROFILE uses the shared credentials.

2. profile is missing from the docs.

[telepath]

@rickard-von-essen

The order of ~/.aws/credentials and AWS_PROFILE is incorrect and AWS_PROFILE uses the shared credentials.

I believe this is not an error in the doc, but in the implementation. First, default credentials are looked up, and if not found, AWS_PROFILE is used. This is wrong and works only if no default credentials are defined. Packer ignores AWS_PROFILE if default credentials are defined in ~/.aws/credentials.

[rickard-von-essen]

This is wrong and works only if no default credentials are defined.

Just test this:

$ aws configure
AWS Access Key ID [****************AWS]:
AWS Secret Access Key [****************AWS]:
Default region name [eu-west-1]:
Default output format [None]:

$ env AWS_PROFILE=packer-demo packer build -only amazon-ebs cloud.json

amazon-ebs output will be in this color.

==> amazon-ebs: Prevalidating AMI Name...
    amazon-ebs: Found Image ID: ami-25acd752
==> amazon-ebs: Creating temporary keypair: packer_58a1d671-ceb7-acb0-3c83-0fdb6d4f669e
==> amazon-ebs: Creating temporary security group for this instance...
==> amazon-ebs: Authorizing access to port 22 the temporary security group...
==> amazon-ebs: Launching a source AWS instance...
     amazon-ebs: Instance ID: i-0c723f33531f2c4f8
==> amazon-ebs: Waiting for instance (i-0c723f33531f2c4f8) to become ready...
[...]

It works fine even when default profile is defined.

[vmrob]

Just going to chime in with my setup and success/failures. It just looks to me like Packer's support of the AWS_PROFILE variable is incomplete.

Configuration

My configuration is pretty straightforward:

~/.aws/credentials

[default]
aws_access_key_id = <key id>
aws_secret_access_key = <secret key>

~/.aws/config

[default]
region = us-east-1
[profile dev]
role_arn = arn:aws:iam::<account>:role/Developer
source_profile = default
region = us-east-1

Then in IAM, I have the permission to assume the Developer role on that other account.

Specifying AWS_PROFILE for normal AWS tooling works as expected and per the AWS documentation:

$ aws s3 ls s3://
<default account s3 bucket>
$ AWS_PROFILE=dev aws s3 ls s3://
<dev account s3 buckets>

The build

Packer, on the other hand, doesn't seem to have complete support for the AWS_PROFILE variable:

demo.json

{
    "variables": {
        "subnet_id": "",
        "vpc_id": ""
    },
    "builders": [{
        "type": "amazon-ebs",
        "region": "us-east-1",
        "vpc_id": "{{user `vpc_id`}}",
        "subnet_id": "{{user `subnet_id`}}",
        "source_ami": "ami-40d28157",
        "instance_type": "t2.micro",
        "ssh_username": "ubuntu",
        "ami_name": "ubuntu-demo-{{timestamp}}"
    }],
    "provisioners": [{
        "type": "shell",
        "inline": [
            "echo hello world"
        ]
    }]
}

$ packer build -var vpc_id=<vpc id> -var subnet_id=<subnet id> demo.json
amazon-ebs output will be in this color.

Build 'amazon-ebs' errored: InvalidSubnetID.NotFound: The subnet ID '<subnet id>' does not exist
    status code: 400, request id: <request guid>

==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: InvalidSubnetID.NotFound: The subnet ID '<subnet id>' does not exist
    status code: 400, request id: <request guid>

==> Builds finished but no artifacts were created.

This is expected. That subnet/vpc doesn't exist in the dev account. I would need to use the dev profile to access them.

Using AWS_PROFILE with packer doesn't work:

$ AWS_PROFILE=dev packer build -var vpc_id=<vpc id> -var subnet_id=<subnet id> demo.json
amazon-ebs output will be in this color.

Build 'amazon-ebs' errored: NoCredentialProviders: no valid providers in chain. Deprecated. 
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors

==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: NoCredentialProviders: no valid providers in chain. Deprecated. 
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors

==> Builds finished but no artifacts were created.

Workaround

If I specify the profile in demo.json using the builders.profile key, it works.

demo.json

{
    "variables": {
        "subnet_id": "",
        "vpc_id": "",
        "aws_profile": "{{env `AWS_PROFILE`}}"
    },
    "builders": [{
        "type": "amazon-ebs",
        "region": "us-east-1",
        "profile": "{{ user `aws_profile`}}",
        "vpc_id": "{{user `vpc_id`}}",
        "subnet_id": "{{user `subnet_id`}}",
        "source_ami": "ami-40d28157",
        "instance_type": "t2.micro",
        "ssh_username": "ubuntu",
        "ami_name": "ubuntu-demo-{{timestamp}}"
    }],
    "provisioners": [{
        "type": "shell",
        "inline": [
            "echo hello world"
        ]
    }]
}

$ AWS_PROFILE=dev packer build -var vpc_id=<vpc id> -var subnet_id=<subnet id> demo.json
amazon-ebs output will be in this color.

==> amazon-ebs: Prevalidating AMI Name...
    amazon-ebs: Found Image ID: ami-40d28157
==> amazon-ebs: Creating temporary keypair: <key id>
==> amazon-ebs: Creating temporary security group for this instance...
==> amazon-ebs: Authorizing access to port 22 the temporary security group...
==> amazon-ebs: Launching a source AWS instance...
    amazon-ebs: Instance ID: <instance id>
==> amazon-ebs: Waiting for instance (<instance id>) to become ready...
==> amazon-ebs: Waiting for SSH to become available...
==> amazon-ebs: Connected to SSH!
==> amazon-ebs: Provisioning with shell script: /var/folders/3t/080l6nyn4cncynjmgk3g_jp00000gn/T/packer-shell522845372
    amazon-ebs: hello world
==> amazon-ebs: Stopping the source instance...
==> amazon-ebs: Waiting for the instance to stop...
==> amazon-ebs: Creating the AMI: <ami id>
    amazon-ebs: AMI: <ami id>
==> amazon-ebs: Waiting for AMI to become ready...
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Cleaning up any extra volumes...
==> amazon-ebs: No volumes to clean up, skipping
==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary keypair...
Build 'amazon-ebs' finished.

==> Builds finished. The artifacts of successful builds are:
--> amazon-ebs: AMIs were created:

us-east-1: <ami id>

Comments

It seems to me that there's a problem if I have to specify the profile in that manner. Packer should aim to function like other standard AWS tools and utilize the AWS_PROFILE variable and correctly assume the necessary role. If the current documentation is accurate, then it should be more explicit about not supporting assume-role.

[telepath]

@rickard-von-essen

$ aws configure 
AWS Access Key ID [None]: AWS
AWS Secret Access Key [None]: AWS
Default region name [eu-central-1]: 
Default output format [None]: 
$ AWS_PROFILE=np-networkadmin packer build packer.json 
amazon-ebs output will be in this color.

Build 'amazon-ebs' errored: NoCredentialProviders: no valid providers in chain. Deprecated. 
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors

==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: NoCredentialProviders: no valid providers in chain. Deprecated. 
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors

==> Builds finished but no artifacts were created.

I've tested this with packer 0.12.2.
My env does not contain any other AWS variables (which is the whole point).

[rickard-von-essen]

@vmrob that is a different issue. That duplicates #3019.

[rickard-von-essen]

@telepath Open a new issue with all details to repro and describe clearly what you expect.