Packer: Packer v1.6.3 - Error loading credentials for AWS Provider

[Manojvnair1]

Packer v1.6.3 giving error while fetching loading credentials from IAM instance role.

Please find log extract:
packer-builder-amazon-ebs plugin: [INFO] Attempting to use session-derived credentials
Build 'amazon-ebs' errored after 3 seconds 133 milliseconds: Error loading credentials for AWS Provider: %!w()
==> Wait completed after 3 seconds 133 milliseconds
==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: Error loading credentials for AWS Provider: %!w()

[nywilken]

Hi there thanks for reaching out. It looks like the AWS builder is running into issues displaying the error being returned when loading credentials.

Can you please share your build template (stripped of any sensitive information) along with a copy the debug logs PACKER_LOG=1 packer build <template>.

Cheers!

[chillosu]

This was working on the previous version and just started breaking with the new version.

We inherit the IAM role from the underlying ec2 instance it's running on to AMI building.

[bretmartin]

This is definitely a regression in 1.6.3 over 1.6.2 (I downgraded to confirm). I suspect this is caused by #9981 but I ran out of time last night (well, early this morning) trying to understand how and confirm that.

The two others above are using EC2 instance profile credentials. I'm using a role assumed via an access key (user → role).

Build template and debug log below. fake-org-name-infrastructure is a named profile in ~/.aws/config and ~/.aws/credentials for a role assumed via a user with access key, also named in those files.

redacted build template

source "amazon-ebs" "amazon" {
  profile = "fake-org-name-infrastructure"
  region = "us-east-1"

  ami_name = "fake-org-name-amazon-linux-2-x86_64-{{ isotime \"20060102\" }}"
  ami_description = "Built from AMI Name \"{{ .SourceAMIName }}\" (AMI ID {{ .SourceAMI }})"
  force_deregister = true
  force_delete_snapshot = true

  iam_instance_profile = "ec2_default"
  instance_type = "t3a.small"
  ssh_agent_auth = true
  ssh_keypair_name = "fake-org-name-infrastructure-us-east-1"
  ssh_username = "ec2-user"

  run_tags = {
    "Function" = "packer-builder",
  }

  source_ami_filter {
    filters = {name = "amzn2-ami-hvm-*-x86_64-*"}
    owners = ["amazon"]
    most_recent = true
  }

  subnet_filter {
    filters = {"tag:Name" = "main-public-c"}
  }
}

build {
  sources = ["source.amazon-ebs.amazon"]

  provisioner "shell" {
    inline = [
      [ REDACTED ]
    ]
  }
}

debug log (from PACKER_LOG=1 packer build foo.pkr.hcl)

2020/09/27 20:50:39 [INFO] Packer version: 1.6.3 [go1.15.2 darwin amd64]
2020/09/27 20:50:39 Checking 'PACKER_CONFIG' for a config file path
2020/09/27 20:50:39 'PACKER_CONFIG' not set; checking the default config file path
2020/09/27 20:50:39 Attempting to open config file: /Users/bam/.packerconfig
2020/09/27 20:50:39 [WARN] Config file doesn't exist: /Users/bam/.packerconfig
2020/09/27 20:50:39 Setting cache directory: /Users/bam/Documents/fake-org-name.org/Development/fake-org-name-packer/packer_cache
2020/09/27 20:50:39 Creating plugin client for path: /usr/local/bin/packer
2020/09/27 20:50:39 Starting plugin: /usr/local/bin/packer []string{"/usr/local/bin/packer", "plugin", "packer-builder-amazon-ebs"}
2020/09/27 20:50:39 Waiting for RPC address for: /usr/local/bin/packer
2020/09/27 20:50:39 packer-builder-amazon-ebs plugin: [INFO] Packer version: 1.6.3 [go1.15.2 darwin amd64]
2020/09/27 20:50:39 packer-builder-amazon-ebs plugin: Checking 'PACKER_CONFIG' for a config file path
2020/09/27 20:50:39 packer-builder-amazon-ebs plugin: 'PACKER_CONFIG' not set; checking the default config file path
2020/09/27 20:50:39 packer-builder-amazon-ebs plugin: Attempting to open config file: /Users/bam/.packerconfig
2020/09/27 20:50:39 packer-builder-amazon-ebs plugin: [WARN] Config file doesn't exist: /Users/bam/.packerconfig
2020/09/27 20:50:39 packer-builder-amazon-ebs plugin: Setting cache directory: /Users/bam/Documents/fake-org-name.org/Development/fake-org-name-packer/packer_cache
2020/09/27 20:50:39 packer-builder-amazon-ebs plugin: args: []string{"packer-builder-amazon-ebs"}
2020/09/27 20:50:39 packer-builder-amazon-ebs plugin: Plugin address: unix /var/folders/ct/qvxb2p2d7rx63h4nq95twfq00000gn/T/packer-plugin139721840
2020/09/27 20:50:39 packer-builder-amazon-ebs plugin: Waiting for connection...
2020/09/27 20:50:39 Received unix RPC address for /usr/local/bin/packer: addr is /var/folders/ct/qvxb2p2d7rx63h4nq95twfq00000gn/T/packer-plugin139721840
2020/09/27 20:50:39 packer-builder-amazon-ebs plugin: Serving a plugin connection...
2020/09/27 20:50:39 packer-builder-amazon-ebs plugin: [INFO] (aws): No AWS timeout and polling overrides have been set. Packer will default to waiter-specific delays and timeouts. If you would like to customize the length of time between retries and max number of retries you may do so by setting the environment variables AWS_POLL_DELAY_SECONDS and AWS_MAX_ATTEMPTS or the configuration options aws_polling_delay_seconds and aws_polling_max_attempts to your desired values.
2020/09/27 20:50:39 Creating plugin client for path: /usr/local/bin/packer
2020/09/27 20:50:39 Starting plugin: /usr/local/bin/packer []string{"/usr/local/bin/packer", "plugin", "packer-provisioner-shell"}
2020/09/27 20:50:39 Waiting for RPC address for: /usr/local/bin/packer
2020/09/27 20:50:39 packer-provisioner-shell plugin: [INFO] Packer version: 1.6.3 [go1.15.2 darwin amd64]
2020/09/27 20:50:39 packer-provisioner-shell plugin: Checking 'PACKER_CONFIG' for a config file path
2020/09/27 20:50:39 packer-provisioner-shell plugin: 'PACKER_CONFIG' not set; checking the default config file path
2020/09/27 20:50:39 packer-provisioner-shell plugin: Attempting to open config file: /Users/bam/.packerconfig
2020/09/27 20:50:39 packer-provisioner-shell plugin: [WARN] Config file doesn't exist: /Users/bam/.packerconfig
2020/09/27 20:50:39 packer-provisioner-shell plugin: Setting cache directory: /Users/bam/Documents/fake-org-name.org/Development/fake-org-name-packer/packer_cache
2020/09/27 20:50:39 packer-provisioner-shell plugin: args: []string{"packer-provisioner-shell"}
2020/09/27 20:50:39 packer-provisioner-shell plugin: Plugin address: unix /var/folders/ct/qvxb2p2d7rx63h4nq95twfq00000gn/T/packer-plugin219357180
2020/09/27 20:50:39 packer-provisioner-shell plugin: Waiting for connection...
2020/09/27 20:50:39 Received unix RPC address for /usr/local/bin/packer: addr is /var/folders/ct/qvxb2p2d7rx63h4nq95twfq00000gn/T/packer-plugin219357180
2020/09/27 20:50:39 packer-provisioner-shell plugin: Serving a plugin connection...
2020/09/27 20:50:39 Build debug mode: false
2020/09/27 20:50:39 Force build: false
2020/09/27 20:50:39 On error:
2020/09/27 20:50:39 Waiting on builds to complete...
2020/09/27 20:50:39 Starting build run: amazon-ebs.amazon
2020/09/27 20:50:39 Running builder:
2020/09/27 20:50:39 [INFO] (telemetry) Starting builder
amazon-ebs.amazon: output will be in this color.

2020/09/27 20:50:40 packer-builder-amazon-ebs plugin: [INFO] Attempting to use session-derived credentials
2020/09/27 20:50:40 packer-builder-amazon-ebs plugin: [INFO] Successfully derived credentials from session
2020/09/27 20:50:40 packer-builder-amazon-ebs plugin: [INFO] AWS Auth provider used: "AssumeRoleProvider"
2020/09/27 20:50:40 [INFO] (telemetry) ending
==> Wait completed after 955 milliseconds 427 microseconds
2020/09/27 20:50:40 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2020/09/27 20:50:40 machine readable: amazon-ebs.amazon,error []string{"Error loading credentials for AWS Provider: %!w(<nil>)"}
==> Builds finished but no artifacts were created.
2020/09/27 20:50:40 [INFO] (telemetry) Finalizing.
Build 'amazon-ebs.amazon' errored after 955 milliseconds 321 microseconds: Error loading credentials for AWS Provider: %!w(<nil>)

==> Wait completed after 955 milliseconds 427 microseconds

==> Some builds didn't complete successfully and had errors:
--> amazon-ebs.amazon: Error loading credentials for AWS Provider: %!w(<nil>)

==> Builds finished but no artifacts were created.
2020/09/27 20:50:41 waiting for all plugin processes to complete...
2020/09/27 20:50:41 /usr/local/bin/packer: plugin process exited
2020/09/27 20:50:41 /usr/local/bin/packer: plugin process exited

[William-Luo0]

Not sure if this is helpful or not since I believe the AWS Go SDK is meant to handle auth; but this was not working for me in Codebuild which utilises ECS under the hood.
The special url for ECS differs from EC2 in that it uses 169.254.170.2 as the endpoint rather than the typical 169.254.169.254.

The full credentials can be retrieved from the url 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI while in an ECS container.

[nywilken]

Hello again, thanks for the additional information - super helpful. I thought it might've been a regression but realize now that I failed to ask, apologies. There looks to be two issues: one being the auth regression, the second being the error logging. We'll work on getting this fixed for the next release if not sooner.

[nywilken]

Hi @William-Luo0 thanks for the added context. When you say this was not working for me in Codebuild.

Are you taking about Packer or the use of session credentials within Codebuild? If Packer related was it working in v1.6.2?

I'm having a little difficulty making the connection so want to make sure I get the full picture. Thanks!

[bretmartin]

@nywilken For the error logging, I think the problem is that err is overwritten in this section of code. Working on this...

[nywilken]

@nywilken For the error logging, I think the problem is that err is overwritten in this section of code. Working on this...

@bretmartin that's correct. The error is actually nil there as GetCredentialsFromSession() returns back successfully. The code should not actually error there. Which is most likely the regression as well - it is returning an error when there is no error. I have a WIP branch created if you want to give it a look over/test. Happy to work with you on pushing forward a fix if you're interested in working on it.

[bretmartin]

@nywilken This is literally my first time touching Go code so you're gonna beat me to this for sure! I'm building your WIP right now to give it a shot, though.

[bretmartin]

Presuming #10007 is a complete fix, is there a test coverage issue here? If so, is it HC practice to handle that with the bug fix or separately?

[nywilken]

Presuming #10007 is a complete fix, is there a test coverage issue here? If so, is it HC practice to handle that with the bug fix or separately?

Great question, tests should accompany the fix when possible. The exception being the reported bug involves an intricate infrastructure setup that we can not mimic. In those cases we try to add as much test coverage as possible and ask the community for help in testing against some of the more advance setups. The practice behind testing is best captured by Terraform here, which we at Packer follow.

The PR is currently listed as a WIP to see what is missing and to identify if there are any other issues not accounted for.

With that said, I'm going to drop a link to the test binaries here in case others who may be running into this issue can test against.

[William-Luo0]

Hi @William-Luo0 thanks for the added context. When you say this was not working for me in Codebuild.

Are you taking about Packer or the use of session credentials within Codebuild? If Packer related was it working in v1.6.2?

I'm having a little difficulty making the connection so want to make sure I get the full picture. Thanks!

Sorry, just had a chance to do another review of my repo, there were other changes causing the failure. Ignore me 😅

[genvidkyle]

Just ran into this as well

[SwampDragons]

Sorry about this, all -- This is my fault. I'll make sure that we have better testing around this authentication code using the Amazon mocks before the next release so this won't happen again.

This is definitely gonna be a wider-spread regression than most, so @nywilken I think we should leave the issue open even when we merge the fix, until Packer 1.6.4 is released, so people can find it easily.

We should also push a "nightly" build as soon as we merge the fix in 10007 so that users have a workaround right away instead of waiting for next Tuesday's "nightly" release.

[Porkepix]

Sorry about this, all -- This is my fault. I'll make sure that we have better testing around this authentication code using the Amazon mocks before the next release so this won't happen again.

This is definitely gonna be a wider-spread regression than most, so @nywilken I think we should leave the issue open even when we merge the fix, until Packer 1.6.4 is released, so people can find it easily.

We should also push a "nightly" build as soon as we merge the fix in 10007 so that users have a workaround right away instead of waiting for next Tuesday's "nightly" release.

Maybe when regression result in complete breakage is it possible to have a dot release with just this fix? I guess some use cases can become quite painful when it comes to downgrade versions.

[SwampDragons]

Yes, I'm weighing right now whether this regression merits a fast-follow release. Either way we need to get the patch reviewed and tested before the release can go out so it'll be a day or two.

[SwampDragons]

We've merged a patch for this and will do a fast-follow release tomorrow to fix this. Sorry again, everyone.

[SwampDragons]

1.6.4 has been released.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.