Terraform: Random AuthFailure (401) from *terraform apply*

[shuoyenl]

Hi,

Here's a summary of the issue we encountered.
_terraform apply_ randomly reports 401 error:
Error launching source instance: AuthFailure: AWS was not able to validate the provided access credentials
status code: 401...

According to the trace log, requests of _RunInstances_ and _DescribeInstances_ randomly failed (while the same actions succeeded other times.) When RunInstances succeeded and the following DescribeInstances failed, the instance was successfully created.

Terraform Version

Terraform v0.6.15

Affected Resource(s)

Multiple resources

Terraform Configuration Files

Even a simple config like below fails when the issue occurs:

resource "aws_instance" "test" {    
    instance_type = "t2.small"
    ami = "ami-08111162"  (also tried different ami, e.g. ami-f05c4e9a)
}
provider "aws" {
   access_key = ...
   secret_key = ...
}

Debug Output

https://gist.githubusercontent.com/shuoyenl/303e450806d7a39595e973844733dc3b/raw/f24b4b66b0f2c2133e6bc1bcbe92ecaae43f7467/randomAuthIssue.txt

Panic Output

NA

Expected Behavior

_terraform apply_ should be able to create instances and respond consistently.

Actual Behavior

_terraform apply_ randomly reports 401 error.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:
terraform apply

Important Factoids

NA

References

NA

[stack72]

Hi @shuoyenl

Thanks for the bug report here. Apologies for the error. In order to try and help get to the bottom of it, can you tell me if this worked pre-0.6.15 release? Or is this something you are encountering for the first time using terraform?

Paul

[apparentlymart]

Hmm... very interesting. Based on that output it looks like the following sequence occurs:

• iam:GetUser cred validation for refresh walk: 200 OK
• iam:GetUser cred validation for plan walk: 200 OK
• iam:GetUser cred validation for apply walk: 200 OK
• ec2:RunInstances for creating aws_instance: 200 OK
• ec2:DescribeInstances to check instances status: 200 OK, but still pending
• ec2:DescribeInstances retry: 401 Unauthorized

So my first theory was going to be that the given credentials have only restricted access to certain operations, but that theory is countered by two facts: first, this is a 401 error rather than a 403 Forbidden, and secondly one call to DescribeInstances succeeded before the second one failed.

So I'm not sure where to proceed from here. I have some questions (in addition to @stack72's question) to try to gather some more details to think about:

Does this error occur every time you run terraform apply, or is it intermittent? Have you seen it fail at other points in the process, or is it always at that second call to `ec2:DescribeInstances?

I'm sure you're not running Terraform with verbose logging every time, but a different way to think about my second question there would be whether the error always starts with "Error waiting for instance (i-xxxxxxxx) to become ready:" or whether it sometimes starts with "Error launching source instance:" or some other prefix.

[shuoyenl]

Hi @stack72 and @apparentlymart,
Thanks for the prompt response. Here' are my answers to your questions:

• if this worked pre-0.6.15 release? Yes and No. It used to work on 0.6.14 and then the issue happened. I tried again using 0.6.15 and the issue was gone for a short time before it came back.
• Does this error occur every time you run terraform apply, or is it intermittent? It is "intermittent" while it used to work with the same settings but then stopped working. I have also seen it fail at other points, e.g. RunInstances, the first DescribeInstances, etc.

Besides, I tried to run the same thing from another machine (with the same OS, i.e. Windows 7, Terraform version, network -- plugged into the same port) and did NOT so far see the same issue while the issue still remains on the original machine.

I can probably try to run a smaller test to narrow down the issue if you can direct me to the code in charge of the action.

Thanks very much!

Shuoyen

[phinze]

Hi @shuoyenl - this is definitely weird! I think if you google around for AWS Auth Issues you might find some possible solutions. For instance, when googling "AWS was not able to validate the provided access credentials" I just stumbled on https://forums.aws.amazon.com/thread.jspa?threadID=175266 which has a bunch of suggestions, one of which was intriguing to me:

The solution for me was to sync the clock for my VM/instance

This is true - clock skew can result in sporadic signature failures from the AWS API. Worth checking!

[apparentlymart]

Oh yes... confirming what @phinze said, we (at my employer) had exactly this problem recently with some of our dev environments interacting with S3 when we'd let the clock skew. The AWS signature format includes a timestamp, so it's important to have a reasonably-accurate clock on all API clients.

[shuoyenl]

Yep - that is it. The debug output actually shows the system time is 5 mins faster:
2016/05/05 13:43:19 [DEBUG] terraform-provider-aws.exe: Date: Thu, 05 May 2016 20:38:17 GMT

Thanks very much for checking this !!

[ashuec90]

this is beacause of time zone/time difference betwwen the host machine and aws machine..please sync your timing by running command as below

sudo ntpdate ntp.ubuntu.com

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.