Aws-cli: Valid MFA token does not work during first 1am hour before daylight savings ends and second 1am hour starts

[ehammond]

I have a number of aws-cli config profiles that let me assume cross-account roles with MFA token. The configs take this form:

[profile name]
role_arn = arn:aws:iam::123456789012:role/rolename
mfa_serial = arn:aws:iam::987654321098:mfa/myname
source_profile = default
region = us-east-1

These roles have worked fine for weeks, but are now giving me an error when I enter the valid MFA code.

$ aws s3 ls
Enter MFA code: 
Cannot refresh credentials: MFA token required.

Interestingly, it gives me a different error if I enter an invalid MFA code (or the same valid code twice):

$ aws s3 ls
Enter MFA code: 
A client error (AccessDenied) occurred when calling the AssumeRole operation:
MultiFactorAuthentication failed with invalid MFA one time pass code. 

This problem is occurring with two source accounts (different MFA devices) and multiple target accounts, all of which worked hours ago.

I am able to use the same MFA devices to sign in to the same AWS accounts through the AWS console, and to assume the same cross-account roles.

Tonight is the night that daylight savings ends in my timezone, but that doesn't happen for another 5 minutes at which point I will repeat the 1am hour. No idea if that could affect it or not.

aws-cli/v1.9.2

[ehammond]

Yep! It's the second 1am this morning (after the change from daylight savings to standard time) and my aws-cli MFA is suddenly working again.

I don't know how you're going to reproduce this without waiting a year, but there is a bug in aws-cli MFA handling.

[ehammond]

Looks like GitHub has a bug displaying the time zones of the above comments. It is showing both as PST when the first was actually PDT. To be clear:

The first comment was submitted at 1:56am PDT (Pacific Daylight Time). The MFA did not work at that time.

The second comment was submitted 6 minutes later at 1:02 PST (Pacific Standard Time). The MFA did work at that time.

[FlorianHeigl]

I'll keep watching this issue so I know when the hype economy successfully tackled the 90's issues.

[kyleknap]

We will look into this to see what the cause may be.

[teamhanded]

Narrator voice: _they never did_.

[justnance]

@ehammond - Thank you again for reporting this issue while using aws-cli/v1.9.2. I appears work was done under #869 to which PR #871 was merged to resolve. Please upgrade to the latest CLI version advise if it is still an issue. We welcome any new information that can help investigate it further.

[ehammond]

Thanks!

I've set a calendar entry for 1a Nov 3, 2019 to verify that this works ;-)

[justnance]

@ehammond - Thank you for your feedback. I'm glad to hear it is working. Closing issue.

[ehammond]

To be clear, I don't know if it works.

There is only one hour a year where this can be tested live.

[porthunt]

Just 1 month to check if it works. Almost there.

[ehammond]

It works.

[ehammond]

However, the GitHub date relativity calculation thinks I posted that last comment from the future.