Aws-sam-cli: SAM 0.17+ prompts for MFA credentials on every request when running API locally

[johnc44]

Description

We use Multi-Factor Authentication to access our AWS accounts.

In previous versions (0.19.0), SAM CLI would prompt for my MFA token when I first invoked the lambda, and would then retain those credentials for future requests. Eventually they would expire and I'd have to quit/restart to get it to work again, but it was quite workable.

In 0.20.1, however, I am asked for my MFA credentials on literally every request. This makes it completely unusable for me, so I've had to downgrade again.

Steps to reproduce

Using Ubuntu WSL:

export AWS_PROFILE=
aws local start-api -n environment.json -p 3100

Then hit the API. It sits there twiddling its thumbs. Switch back to Ubuntu and it's asking for the MFA code. Enter it in, everything works.

Hit the API a second time. Again, it sits there thinking about it. Switch back to Ubuntu and it's asking for the MFA again. This is not the case in 0.19.0 which would remember my token.

Observed result

Asked for MFA code on every single request.

Expected result

MFA token should be cached between requests.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Windows 10, patched up to date. Using Ubuntu WSL.
2. sam --version: 0.20.1

[johnc44]

Hmm.. OK on closer inspection, dropping down to 0.19.0 did not fix the issue. Neither has going down to 0.18.0.

It used to work as expected. Last week I upgraded to sam cli 0.20.1 and presumed it was the culprit. Maybe it's something else? (I can't guarantee what version I was on before, I think it was 0.18.0)

Is there anything I can do to work around this? Having to type the MFA code in every time is driving me crazy.

[johnc44]

OK, I went back to sam 0.16 and the problem has now cleared up. So either in 0.17 or 0.18 it seems to have changed behaviour.

As a side note - I've also noticed that if I Ctrl+C to quit the process whilst it's asking me for an MFA then it renders the whole terminal unusable. (Well, by that I mean I can't see what I'm typing - but I can actually type blindly in and it works).

But, anyway... any idea what's changed in SAM that would affect this behaviour? Any workaround or do I just need to stick to version 0.16?

[sriram-mv]

Interesting is this just a windows 10 issue, do you see it on any other platform?

[johnc44]

I've only tried it on Ubuntu WSL on Win10. I'll try messing around with it on Win10 itself, and maybe on my Mac. Just need to find a few spare minutes from somewhere first. I also need to try 0.17, but frankly yesterday I was so relieved to put a stop to it I didn't want to mess with it any more!

[johnc44]

Tried in normal Win10 command prompt and same behaviour on 0.21.0. (IE being asked over and over again)l

[johnc44]

Sorry, only just had a few moments spare.

I've tried this at home using OSX Mojave.

Sam 0.16 - I only get prompted for MFA credentials once.
Sam 0.17 - I get prompted every single request.

So looks like it's not OS specific, and it was introduced in version 0.17.

[johnc44]

Any thoughts? It means we can't upgrade beyond 0.16 unless we are going to spend the entire day entering in mfa credentials.

[johnc44]

Saw 0.22 has been released, so tried it there as well but no difference.

Rolling back to 0.16 again. I really hope this can be resolved.

[jfuss]

@johnc44 Thanks for the all the information you have provided and keeping this up-to-date. I am looking at the changes we did from 0.16 to 0.17, one PR stood out to me: #1059. I have not verified yet, but my hunch is this line: https://github.com/awslabs/aws-sam-cli/pull/1059/files#diff-71d3cd43730ae0d5ad972d2db90e44c2R213

Looking at the history, we originally had it the way it was today: e607d66fbaa376f8d2f42a5c830be8aa4fb873eb but was changed to match what the comment suggested: f88fb592e072314a4902e1f00489329a58c79519. Later we noticed some race conditions happening so we reverted in #1059 back to what is was originally (in commit e607d66fbaa376f8d2f42a5c830be8aa4fb873eb).

I need do get some things setup to verify this but this is my best guess, as it would match the need to get creds on every invoke.

[jfuss]

@johnc44 I am struggling getting my MFA setup and working. Can you try two things, when you get some time:

1. Install v0.7.0 and run sam local start-api. I would expect the same behavior of always needing to provide MFA code.
2. Install v0.8.1 and run sam local start-api. I would expect your desired behavior to be in this build (only single MFA code until it expires).

I am still trying to get this MFA setup, but would at least give me the confidence my hunch is correct.

[sosdmike]

@jfuss I am having the same problem as @johnc44 so I thought I would give you a hand with this.

I created a new project and installed v0.7.0 with pip pip install aws-sam-cli==0.7.0, every invoke requires 2fa.
In the same project I removed v0.7.0 pip uninstall aws-sam-cli and installed 0.8.1 with pip install aws-sam-cli==0.7.0, the behaviour is as you described (only single mfa code until it expires).

[jfuss]

@sosdmike Thank you! After talking with some team members, I think the always creating the session is the culprit here. I think what we want to do is cache this somehow.

[jfuss]

After some investigation here are some deeper details:

Boto3 sessions are not thread safe: https://github.com/awslabs/aws-sam-cli/blob/develop/samcli/commands/local/lib/local_lambda.py#L220 this is due to boto3 sessions dependency on urllib which is (also) not thread safe. This is why we get credentials every time for invoke. For most of the use-cases, this isn't a problem but for the occasions of multiple requests on sam local start-api or sam local start-lambda, we need to make sure we we can get credentials in a thread safe fashion. This brings me to 3 options:

1. We cache the whole session object
2. We get frozen credentials from the session object.
3. ~Keep as is, which would mean recommending not to use MFA for local testing.~

Option 3 isn't a really valid option, as this really hinders customers using MFA with start-api and start-lambda.

Option 2 seems like the best out of the remaining options. To be done right (I think), we would need a way to invalidate the creds and get new ones for long sessions. Otherwise, we will be mounting expired credentials into the container and the only way to refresh would to be rerun the command (not the end of the world but not the greatest experience). I would have to dig deeper to understand if there is timestamp information we could use for this. It's worth calling out that this would only affect debugging/invoking sessions that are longer than the life of the credentials.

There are trade-offs on both Option 1 and Option 2. To make sure we feel we are making the right decision, I will bring this up with the team and circle back here.

[johnc44]

Pre 0.17, the credentials would expire after a bit, and I'd have to run sam local again. It's very much less annoying than being prompted every time, so I'd personally be quite happy if it reverted to that behaviour.

[jfuss]

Patched was released in 0.38.0.

Closing

[gilkong]

@jfuss this seems to be happening again in 0.40.0. Running sam validate twice in a row asks for MFA both times.

[jfuss]

@gilkong We do not cache across command runs, so this is currently expected behavior. We are looking at some caching options to make this possible with #1682, though.

[silvait]

@jfuss Was the fix for #1682 specific to the deploy command?
I'm still getting prompted for a MFA code on every run of local start-api and local invoke

[jfuss]

@silvait What version of SAM CLI are you running?

In the local suite of commands, we cache for the life time of the command or when the creds expire with MFA. I can't remember if this prompts after the MFA expires or not.

[silvait]

@jfuss I'm using 0.46.2.
sam prompts me for a MFA code every time I run sam local invoke.
For sam local start-api it prompts me once per run, but not on every request like it used to, which is a big improvement.
Are there plans of caching credentials across multiple local executions (similar to AWS CLI)?

[jcop]

While a general solution isn't created, I would suggest using aws-vault

A very quick usage guide (works for me):

Place a entry in your aws config ~/.aws/config:

[profile my-profile-vault]
role_arn = arn:aws:iam::<Role you want to use>
mfa_serial = arn:aws:iam::<your aws mfa arn>
region = <region>
output = json

Then on your command line:

aws-vault exec my-profile-vault

This will launch a new bash session and request your MFA (my case, keeps it in the identity manager), and will keep using it for as long as it is active.