Aws-sam-cli: Executing "sam deploy" fails when using MFA

[matsev]

Description

sam deploy fails when using AWS CLI profiles that have configured to use MFA

Steps to reproduce

1. Setup MFA using IAM and AWS CLI profiles as described in this blog post
2. Export the CLI profile so that it does not need to be provided in the CLI commands:
   $ export AWS_PROFILE=[an AWS CLI profile configured with MFA]
3. $ sam deploy

Observed result

The deploy command prompts for the MFA code twice, then it fails:

Initiating deployment
=====================
Enter MFA code for arn:aws:iam::[account id]:mfa/[IAM user]: 
[entering MFA]
Enter MFA code for arn:aws:iam::[account id]:mfa/[IAM user]: 
[entering MFA again]
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': 'eu-west-1', 'commandName': 'sam deploy', 'duration': 11627, 'exitReason': 'RefreshWithMFAUnsupportedError', 'exitCode': 255, 'requestId': 'request id', 'installationId': 'installation id', 'sessionId': 'session id', 'executionEnvironment': 'CLI', 'pyversion': '3.7.5', 'samcliVersion': '0.38.0'}}]}
HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Traceback (most recent call last):
  File "/usr/local/bin/sam", line 11, in 
    load_entry_point('aws-sam-cli==0.38.0', 'console_scripts', 'sam')()
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/samcli/lib/telemetry/metrics.py", line 93, in wrapped
    raise exception  # pylint: disable=raising-bad-type
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/samcli/lib/telemetry/metrics.py", line 62, in wrapped
    return_value = func(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/samcli/commands/deploy/command.py", line 187, in cli
    ctx.profile,
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/samcli/commands/deploy/command.py", line 305, in do_cli
    deploy_context.run()
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/samcli/commands/deploy/deploy_context.py", line 129, in run
    self.confirm_changeset,
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/samcli/commands/deploy/deploy_context.py", line 156, in deploy
    tags=tags,
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/samcli/lib/deploy/deployer.py", line 408, in create_and_wait_for_changeset
    stack_name, cfn_template, parameter_values, capabilities, role_arn, notification_arns, s3_uploader, tags
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/samcli/lib/deploy/deployer.py", line 171, in create_changeset
    s3_uploader.upload_with_dedup(temporary_file.name, "template"), version_property="Version"
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/samcli/lib/package/s3_uploader.py", line 127, in upload_with_dedup
    return self.upload(file_name, remote_path)
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/samcli/lib/package/s3_uploader.py", line 76, in upload
    if not self.force_upload and self.file_exists(remote_path):
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/samcli/lib/package/s3_uploader.py", line 141, in file_exists
    self.s3.head_object(Bucket=self.bucket_name, Key=remote_path)
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/botocore/client.py", line 272, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/botocore/client.py", line 563, in _make_api_call
    operation_model, request_dict, request_context)
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/botocore/client.py", line 582, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/botocore/signers.py", line 152, in sign
    auth = self.get_auth_instance(**kwargs)
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/botocore/signers.py", line 232, in get_auth_instance
    frozen_credentials = self._credentials.get_frozen_credentials()
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/botocore/credentials.py", line 591, in get_frozen_credentials
    self._refresh()
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/botocore/credentials.py", line 486, in _refresh
    self._protected_refresh(is_mandatory=is_mandatory_refresh)
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/botocore/credentials.py", line 502, in _protected_refresh
    metadata = self._refresh_using()
  File "/usr/local/Cellar/aws-sam-cli/0.38.0/libexec/lib/python3.7/site-packages/botocore/credentials.py", line 248, in __call__
    raise RefreshWithMFAUnsupportedError()
botocore.exceptions.RefreshWithMFAUnsupportedError: Cannot refresh credentials: MFA token required.

Expected result

The sam deploy command should deploy successfully

Comments

The failure mode is inconsistent, I have actually been able to deploy the stack once in about ten or more attempts.

The SAM CLI does not remember the MFA token between command execution. For example, when calling sam package I have to enter a MFA token once, and then if immediately calling sam deploy I have to enter it again (twice actually, as observed above), even if these command are executed in the same bash script.

In contrast when using the same AWS CLI profile configuration together with the aws cli command the aws cloudformation package prompts for MFA once, then calling aws cloudformation deploy I do not have to re-enter the MFA code. Moreover, after the aws cloudformation deploy I can continue to call other aws CLI commands for the duration of the session without entering MFA.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: OS X Mojave, 10.14.6
2. sam --version: 0.38.0

[sriram-mv]

We have a similar issue here: https://github.com/awslabs/aws-sam-cli/issues/1623. It definitely looks like we need to cache sessions better.

[drAlberT]

any ETA or any way to bypass this bug and achieve the deploy in sam cli?

[matsev]

FYI: After reading the release notes of version 0.4.0, I tested to upgrade my SAM CLI installation because it was not clear to me whether or not this issue was resolved. Regrettably the problem is still present.

[sriram-mv]

@matsev This is high priority and is being actively looked at.

[matsev]

TIL: The AWS CLI caches the temp access keys in a file ~/.aws/cli/cache/<some-guid>.json if an AWS CLI profile is configured with mfa_serial.

Ref: https://github.com/aws/aws-cdk/issues/1248#issue-384565974

[awood45]

The fix for this was released with aws-sam-cli 0.41.0 yesterday afternoon, closing.