Scoutsuite: Issue : Scoutsuite not honoring the aws profile with "role_arn"

[rams3sh]

I am currently using assume role based cross account access across all my aws accounts.

My aws credentials file looks something like this :-

[some_profile]
role_arn=arn:aws:iam::XXXXXXXXXXXX:role/cross_role
credential_source = Ec2InstanceMetadata

The role does not require any external ID.

When run using aws cli with respective profile as argument, it honors the profile configuration and gives me output pertaining to the account which is mapped to respective profile.

However , when scoutsuite is run using --profile argument (same as what was used for aws cli) , it fetches info from the current host account from whose instance scoutsuite is being run and generates the output.

Couldnt figure out what could be the reason of the issue.

Other details that may help is given below :-

$ pip freeze
adal==1.2.2
antlr4-python3-runtime==4.7.2
applicationinsights==0.11.9
argcomplete==1.10.0
asn1crypto==0.24.0
azure-cli-core==2.0.70
azure-cli-nspkg==3.0.4
azure-cli-telemetry==1.0.3
azure-common==1.1.23
azure-mgmt-keyvault==2.0.0
azure-mgmt-monitor==0.7.0
azure-mgmt-network==4.0.0
azure-mgmt-redis==6.0.0
azure-mgmt-resource==2.2.0
azure-mgmt-security==0.3.0
azure-mgmt-sql==0.12.0
azure-mgmt-storage==4.0.0
azure-mgmt-web==0.42.0
azure-nspkg==3.0.2
backports.functools-lru-cache==1.5
bcrypt==3.1.7
boto3==1.9.204
botocore==1.12.204
cachetools==3.1.1
certifi==2019.6.16
cffi==1.12.3
chardet==3.0.4
cheroot==6.5.5
CherryPy==18.1.2
cherrypy-cors==1.6
colorama==0.4.1
coloredlogs==10.0
cryptography==2.7
docutils==0.14
google-api-core==1.14.2
google-api-python-client==1.7.10
google-auth==1.6.3
google-auth-httplib2==0.0.3
google-cloud-container==0.3.0
google-cloud-core==1.0.3
google-cloud-iam==0.2.0
google-cloud-kms==1.2.0
google-cloud-logging==1.12.1
google-cloud-monitoring==0.32.0
google-cloud-resource-manager==0.29.2
google-cloud-storage==1.18.0
google-resumable-media==0.3.2
googleapis-common-protos==1.6.0
grpc-google-iam-v1==0.12.3
grpcio==1.22.0
httpagentparser==1.8.2
httplib2==0.13.1
humanfriendly==4.18
iampoliciesgonewild==1.0.6.2
idna==2.8
isodate==0.6.0
jaraco.functools==2.0
jmespath==0.9.4
knack==0.6.3
more-itertools==7.2.0
msrest==0.6.9
msrestazure==0.6.1
netaddr==0.7.19
oauth2client==4.1.3
oauthlib==3.1.0
paramiko==2.6.0
portalocker==1.5.1
portend==2.5
protobuf==3.9.1
pyasn1==0.4.6
pyasn1-modules==0.2.6
pycparser==2.19
Pygments==2.4.2
PyJWT==1.7.1
PyNaCl==1.3.0
pyOpenSSL==19.0.0
python-dateutil==2.8.0
pytz==2019.2
PyYAML==5.1.2
requests==2.22.0
requests-oauthlib==1.2.0
rsa==4.0
s3transfer==0.2.1
six==1.12.0
sqlitedict==1.6.0
tabulate==0.8.3
tempora==1.14.1
uritemplate==3.0.0
urllib3==1.25.3
zc.lockfile==2.0

$ python scout.py --version
Scout Suite 5.2.0

$ aws --version
aws-cli/1.16.102 Python/2.7.16 Linux/4.14.123-111.109.amzn2.x86_64 botocore/1.12.92

$python --version
Python 3.7.3

However , it works when temporary credentials are updated in the credentials profile for the same assumed role. Seems little weird on why it acts this way.

[alxjsn]

I found that a temporary workaround is to configure role_arn under [default] profile instead of a separate profile.

[rams3sh]

I will check that and get back.

But in case of multiple accounts , I dont think that would be a scalable workaround.

[x4v13r64]

@alxjsn please have a look at https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#assume-role-provider, as we leverage boto3 for authentication.

Root cause looks like The credential_source and source_profile settings are mutually exclusive..

[x4v13r64]

Moving back to 5.4.0, will close if no feedback.

[alxjsn]

@j4v this is my scenario:

~/.aws/credentials

[company]
aws_access_key_id = [REDACTED]
aws_secret_access_key = [REDACTED]
aws_session_token = [REDACTED]

[cool]
source_profile = company
role_arn = arn:aws:iam::[REDACTED]:role/AUDIT

Running scout aws --profile cool gave me the following error:

$ scout aws --profile cool 
2019-08-16 14:34:06 laptop scout[2339] INFO Launching Scout
2019-08-16 14:34:06 laptop scout[2339] INFO Authenticating to cloud provider
Traceback (most recent call last):
  File "/home/user/.local/bin//scout", line 11, in <module>
    load_entry_point('ScoutSuite==5.2.0', 'console_scripts', 'scout')()
  File "/home/user/.local/lib/python3.7/site-packages/ScoutSuite/__main__.py", line 57, in run_from_cli
    programmatic_execution=False)
  File "/home/user/.local/lib/python3.7/site-packages/ScoutSuite/__main__.py", line 96, in run
    result = loop.run_until_complete(_run(**locals()))  # pass through all the parameters
  File "/usr/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "/home/user/.local/lib/python3.7/site-packages/ScoutSuite/__main__.py", line 172, in _run
    credentials=credentials)
  File "/home/user/.local/lib/python3.7/site-packages/ScoutSuite/providers/__init__.py", line 43, in get_provider
    **kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/ScoutSuite/providers/aws/provider.py", line 37, in __init__
    self.account_id = get_aws_account_id(self.credentials)
  File "/home/user/.local/lib/python3.7/site-packages/ScoutSuite/providers/aws/utils.py", line 87, in get_aws_account_id
    caller_identity = get_caller_identity(credentials)
  File "/home/user/.local/lib/python3.7/site-packages/ScoutSuite/providers/aws/utils.py", line 83, in get_caller_identity
    return api_client.get_caller_identity()
  File "/home/user/.local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/client.py", line 648, in _make_api_call
    operation_model, request_dict, request_context)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/client.py", line 667, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/signers.py", line 157, in sign
    auth.add_auth(request)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials

Where if I switched the profile name from cool to default and ran scout without any profile argument it worked just fine.

EDIT: Also, just to clarify, this set up works when using aws-cli, but not scout.

[x4v13r64]

@alxjsn if you look at the boto doc it seems that the [cool] profile would have to be in the ~/.aws/config file (not ~/.aws/credentials). Might be a discrepancy between aws-cli and boto - if so I'll make sure to open an issue on their repo.

[alxjsn]

Hmm that's interesting. When I move the [cool] profile to ~/.aws/config and try using aws-cli I get the following error:

$ aws s3 ls --profile cool                                           

The config profile (cool) could not be found

Also, I get this in scout with the setup described above:

$ scout aws --profile cool
2019-08-19 09:19:17 laptop scout[15869] INFO Launching Scout
2019-08-19 09:19:17 laptop scout[15869] INFO Authenticating to cloud provider
2019-08-19 09:19:17 laptop scout[15869] ERROR __main__.py L153: Authentication failure: The config profile (cool) could not be found

[x4v13r64]

@alxjsn could be a cache issue?

[alxjsn]

Oh so in ~/.aws/config I forgot to put the word profile in front of cool. Here is my file currently:

/.aws/config

[profile cool]
source_profile = company
role_arn = arn:aws:iam::[REDACTED}:role/[REDACTED]

In aws-cli it works with this new setup, but in scout I get the following error:

2019-08-19 09:25:15 laptop scout[16068] INFO Launching Scout
2019-08-19 09:25:15 laptop scout[16068] INFO Authenticating to cloud provider
Traceback (most recent call last):
  File "/home/user/.local/bin//scout", line 11, in <module>
    load_entry_point('ScoutSuite==5.2.0', 'console_scripts', 'scout')()
  File "/home/user/.local/lib/python3.7/site-packages/ScoutSuite/__main__.py", line 57, in run_from_cli
    programmatic_execution=False)
  File "/home/user/.local/lib/python3.7/site-packages/ScoutSuite/__main__.py", line 96, in run
    result = loop.run_until_complete(_run(**locals()))  # pass through all the parameters
  File "/usr/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "/home/user/.local/lib/python3.7/site-packages/ScoutSuite/__main__.py", line 172, in _run
    credentials=credentials)
  File "/home/user/.local/lib/python3.7/site-packages/ScoutSuite/providers/__init__.py", line 43, in get_provider
    **kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/ScoutSuite/providers/aws/provider.py", line 37, in __init__
    self.account_id = get_aws_account_id(self.credentials)
  File "/home/user/.local/lib/python3.7/site-packages/ScoutSuite/providers/aws/utils.py", line 87, in get_aws_account_id
    caller_identity = get_caller_identity(credentials)
  File "/home/user/.local/lib/python3.7/site-packages/ScoutSuite/providers/aws/utils.py", line 83, in get_caller_identity
    return api_client.get_caller_identity()
  File "/home/user/.local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/client.py", line 648, in _make_api_call
    operation_model, request_dict, request_context)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/client.py", line 667, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/signers.py", line 157, in sign
    auth.add_auth(request)
  File "/home/user/.local/lib/python3.7/site-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials

[x4v13r64]

That's odd, I'd have to try this out to confirm (I've used this setup successfully in the past I believe).

Might have something to do with the session token in the company profile...

[x4v13r64]

@rams3sh are you running the latest version of scout? We've just tested this and didn't run in any issues.

[rams3sh]

@j4v I will check in 3 days time and get back. Stuck up with something.

[munntjlx]

My workaround was to re-write the config for profile default (for each separate account) programmatically. I tried almost every combination of profiles and they all failed. The ONLY one that worked was using default with a parameterized config file prior to python run.

[munntjlx]

The 'successful' config looked like this:

[default]
region=us-east-1
output = json
role_arn=arn:aws:iam::999999999:role/MyExternalCrossSecurityReviewRole
source_profile=default
external_id=YourExternalID
duration_session=900

[x4v13r64]

@munntjlx do you mean that the setup described in https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#assume-role-provider does not work for you?

I'm not sure what this issue is about anymore...

[munntjlx]

That would be correct. I tried using profiles (in the config file), environment variables (without and with the config & credentials file), and the only one that worked consistently was using ‘default’ and then changing the role account # to whatever I wanted. Every other way either failed or simply audited the account I was using to assume the x-account role.

[x4v13r64]

@munntjlx are you running the latest version of boto? Are you on windows? This looks to me like a boto3 issue, not Scout.

To confirm this is a boto issue, could you run the following in python:

import boto3
client = boto3.client('sts')
response = client.get_caller_identity()

[munntjlx]

That works fine (with config that looks thus:
[default]
output = json
region = us-east-1
role_arn=arn:aws:iam::1234567890:role/my_security_role_1111/MySecurityRole
source_profile=default
external_id=MyExternalID
duration_session=900

Sincerely,

Thomas J. Munn
Consulting Security Engineer
LexisNexis

On 9/23/19, 10:06, "Xavier Garceau-Aranda" <[email protected][e-mail]> wrote:

* External email: use caution *

@munntjlxhttps://github.com/munntjlx are you running the latest version of boto? Are you on windows? This looks to me like a boto3 issue, not Scout.

To confirm this is a boto issue, could you run the following in python:

import boto3

client = boto3.client('sts')

response = client.get_caller_identity()

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/nccgroup/ScoutSuite/issues/490?email_source=notifications&email_token=ALVIUB5KKI3STAVJWMICYPLQLDEOPA5CNFSM4IK25TWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7K7EZA#issuecomment-534114916, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALVIUB7JSRGQI55JNHK6AP3QLDEOPANCNFSM4IK25TWA.

[munntjlx]

I am running on amazon linux 2 with a venv created from github version (as of 1 week ago). Using these (gist link included) https://gist.github.com/munntjlx/3fbda28c61d45516bd7e0fca04122b81

[x4v13r64]

Closing the issue as it doesn't seem to be caused by Scout.

Refer to https://github.com/nccgroup/ScoutSuite/wiki/Amazon-Web-Services and https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html for setting up authentication for AWS.

[munntjlx]

Done and Done. I would be willing to write more terse documentation, as the provided information was insufficient for me to get this working. My end result was a perl program that re-writes the .config file for each run, which was the ONLY permutation I could get working. Hopefully this will help other people having issues with cross authentication & scoutsuite. Otherwise it’s a great product, just docs need some improving, which I would be willing to do, provided I could find someone to help me.

Sincerely,

Thomas J. Munn
Consulting Security Engineer
LexisNexis

On 9/23/19, 10:55, "Xavier Garceau-Aranda" <[email protected][e-mail]> wrote:

* External email: use caution *

Closing the issue as it doesn't seem to be caused by Scout.

Refer to https://github.com/nccgroup/ScoutSuite/wiki/Amazon-Web-Serviceshttps://github.com/nccgroup/ScoutSuite/wiki/Amazon-Web-Services and https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.htmlhttps://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html for setting up authentication for AWS.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/nccgroup/ScoutSuite/issues/490?email_source=notifications&email_token=ALVIUB5XFUTM7CAILLVA42DQLDKDZA5CNFSM4IK25TWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7LEJHY#issuecomment-534135967, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALVIUB2CT6W6WX3HPYHYA2LQLDKDZANCNFSM4IK25TWA.

[rams3sh]

@j4v Sorry for delay. Just using this to document my case on the same for future reference.
This bug is fixed for me. I think the mistake on my part was to use credentials file to store my role_arn and source credentials.

I use scoutsuite out of an amazon ec2 box and the following worked for me. Gathered all the roles of cross accounts and created a config file of the following format

[profile default]
credential_source = Ec2InstanceMetadata
[profile  ABC ]
role_arn =  arn:aws:iam::XXXXXXXXXXXX:role/Role
credential_source = Ec2InstanceMetadata
[profile  DEF ]
role_arn =  arn:aws:iam::YYYYYYYYYYYY:role/Role

.. 

I completely deleted my credentials file as my use case didnt require it , also to ensure that there were no conflicts with the roles configured in config file.

Also , I use Scoutsuite version 5.3.3.

Though I understand that this is totally related to aws cli configuration and not scoutsuite , it would be nice to have it documented as part of the documentation to avoid confusion in future.
My case is closed.

[x4v13r64]

Though I understand that this is totally related to aws cli configuration and not scoutsuite , it would be nice to have it documented as part of the documentation to avoid confusion in future.

@rams3sh apologies for being thick, but what exactly was not in the boto3 documentation that would have helped you setup authentication easier?

[x4v13r64]

I would be willing to write more terse documentation, as the provided information was insufficient for me to get this working

I'm just not sure what isn't in the boto3 documentation that would have helped you? I've never had any problems setting auth by just following their instructions.

If you could details exactly what wasn't in https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html that would have helped you, that would be great.

[rams3sh]

@j4v everything is documented as part of boto3 documentation . It was my mistake in not following it!!

My suggestion was to have that link be pasted as part of README of Scoutsuite so that this issue doesnt get created again.

There is no issue in scoutsuite for my case (I speak for my case alone). The ask / suggestion is to just highlight the above provided document in the README .. Thats it !!

[x4v13r64]

It's mentioned here https://github.com/nccgroup/ScoutSuite/wiki/Amazon-Web-Services#introduction.