Scoutsuite: Receiving bucket list gives AccessError even though it works with aws_cli

[hashier]

I'm using,

macOS 10.14.5
Python 3.7.3

with ScoutSuite installed via pip3:
$ pip3 freeze | grep -i scout
ScoutSuite==5.0.post3

and I tried as well the latest git checkout:
$ git rev-parse HEAD
5a498fbd9fc558d63409123ab5aba19ad4e5097d

What did you do?

When I call:

$ aws s3 ls --profile security
Enter MFA code for arn:aws:iam::xxxx:mfa/username:
2018-04-12 16:20:13 foo
2018-04-20 14:43:42 bar
[...]

I can see all s3 buckets, but when I try fetch the buckets via Scout I end up with the following error:

$ python3 scout.py aws --profile security --services s3 --no-browser
2019-05-21 16:42:21 PinkRobin.localdomain scout[33071] INFO Launching Scout
2019-05-21 16:42:21 PinkRobin.localdomain scout[33071] INFO Authenticating to cloud provider
Enter MFA code for arn:aws:iam::xxxxx:mfa/username:
2019-05-21 16:42:29 PinkRobin.localdomain scout[33071] INFO Gathering data from APIs
2019-05-21 16:42:29 PinkRobin.localdomain scout[33071] INFO Fetching resources for the S3 service
2019-05-21 16:42:30 PinkRobin.localdomain scout[33071] ERROR s3.py L15: Failed to list buckets: An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied
2019-05-21 16:42:30 PinkRobin.localdomain scout[33071] INFO Running rule engine
2019-05-21 16:42:30 PinkRobin.localdomain scout[33071] INFO Applying display filters
2019-05-21 16:42:30 PinkRobin.localdomain scout[33071] INFO Saving data to scoutsuite-report/scoutsuite-results/scoutsuite_results_aws-security.js
2019-05-21 16:42:30 PinkRobin.localdomain scout[33071] INFO Saving data to scoutsuite-report/scoutsuite-results/scoutsuite_exceptions_aws-security.js
2019-05-21 16:42:30 PinkRobin.localdomain scout[33071] INFO Saving data to scoutsuite-report/scoutsuite-results/scoutsuite_errors_aws-security.json
2019-05-21 16:42:30 PinkRobin.localdomain scout[33071] INFO Creating scoutsuite-report/aws-security.html

What did you expect to see?

A generated report which includes the buckets.

What did you see instead?

AccessDenied error.

Some things I tried:

I followed from #28 to https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html and created my config and credentials how it is mentioned on that webpage without any success.

I also tried to add the role into my [default] configuration, that didn't work either.

Thoughts

I think it is not using the profile/role correctly. When I run scout fully I get error messages that look like this:

2019-05-21 16:57:11 PinkRobin.localdomain scout[33314] ERROR cloudtrail.py L13: Failed to describe CloutTrail trail: An error occurred (AccessDeniedException) when calling the DescribeTrails operation: User: arn:aws:iam::ACCOUNTID:user/ACCOUNT is not authorized to perform: cloudtrail:DescribeTrails

Those show that it is using the account instead of my role.

[x4v13r64]

Those show that it is using the account instead of my role.

Is the profile you are using for a user account or temporary credentials for an assumed role?

The easiest way is to create a profile with the temporary credentials, see https://github.com/nccgroup/ScoutSuite/wiki/Amazon-Web-Services#option-aws-credentials-file-with-mfa.

[hashier]

Oh, I used my account key. I thought it would then assume the role since the profile mentions the role. Pretty new to all this. Will try what I read on your linked page and sts.
That describes it pretty well. Sorry that I didn’t find that manual myself.

Will report back then

[x4v13r64]

When you call sts-assume-role, the service returns a set of credentials (which has the key and key secret + a token). You have to create a new profile with these 3 values and use the name of that profile when running Scout.

[hashier]

I will try that. Right now my setup is how it is described under https://github.com/nccgroup/ScoutSuite/wiki/Amazon-Web-Services#option-aws-credentials-file-with-roles that I just added the role in the profile. Will try with sts, sounds promising. Thanks. Will try as soon as I’m in front of my computer.

[hashier]

tl;dr Use https://github.com/nccgroup/ScoutSuite/issues/425#issuecomment-494699198 and it works

So I have managed to get it working, but I feel like my first method should work as well, since it is mentioned in the wiki.

This works:

$ aws --profile default sts assume-role --role-arn arn:aws:iam::11111:role/SecurityAuditor --role-session-name "RoleSession1" --serial-number arn:aws:iam::22222:mfa/userAccount --token-code 123456

And then export those variables I get as AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN.

I can then verify my role with:

~ $ aws sts get-caller-identity
{
    "UserId": "xxxxx:RoleSession1",
    "Account": "1111",
    "Arn": "arn:aws:sts::1111:assumed-role/SecurityAuditor/RoleSession1"
}

and then just run python3 scout.py aws --regions eu-west-1 --max-workers 1 ... and it works.

I feel though, that it should as well work with my created security profile in ~/.aws/config because it yields the following:

$ aws sts get-caller-identity --profile security
{
    "UserId": "xxxxx:botocore-session-2558709005",
    "Account": "1111",
    "Arn": "arn:aws:sts:: 1111:assumed-role/SecurityAuditor/botocore-session-2558709005"
}

which shows as well the correct role. But when I try running scout --profile security I run into the problem, that it still wants to use my user account, even though sts get-caller-identity clearly shows the correct role...

Anyway, I'm glad I got it running and just write this down if anyone else ends up in the same problem.

[x4v13r64]

Thanks for the above, I'll investigate.

[saurabh0mishra]

I was facing the same issue while scanning the s3 bucket.
I think this bug was because of the installed botocore version of my machine (in my case it was botocore-1.12.146).
I downgraded botocore version to botocore-1.12.130 and the issue got resolved.

[x4v13r64]

Thanks @saurabh0mishra that seems like a duplicate of https://github.com/nccgroup/ScoutSuite/issues/381? @hashier your issue seems different though?

[hashier]

Yes, that looks like a different kind of problem. S3 scanning was just one example of problem that I picked. The main problem for me was that it used the wrong account/profile. Not the scanning itself.

[x4v13r64]

@hashier sorry for the delay, this should now be resolved in branch https://github.com/nccgroup/ScoutSuite/tree/bugfix/aws-session-handling - could you please test?

[hashier]

Just a quick test, doesn't seem to work \: But I have only given it 2 minutes to test.

I still get the same error, that is uses my personal user instead of my IAM role.

[hashier]

I ran on commit 0254a4946a288a4da7a376ab2848e9c11fcc4914

[x4v13r64]

Thanks @hashier, I think it's due to the MFA. I've tested with source_profile switching and it should now work.

Will test with MFA.

[x4v13r64]

@hashier are you sure this isn't working for you? Worked for me in the following configuration:
".aws/credentials":

[source]
aws_access_key_id = AKIA[...]
aws_secret_access_key = [...]
output = json
region = eu-west-1

Where source profile is in account 111111111111

".aws/config":

[switch]
role_arn=arn:aws:iam::222222222222:role/test-switch
source_profile=source
mfa_serial=arn:aws:iam::111111111111:mfa/source

Run Scout:

(venv) ~/Git/ScoutSuite (bugfix/aws-session-handling ✔) ᐅ python scout.py aws --force --no-browser --services iam --profile switch

2019-07-04 12:58:27 wrkbx scout[7443] INFO Launching Scout
2019-07-04 12:58:27 wrkbx scout[7443] INFO Authenticating to cloud provider
Enter MFA code for arn:aws:iam::111111111111:mfa/source: 
2019-07-04 12:58:38 wrkbx scout[7443] INFO Gathering data from APIs
2019-07-04 12:58:38 wrkbx scout[7443] INFO Fetching resources for the IAM service
2019-07-04 12:58:39 wrkbx scout[7443] INFO Running rule engine
2019-07-04 12:58:40 wrkbx scout[7443] INFO Applying display filters
2019-07-04 12:58:41 wrkbx scout[7443] INFO Saving data to scoutsuite-report/scoutsuite-results/scoutsuite_results_aws-switch.js
2019-07-04 12:58:41 wrkbx scout[7443] INFO Saving data to scoutsuite-report/scoutsuite-results/scoutsuite_exceptions_aws-switch.js
2019-07-04 12:58:41 wrkbx scout[7443] INFO Creating scoutsuite-report/aws-switch.html

The report generated correctly shows it's been run against 222222222222.

[x4v13r64]

Closing, this should be resolved by https://github.com/nccgroup/ScoutSuite/pull/460. If not please reopen the issue.

[hashier]

It works now for me as well. Used again same commit, but moved a bit things around in ~/.aws/config and ~/.aws/credentials. Weirdly it worked before with aws-cli already.

Anyway, now it works. Thanks for the fix!

[x4v13r64]

Cheers, should be merged into master within a few weeks.