Scoutsuite: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

[rams3sh]

I get the following error when I run scoutsuite for my aws account.

I had already tried setting the max-workers to 1 just in case. I had also downgraded the boto3-1.9.193 and botocore-1.12.193 to botocore==1.12.135 boto3==1.9.135 as per the suggestion in #381 as just because "Task exception was never retrieved" keyword (which is also present in the below error) was found in it. Just tried whatever I could to resolve the issue, but no luck.

Error snippet below :-
```Task exception was never retrieved
future: exception=UnicodeDecodeError('utf-8', b'\x1f\x8b\x08\x00\xf8\xc56]\ < long string of hex> \x00', 1, 2, 'invalid start byte')>
Traceback (most recent call last):
File "/home/user/Tools/ScoutSuite/ScoutSuite/providers/aws/resources/ec2/instances.py", line 17, in fetch_all
name, resource = await self._parse_instance(raw_instance)
File "/home/user/Tools/ScoutSuite/ScoutSuite/providers/aws/resources/ec2/instances.py", line 26, in _parse_instance
instance['user_data'] = await self.facade.ec2.get_instance_user_data(self.region, id)
File "/home/user/Tools/ScoutSuite/ScoutSuite/providers/aws/facade/ec2.py", line 33, in get_instance_user_data
return base64.b64decode(user_data_response['UserData']['Value']).decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
2019-07-23 16:02:50 user.internal scout[22887] INFO Matching EC2 instances and IAM roles
2019-07-23 16:02:50 user.internal scout[22887] INFO Processing CloudTrail config
2019-07-23 16:02:50 user.internal scout[22887] INFO Running rule engine
2019-07-23 16:02:55 user.internal scout[22887] ERROR browser.py L96: 'enable_user_activity_logging'
2019-07-23 16:02:55 user.internal scout[22887] ERROR browser.py L96: 'value'
2019-07-23 16:02:56 user.internal scout[22887] INFO Applying display filters
2019-07-23 16:03:00 user.internal scout[22887] ERROR browser.py L96: 'enable_user_activity_logging'
2019-07-23 16:03:00 user.internal scout[22887] ERROR browser.py L96: 'value'

Current Enviroment Details :- 

(env) [[email protected] ScoutSuite]$ pip freeze
adal==1.2.2
antlr4-python3-runtime==4.7.2
applicationinsights==0.11.9
argcomplete==1.10.0
asn1crypto==0.24.0
azure-cli-core==2.0.69
azure-cli-nspkg==3.0.4
azure-cli-telemetry==1.0.3
azure-common==1.1.23
azure-mgmt-keyvault==2.0.0
azure-mgmt-monitor==0.7.0
azure-mgmt-network==4.0.0
azure-mgmt-redis==6.0.0
azure-mgmt-resource==2.2.0
azure-mgmt-security==0.2.0
azure-mgmt-sql==0.12.0
azure-mgmt-storage==4.0.0
azure-mgmt-web==0.42.0
azure-nspkg==3.0.2
backports.functools-lru-cache==1.5
bcrypt==3.1.7
boto3==1.9.135
botocore==1.12.135
cachetools==3.1.1
certifi==2019.6.16
cffi==1.12.3
chardet==3.0.4
cheroot==6.5.5
CherryPy==18.1.2
cherrypy-cors==1.6
colorama==0.4.1
coloredlogs==10.0
cryptography==2.7
docutils==0.14
google-api-core==1.14.0
google-api-python-client==1.7.9
google-auth==1.6.3
google-auth-httplib2==0.0.3
google-cloud-container==0.2.1
google-cloud-core==1.0.2
google-cloud-iam==0.1.0
google-cloud-kms==1.1.0
google-cloud-logging==1.11.0
google-cloud-monitoring==0.31.1
google-cloud-resource-manager==0.29.1
google-cloud-storage==1.16.1
google-resumable-media==0.3.2
googleapis-common-protos==1.6.0
grpc-google-iam-v1==0.11.4
grpcio==1.22.0
httpagentparser==1.8.2
httplib2==0.13.0
humanfriendly==4.18
iampoliciesgonewild==1.0.6.2
idna==2.8
isodate==0.6.0
jaraco.functools==2.0
jmespath==0.9.4
knack==0.6.3
more-itertools==7.2.0
msrest==0.6.8
msrestazure==0.6.1
netaddr==0.7.19
oauth2client==4.1.3
oauthlib==3.0.2
paramiko==2.6.0
portalocker==1.5.0
portend==2.5
protobuf==3.9.0
pyasn1==0.4.5
pyasn1-modules==0.2.5
pycparser==2.19
Pygments==2.4.2
PyJWT==1.7.1
PyNaCl==1.3.0
pyOpenSSL==19.0.0
python-dateutil==2.8.0
pytz==2019.1
PyYAML==5.1.1
requests==2.22.0
requests-oauthlib==1.2.0
rsa==4.0
s3transfer==0.2.1
six==1.12.0
sqlitedict==1.6.0
tabulate==0.8.3
tempora==1.14.1
uritemplate==3.0.0
urllib3==1.24.3
zc.lockfile==1.4
(env) [[email protected] ScoutSuite]$ python --version
Python 3.7.3
```

[rams3sh]

Changing 'ScoutSuite/ScoutSuite/providers/aws/facade/ec2.py' line 33

from
return base64.b64decode(user_data_response['UserData']['Value']).decode('utf-8')
to
return zlib.decompress(base64.b64decode(user_data_response['UserData']['Value']), zlib.MAX_WBITS | 32).decode('utf-8')

solves the issue. Seems like the type of data is either gzip or zlib format. Left the zlib library to automatically identify the header.

Reference for code : https://stackoverflow.com/a/22310760 (under heading Automatic Header detection) .

P.S : zlib has to be imported as part of the code :P

[x4v13r64]

Thanks @rams3sh - that's odd, should be b64 so not sure why it's throwing that error. Does the user data have any special characters?

[rams3sh]

However correcting with above solution lead to the following error :-

2019-07-23 19:52:08 user.internal scout[2570] INFO Launching Scout
2019-07-23 19:52:08 user.internal scout[2570] INFO Authenticating to cloud provider
2019-07-23 19:52:08 user.internal scout[2570] INFO Gathering data from APIs
2019-07-23 19:52:08 user.internal scout[2570] INFO Fetching resources for the EC2 service
2019-07-23 19:53:19 user.internal scout[2570] ERROR provider.py L376: 'subnet-01d**************'
2019-07-23 19:53:19 user.internal scout[2570] ERROR provider.py L376: 'subnet-04e**************'
2019-07-23 19:53:19 user.internal scout[2570] ERROR provider.py L376: 'subnet-04e**************'

This was happening due to subnet_map was empty when the above subnet-id's were being queried as keys in subet_map.
modifying function match_instances_and_subnets_callback of ScoutSuite/ScoutSuite/providers/aws/provider.py
to

 def match_instances_and_subnets_callback(self, current_config, path, current_path, instance_id, callback_args):
        subnet_id = current_config['SubnetId']
        if subnet_id:
            if self.subnet_map != {}:
                vpc = self.subnet_map[subnet_id]
                subnet = self.services['vpc']['regions'][vpc['region']
                ]['vpcs'][vpc['vpc_id']]['subnets'][subnet_id]
                manage_dictionary(subnet, 'instances', [])
                if instance_id not in subnet['instances']:
                    subnet['instances'].append(instance_id)

solved the error.

I am not sure if there is any logical flaw with the provided solutions which might hinder with scoutsuite processing as I am not well versed with the entire code as my approach was more from removing the error. Keeping this issue open for review

[rams3sh]

Thanks @rams3sh - that's odd, should be b64 so not sure why it's throwing that error. Does the user data have any special characters?

Will check and get back with original code. !!

[rams3sh]

Thanks @rams3sh - that's odd, should be b64 so not sure why it's throwing that error. Does the user data have any special characters?

code changed to

The following is a redacted version of the original output :-

```
Raw Data :-
H4sIAKzrNl0AA7U8Z2PjNrLf9SsQrS .......

Base 64 Decoded Data :-
b'\x1f\x8b\x08\x00\xac\xeb6]\x00\x03\xb5

Zlib Decoded Data:-
b'MIME-Version: 1.0\nContent-Type: multipart/mixed; boundary=BlRcNGG8TfGQbWESl6i4GPmOKQjs ......

UTF Decoded Data:-
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary=BlRcNGG8TfGQbWESl6i4GPmOKQjs

..........
````

Also if you see the initial hex in base64 decoded data , you will know that it is magic number of gzip. Also thats the reason utf decode was giving errors before that codec can't decode byte 0x8b .

Hope this helps .

[x4v13r64]

Thanks that's very helpful. So essentially it's breaking because the user data contains gzip encoded data?

[rams3sh]

Yes.

Also found the root cause. Basically the assumption of user data being always base64 encoded in the code is wrong.

Why ?
Official Link :-
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/amazon-linux-ami-basics.html (Check under the heading "Supported User-Data Formats") - Couldnt link the title directly.

Though not directly related , this article led me to above reference. Link: https://bugs.launchpad.net/ec2-api/+bug/1776398

In my case , user data was gzip compressed data. Hence this issue.

To resolve this issue , you need to consider all the possibilities of supported user-data and then return the output.

That should solve this bug.

[rams3sh]

However correcting with above solution lead to the following error :-

2019-07-23 19:52:08 user.internal scout[2570] INFO Launching Scout
2019-07-23 19:52:08 user.internal scout[2570] INFO Authenticating to cloud provider
2019-07-23 19:52:08 user.internal scout[2570] INFO Gathering data from APIs
2019-07-23 19:52:08 user.internal scout[2570] INFO Fetching resources for the EC2 service
2019-07-23 19:53:19 user.internal scout[2570] ERROR provider.py L376: 'subnet-01d**************'
2019-07-23 19:53:19 user.internal scout[2570] ERROR provider.py L376: 'subnet-04e**************'
2019-07-23 19:53:19 user.internal scout[2570] ERROR provider.py L376: 'subnet-04e**************'

This was happening due to subnet_map was empty when the above subnet-id's were being queried as keys in subet_map.
modifying function match_instances_and_subnets_callback of ScoutSuite/ScoutSuite/providers/aws/provider.py
to

 def match_instances_and_subnets_callback(self, current_config, path, current_path, instance_id, callback_args):
        subnet_id = current_config['SubnetId']
        if subnet_id:
            if self.subnet_map != {}:
                vpc = self.subnet_map[subnet_id]
                subnet = self.services['vpc']['regions'][vpc['region']
                ]['vpcs'][vpc['vpc_id']]['subnets'][subnet_id]
                manage_dictionary(subnet, 'instances', [])
                if instance_id not in subnet['instances']:
                    subnet['instances'].append(instance_id)

solved the error.

I am not sure if there is any logical flaw with the provided solutions which might hinder with scoutsuite processing as I am not well versed with the entire code as my approach was more from removing the error. Keeping this issue open for review

Given above the root cause , this seems like a different bug altogether. There seems a possibility of subnet_map being empty in some cases.

[x4v13r64]

this seems like a different bug altogether

yep

[x4v13r64]

Closing, this should be now fixed in develop.