Scoutsuite: GCP "OSError: [Errno 24] Too many open files"

[x4v13r64]

When scanning a large GCP organization, Scout will create a large number of clients for services where the library isn't thread-safe (i.e. GCE and GKE). This in turn reaches the maximum # of open files the host system can handle, and triggers the following error:

2019-06-13 09:14:55 <user> scout[30183] ERROR gke_private.py L15: Failed to retrieve clusters: [Errno 24] Too many open files
Traceback (most recent call last):
  File "/home/<user>/Git/ScoutSuite/ScoutSuite/providers/gcp/facade/gke_private.py", line 15, in get_clusters
    gke_client = self._get_client()
  File "/home/<user>/Git/ScoutSuite/ScoutSuite/providers/gcp/facade/basefacade.py", line 18, in _get_client
  File "/home/<user>/Git/ScoutSuite/ScoutSuite/providers/gcp/facade/basefacade.py", line 10, in _build_client
  File "/home/<user>/Git/ScoutSuite/ScoutSuite/providers/gcp/facade/basefacade.py", line 13, in _build_arbitrary_client
  File "/home/<user>/Git/ScoutSuite/venv/lib/python3.6/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/<user>/Git/ScoutSuite/venv/lib/python3.6/site-packages/googleapiclient/discovery.py", line 224, in build
  File "/home/<user>/Git/ScoutSuite/venv/lib/python3.6/site-packages/googleapiclient/discovery.py", line 274, in _retrieve_discovery_doc
  File "/home/<user>/Git/ScoutSuite/venv/lib/python3.6/site-packages/httplib2/__init__.py", line 1761, in request
    disable_ssl_certificate_validation=self.disable_ssl_certificate_validation,
  File "/home/<user>/Git/ScoutSuite/venv/lib/python3.6/site-packages/httplib2/__init__.py", line 1239, in __init__
    self.disable_ssl_certificate_validation, self.ca_certs, cert_file, key_file
  File "/home/<user>/Git/ScoutSuite/venv/lib/python3.6/site-packages/httplib2/__init__.py", line 194, in _build_ssl_context
    context.load_verify_locations(ca_certs)
OSError: [Errno 24] Too many open files

[rusty-martin]

I was able to resolve this by increase the open file count with the below command as a work around. You may need to increase depending on the size of the environments but 1000 did the trick for me. The default I had was 256 which was also having issues with other apps.

ulimit -Sn 1000

to verify size use "ulimit -a"

[x4v13r64]

Thank you @rusty-martin. Nonetheless ideally I'd like to resolve this issue in Scout. We really shouldn't be spawning that many clients - if only the libraries were thread-safe...

[l01cd3v]

fwiw, this is not GCP-specific; I hit the same issue when running scout for AWS and the same workaround worked.

[x4v13r64]

@l01cd3v was this for a specific service? I'm guessing in this case too we're spawning a large amount of clients.

[l01cd3v]

This impacted pretty much all the services. Before changing the open file count, Scout would work when running just a handful of services at a time. I tried to use the --update feature but, while the data remains available in the JSON results, the HTML report does not allow access to data fetched in previous runs.

./scout.py aws --services iam
./scout.py aws --services s3 --update

The results JS file still has all the IAM data, but it's not accessible via the HTML report. I tried running a local analysis afterwards and it didn't seem to help. Saw that you may have some plans for the update feature (https://github.com/nccgroup/ScoutSuite/issues/361), so not sure if you care about opening an issue about it at all.

[x4v13r64]

Yeah not sure what the state of that functionality is currently - may be worth adding any comments to https://github.com/nccgroup/ScoutSuite/issues/361.

[x4v13r64]

Closing, this should now be fixed in develop.