Botocore: filedescriptor out of range in select() while calling message.delete

[aniljava]

s = boto3.session.Session(aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key, region_name=region_name)
session = s.resource('sqs')

queue = sqs.get_queue_by_name(QueueName=queue_name)
while True:
    messages = queue.receive_messages(MaxNumberOfMessages=10, AttributeNames=['All'], MessageAttributeNames=['All'])
    if len(messages) == 0:
        break
    for message in messages:
        save_message(message)
        message.delete()

Stack Trace

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/ws/amazon/bin/sqs", line 59, in each_thread
    message.delete()
  File "/usr/local/lib/python2.7/dist-packages/boto3/resources/factory.py", line 455, in do_action
    response = action(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/boto3/resources/action.py", line 79, in __call__
    response = getattr(parent.meta.client, operation_name)(**params)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 310, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 384, in _make_api_call
    operation_model, request_dict)
  File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 111, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 140, in _send_request
    success_response, exception):
  File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 213, in _needs_retry
    caught_exception=caught_exception)
  File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 226, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 209, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 250, in __call__
    caught_exception)
  File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 265, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 313, in __call__
    caught_exception)
  File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 222, in __call__
    return self._check_caught_exception(attempt_number, caught_exception)
  File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 355, in _check_caught_exception
    raise caught_exception
ValueError: filedescriptor out of range in select()

[JordonPhillips]

This looks like an issue with the underlying HTTP library shazow/urllib3#589. How often does it happen?

[aniljava]

Errors are more prominent when the load is high. My feeling is lots of threads trying to squeeze the available bandwidth, (timeouts ?).

I am currently letting the errors pass and retry later.

[JordonPhillips]

It seems that the bug I linked is related to the pyopenssl bindings. We recently introduced a change that disables pyopenssl integration in our vendored version of urllib3, so that should fix the issue. Could you update and see if it goes away?

[RussellLuo]

Has this issue been fixed?

I encountered the same issue, but I didn't see any change in botocore as @JordonPhillips mentioned.

If I am not mistaken, https://github.com/shazow/urllib3/issues/589 was finally fixed by https://github.com/shazow/urllib3/pull/1001, in which urllib3/contrib/pyopenssl.py is changed to use util.wait_for_read()/util.wait_for_write(). But select.select() is still used in botocore/vendored/requests/packages/urllib3/contrib/pyopenssl.py

[marktheunissen]

Make sure you're closing all the file descriptors you're opening. We had this error and found it was related to opening temp files and not closing them

[alsrgv]

@JordonPhillips, can botocore upgrade to latest version of requests/urllib3 to take advantage of the fix mentioned by @RussellLuo ?

[tkaria]

@alsrgv I'm running urllib3 1.21.1 (the latest as of today) and still getting this error. Any other thoughts?

[dustinboswell]

In our application we routinely open > 1024 file descriptors, and run into this issue too. In our case, we can't close the underlying files (they are not temp files). Any ideas on a workaround?

[shrishinde]

Facing same issue while uploading file to S3.

s3 = boto3.resource('s3')
s3.Object(target_bucket, file_name).put(Body=data)

data is fd for file to be uploaded.

[shrishinde]

For me problem was with Python program where this code was being invoked.
In some other module file descriptor were leaking and few thousands of FDs were not getting cleaned. Probably even Python 2.7 also has some limit with # of FDs it can handle.

[agajews]

This issue has not been fixed; I think updating the vendored version of urllib3 would do the trick, but there have been enough changes to urllib since version 1.10.x that naively replacing the whole package with the latest version breaks things.

[tobycatlin]

I am having the exact same issue today. I cannot see why it was closed, is there a duplicate issue?

[eliasbrange]

This needs to be reopened. We regularly have more than 1024 sockets open and using boto to upload a file to S3 fails at that state since your vendored urllib3 version uses select.select().

[Hernrup]

We are also experiencing this issue now. We rely need a new urllib to fix multiple bugs. Boto3 is currently not usable for our company.

[Hernrup]

We may be able to put some time on a PR if needed or feasible but we would much appropriate having a discussion on how to resolve this and find a plan and timetable for moving forward.

[mixeract]

I'm getting the same error with s3 client upload_file call.

[joguSD]

We have un-vendored requests/urllib3 and are now using urllib3 directly as of botocore v1.11.0.
Relevant upgrade notes.