Boto3: ResourceWarning: unclosed ssl.SSLSocket

[LinusU]

For some reason I'm getting a ResourceWarning about a unclosed socket, even when I'm specifically closing the socket myself. See testcase below:

python3 -munittest discover

import sys
import boto3
import unittest

BUCKET = ''
KEY = ''


def give_it_to_me():
    client = boto3.client('s3')
    obj = client.get_object(Bucket=BUCKET, Key=KEY)
    try:
        yield from iter(lambda: obj['Body'].read(1024), b'')
    finally:
        print('Im closing it!', file=sys.stderr, flush=True)
        obj['Body'].close()


class TestSomeShit(unittest.TestCase):
    def test_it(self):
        res = give_it_to_me()
        for chunk in res:
            pass
        print('Done', file=sys.stderr, flush=True)

Fill in any BUCKET and KEY to see the problem. Attaching my output below:

Im closing it!
test.py:22: ResourceWarning: unclosed <ssl.SSLSocket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('...', 55498), raddr=('...', 443)>
  for chunk in res:
Done
.
----------------------------------------------------------------------
Ran 1 test in 0.696s

OK

[kyleknap]

I am able to reproduce this. Note you can only reproduce this with python3. More research needs to be done to see if it is an issue with our underlying http library or it is something with our implementation. I have found a few related python bugs, but not sure if those fit the bill especially on Python3.5 I am still seeing them:
http://bugs.python.org/issue16900
http://bugs.python.org/issue12133

[LinusU]

Should probably mention that I'm running python 3.5 as well, 3.5.1 to be specific.

[nchammas]

I'm seeing this as well. Enabling ResourceWarning yields a lot of complaints related to unclosed sockets. Also running Python 3.5.1.

[Naddiseo]

I have also run into this since upgrading to python 3.5. From what I can tell it's something to with connection pooling [0]. I have found a way to "fix" the warnings, although I'm not sure it's the correct way to do so: in botocore/awsrequest.py, explicitly set the Connection: close header in the AWSRequest class:

``` #!diff
--- awsrequest.py.old 2016-04-05 15:18:56.931892889 -0600
+++ awsrequest.py 2016-04-05 14:56:17.422863960 -0600
@@ -340,6 +340,7 @@ class AWSRequest(models.RequestEncodingM
del kwargs['auth_path']
models.Request.__init__(self, args, *kwargs)
headers = HTTPHeaders()

• headers['Connection'] = 'close'
  if self.headers is not None:
  for key, value in self.headers.items():
  headers[key] = value
  ```

[0] https://github.com/kennethreitz/requests/issues/2963#issuecomment-169631513

[rehmanzile]

I am also experiencing issues with python 3.5. Had to go back to 2.7.

Make sure to use the following command to install AWS cli:

pip install awscli --ignore-installed six

[jucor]

Pinging this thread: is there a solution merged somewhere, please?

[robmcdan]

+1, also on python 3.5 -- any update?

[aausch]

+1, seeing on python 3.6

[xiaochuan-du]

+1, seeing on python 3.6.1

[muppetjones]

+1 Still an issue. The fix given by @Naddiseo works.

[DinoBytes]

+1, seeing on python 3.6.1

[nueverest]

python 3.6.1 I'm testing cognito.

[diegojancic]

+1 boto3=1.4.4, python=3.6.1

[muppetjones]

+1

boto3==1.4.4
botocore==1.5.38
Python 3.5.2

[sleepdeprivation]

For future readers... I did a little research, and also clicked the link in the above discussion given by @Naddiseo and according to someone working on Requests lib:

You don't have to. The warning is just that: a warning. No error occurs when you see it, and it is not an indication of the program doing the wrong thing. It's entirely expected behaviour, and this is working as designed. However, if you're concerned about them, you may call close.

and

The reason the resource warnings are back is because we're much better about not returning partially complete connections to the connection pool. The lack of warnings in 2.5.4 was actually symptomatic of a problem: weirdly, the warnings are an indication that everything is working as intended.

So looks like not an issue. Why is it coming up though? It looks like Python is supposed to suppress such warnings by default. Seems like somewhere around 3.2 (?) they changed the behavior of unittest to override the default behavior on warnings. You can suppress the warnings following this SO. This makes it clear why, if you google around for this issue, it seems to occur a lot in the context of "I did a unit test and now this!".

(* edit, added credit where credit is due)

[jdufresne]

You don't have to. The warning is just that: a warning. No error occurs when you see it, and it is not an indication of the program doing the wrong thing. It's entirely expected behaviour, and this is working as designed. However, if you're concerned about them, you may call close.

I find this advice contrary to Python's typical approach to resource handling. Python encourages deterministic resource handling. For example, the stdlib produces warnings when files are mismanaged in a non-deterministic way to help guide developers. Developers _should_ take action on the warnings produced by stdlib. See the following Python bug where core developers discuss warnings during destructors:

https://bugs.python.org/issue28867

Additionally, Python docs warn against relying on __del__ for resource management:

It is not guaranteed that __del__() methods are called for objects that still exist when the interpreter exits.

And, exceptions occurring during __del__ are ignored and can't be handled:

Due to the precarious circumstances under which __del__() methods are invoked, exceptions that occur during their execution are ignored, and a warning is printed to sys.stderr instead.

I run all my tests and environments with warnings enabled to help catch _real_ bugs and upgrade paths. If this warning really should be ignored as suggested here, then maybe it shouldn't be logged. As it is now, people are taking this advice of not calling close and this results in lots of (ignorable?) noise in my output.

[Naddiseo]

Another work around based upon https://github.com/boto/botocore/pull/1231 is using the event system:

def http_closer(http_response, **kwargs):
    if http_response:
        http_response.close()

s3 = boto3.resource('s3', aws_access_key_id="XXX", aws_secret_access_key="YYY")
events = s3.meta.client.meta.events
# For HeadBucket for example, 'after-call.s3.*' could also work
events.register("after-call.s3.HeadBucket", http_closer)

try:
    s3.meta.client.head_bucket(Bucket="mybucket")
except botocore.exceptions.ClientError as e:
    print("Could not find bucket", e)

[scchess]

So no fix? Warning is still important. boto3 developers are lazy.

[nchammas]

@student-t

So no fix? Warning is still important. boto3 developers are lazy.

Don’t use this kind of language, especially not on an open source project where the developers owe you nothing and you are getting their work for free.

If this issue is so important to you that you can’t wait for a fix to be submitted and merged in, then fork the project and fix it yourself, and spare us all this kind of low quality whining.

[lmeraz]

I was receiving the same warning when working with Polly. The following implementation removed the warning. Running python=3.6, Boto3=1.5.

# core.py
import boto3
from botocore.exceptions import BotoCoreError
from contextlib import closing

polly = boto3.client('polly')

def polly_stream(text):
    try:
        response = polly.synthesize_speech(
            OutputFormat='mp3',
            Text=text,
            TextType='text',
            VoiceId='Salli',)
        with closing(response["AudioStream"]) as stream:
            return stream.read()
    except BotoCoreError as error:
        raise error

# test.py
from unittest import main, TestCase
from core import polly_stream


class TestVoice(TestCase):

    def test_polly_stream(self):
        stream = polly_stream('I')
        self.assertIsInstance(stream, bytes)


if __name__ == '__main__':
     main()



md5-9ef56f1aa51d937de25ed0de5f7aa9dd



Ran 1 test in 0.229s

OK

Process finished with exit code 0

Hope it helps.

[wimglenn]

Same issue for me. If the warning can be safely ignored, please squash it in lib. I don't want to disable all ResourceWarning.

Python 3.6.4 on Linux
boto3==1.6.19
botocore==1.9.19

[jannesh]

Having the same issue. The other fixes did not work for me, but this one did. Added the following to my setUp:

def setUp(self):
    warnings.filterwarnings("ignore", category=ResourceWarning, message="unclosed.*<ssl.SSLSocket.*>") 

The message parameter is a regex match for the message part of the warning, so this can be adjusted to filter out only this one warning and leave the others intact.

Python: 3.6.2 on Windows
boto3: 1.4.4
botocore:1.5.84

[epicfaace]

Any progress on this issue?

[sjatkins]

Same thing in 3.7. Saw it first around S3 resource to put stuff in buckets. So it is systemic.

[ExplodingCabbage]

Note that by running Python with -X tracemalloc=100 we can get a traceback indicating where the error occurred. Here's an example (slightly anonymised) from my employer's codebase:

mytests.py:42: ResourceWarning: unclosed <ssl.SSLSocket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.10.129.97', 50203), raddr=('52.218.204.249', 443)>
  foo()
Object allocated at (most recent call last):
  File "mytests.py", lineno 61
    unittest.main()
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/main.py", lineno 101
    self.runTests()
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/main.py", lineno 271
    self.result = testRunner.run(self.test)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/runner.py", lineno 176
    test(result)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/suite.py", lineno 84
    return self.run(*args, **kwds)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/suite.py", lineno 122
    test(result)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/suite.py", lineno 84
    return self.run(*args, **kwds)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/suite.py", lineno 122
    test(result)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/case.py", lineno 676
    return self.run(*args, **kwds)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/case.py", lineno 628
    testMethod()
  File "mytests.py", lineno 42
    foo()
  File "/Users/markamery/myrepo/somemodule.py", lineno 72
    s3_file.put(Body=local_f, ServerSideEncryption='AES256')
  File "/usr/local/lib/python3.7/site-packages/boto3/resources/factory.py", lineno 520
    response = action(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/boto3/resources/action.py", lineno 83
    response = getattr(parent.meta.client, operation_name)(**params)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", lineno 357
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", lineno 648
    operation_model, request_dict, request_context)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", lineno 667
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", lineno 102
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", lineno 135
    request, operation_model, context)
  File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", lineno 167
    request, operation_model)
  File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", lineno 200
    http_response = self._send(request)
  File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", lineno 244
    return self.http_session.send(request)
  File "/usr/local/lib/python3.7/site-packages/botocore/httpsession.py", lineno 262
    chunked=self._chunked(request.headers),
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", lineno 600
    chunked=chunked)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", lineno 343
    self._validate_conn(conn)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", lineno 849
    conn.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", lineno 356
    ssl_context=context)
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/ssl_.py", lineno 359
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", lineno 423
    session=session
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", lineno 833
    self = cls.__new__(cls, **kwargs)

I suppose that somewhere in that rather hefty call stack there is a method in which somebody forgot to call .close() on something.

[hannes-ucsc]

To summarize, the warning is a false positive when it is made about a connection that's part of a urllib connection pool. The connections are meant to stay open because they are idle and waiting to be reused for subsequent HTTP requests.

Once it has been determined that the warning is indeed a false positive, the developers of urllib should prevent it from being logged. As far as I know, suppression via filterwarnings is imprecise and could lead to false negatives. Not every unclosed socket or SSLSocket is part of a urllib3 connection pool.

It is this line in socketmodule.c that logs the warning in the finalizer of socket.socket instances. I think a solution may be to add an idle attribute to those instances. When urllib returns a connection to the pool, it should mark that connection as idle by setting that attribute. The warning should only be emitted for socket instances that are not idle. Once a connection is taken out of the pool and actively used, the idle attribute should be reset. This would require a change to CPython codebase.

[vtermanis]

I think a solution may be to add an idle attribute to those instances

It would be far easier to expose a close() method to allow for the underlying pool to explicitly be closed. (This is already supposed by e.g URLLib3Session)
Using coarse filter warnings for all unclosed sockets seems like overkill - especially if the application uses more than just a urllib pool or boto3.

[straz]

I'm getting this warning too. I'm using the above trick to silence the warning during testing, but I would prefer an explicit close() as previously suggested.

import unittest
import boto3
import json

class MyTest(unittest.TestCase):

    MY_SECRET = 'secret-treasure-map'

    def test_a_secret(self):
        client = boto3.client('secretsmanager')
        response = client.get_secret_value(SecretId=self.MY_SECRET)
        secret = json.loads(response['SecretString'])
        self.assertIn('password', secret)

```shell
executing: python -m unittest discover -p *_test.py --top-level-dir /Users/me/myproject
/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/unittest/case.py:633: ResourceWarning: unclosed method()
ResourceWarning: Enable tracemalloc to get the object allocation traceback

[khuongduybui]

I wish there were a static boto3.tearDown() that I can call in my unittest.TestCase#tearDown.

[lobsterkatie]

Using AWS lambda, I had to do:

def close_client_connection(client):
    session = client._endpoint.http_session
    managers = [session._manager, *session._proxy_managers.values()]
    for manager in managers:
        manager.clear()

client.meta.events.register(
    "after-call.lambda.DeleteFunction",
    lambda **kwargs: close_client_connection(client),
)

(My test has a finalizer which deletes the lambda function, but presumably you could use after-call.lambda.* - whatever operation you're doing.)

Thanks to @Naddiseo for this comment and @vtermanis for his work on this PR, as both provided inspiration.

[vtermanis]

(Shameless bump) Dear boto3 team, please either consider https://github.com/boto/botocore/pull/1810 or add a more appropriate cleanup approach which end-users can access.