Boto3: Boto3 causing segmentation fault

[JasonXJ]

In some special circumstance, using boto3 will cause segmentation fault.

The code that can reproduce it is as follows. There are totally 3 files (and one subdirectory) need to be created because I am not able to reproduce it with only one file. You can also clone from https://github.com/JasonXJ/segmentation_fault_with_boto3

./mypackage/__init__.py:

import sys
import boto3

def foo():
    bar(boto3.resource('dynamodb', region_name='us-east-2'))
    sys.exit(0)  # Unable to reproduce without this line


# Unable to reproduce if merge this function with foo()
def bar(dynamodb):
    import mypackage.justtyping  # Unable to reproduce without this line

    # Unable to reproduce if do not get item twice, or use `dynamodb` in both
    # cases, or create new resource in both cases.
    # Note that the table must exist, but it does not need to have the item.
    dynamodb.Table('sigsegv').get_item(Key={'pkey': 'xxx'})
    boto3.resource('dynamodb', region_name='us-east-2').Table('sigsegv').get_item(Key={'pkey': 'xxx'})

./mypackage/justtyping.py

from typing import Sequence
x = Sequence['xxx']  # Unable to reproduce without this line, or replace string `'xxx'` with a class (e.g. `bytes`)

./run.py

import mypackage
mypackage.foo()

Run the code and get the segmentation fault:

$ python3 run.py
[1]    26495 segmentation fault (core dumped)  python3 run.py
FAIL

Versions: python 3.5.2, boto3 1.4.4, ubuntu 16.04.2

[JordonPhillips]

I can confirm the issue on the specified versions. I'm continuing to investigate, but for now you can work around the issue by upgrading to python 3.6, which does not seem to be impacted.

[JordonPhillips]

Python 3.5.3 is also not impacted.

The CPython raw stack trace wasn't super helpful, but py-bt turned up something interesting:

Traceback (most recent call first):
  File "/usr/lib/python3.5/socket.py", line 149, in __repr__
    closed = getattr(self, '_closed', False)
  Garbage-collecting

I'll need to keep looking.

[JordonPhillips]

No significant changes between 3.5.2 and 3.5.3:

diff --git a/socket.py b/socket.py
index 8977e68..1774df5 100644
--- a/socket.py
+++ b/socket.py
@@ -258,7 +258,7 @@ class socket(_socket.socket):
                 raise _GiveupOnSendfile(err)  # not a regular file
             try:
                 fsize = os.fstat(fileno).st_size
-            except OSError:
+            except OSError as err:
                 raise _GiveupOnSendfile(err)  # not a regular file
             if not fsize:
                 return 0  # empty file
@@ -519,6 +519,7 @@ else:
         finally:
             lsock.close()
         return (ssock, csock)
+    __all__.append("socketpair")

 socketpair.__doc__ = """socketpair([family[, type[, proto]]]) -> (socket object, socket object)
 Create a pair of socket objects from the sockets returned by the platform

So that may be a dead end.

[JordonPhillips]

This just seems to be a bug in python 3.50-3.5.2 garbage collection. I've not been able to figure out exactly why it's happening, but I don't think there's anything we could do regardless. My recommendation is to upgrade to at least 3.5.3

[thihara]

@JordonPhillips

Experienced the same thing.

I was using Apache 2.4, mod_wsgi and python 2.7.12 to run an django app.

Finally switched back to boto from boto3 and it worked.

Here's a GDB backtrace.

(gdb) backtrace.txt

[thihara]

@JordonPhillips

Oh and I reinstalled python, upgraded to 2.7.13 and even launched a new Ubuntu instance, so I'm guessing the fault is strictly in boto3 instead of python.

Also running the built in dev server of django didn't throw any errors either.

[JordonPhillips]

Segmentation faults are python bugs, python is supposed to protect against them. This isn't something that we can fix unfortunately.

[icygorilla]

i got the same error, when i import it in my project in docker container.

any recomendation to solving it ?

i used python 3.6.1

[ztravis]

FYI I think the relevant bug is:
https://bugs.python.org/issue30260

Open sockets (see #454) may cause segfaults when GC'd at shutdown. This has been fixed in python 3.6 (definitely still broken for me in 3.5.2, not sure about 3.5.3).

[kmurphy4]

FYI, this works for me in python 3.6.8, but not in python 3.7.3. Maybe this is worth reporting to python?

[anuj9196]

I'm also facing this issue. I'm using django-storages and files are being created in the S3 but while reading the file, it gives Segmentation fault (core dumped) error.

[kdpisda]

I switched to Python 3.6.9 and it is working, earlier it was not working with Python 3.7.x

[takarzyna]

I had the same problem in Python 3.7.6, but it worked after upgrading to Python 3.7.7

[chrisdlangton]

i am also seeing segfault #2534
but only when using S3, in my testing i went back to python 3.7 and saw the error existed so i ruled out it being a python bug and didn't go back further.

I think there might be a commonality somewhere in boto3 session code paths, if we assume there is a connection.