Maskrcnn-benchmark: Segmentation fault (core dumped)

[youngkyoonjang]

It breaks by returning message 'Segmentation fault (core dumped)'.

I think I followed the installation guide correctly. But when I run the code (maskrcnn-benchmark/demo/webcam.py), it breaks by returning 'Segmentation fault (core dumped)'.

The specific location returning the message is
line 27 in the file 'boxlist_ops.py': keep = _box_nms(boxes, score, nms_thresh)
from
line 114 in the file 'inference.py': boxlist = boxlist_nms( .... )
from
line 138 in the file 'inference.py': sampled_boxes.append(self.forward_for_single_feature_map(a, o, b))
from
line 122 in the file 'rpn.py': boxes = self.box_selector_test(anchors, objectness, rpn_box_regression)
from
line 96 in the file 'rpn.py': return self._forward_test(anchors, objectness, rpn_box_regression)
from
line 50 in the file 'generalised_rcnn.py': proposals, proposal_losses = self.rpn(images, features,
targets)
from
line 205 in the file 'predictor.py': predictions = self.model(image_list)
from
file 'webcam.py': composite = coco_demo.run_on_opencv_image(img)

I thought it was because of my small GPU memory problem on my local machine (PC). But when I tried to run the same code on an available deep learning machine which has enough (about 11GB GPU) memory. It still returns the same message and breaks.

What is my problem? It would be grateful if someone has an answer. Thanks in advance.

Best,
Young

[senlinuc]

updating gcc to 4.9 and rebuild can fix it
https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6

[fmassa]

Hi,

Could you run the python script via gdb and show the traceback?

gdb python
 >> run demo.py
.....
>> bt

Also, Does this segfault when running on the CPU?

Thanks!

[ll490187880]

@youngkyoonjang Hello,I also encountered the problem, after the output of "Start training" ,the process stopped without any output. And I saw "[1] 8817 segmentation fault (core dumped) python tools/train_net.py --config-file" in the shell.

[fmassa]

Hi @ll490187880 ,

Could you try running the aforementioned commands and give the output of the stack trace?
Something like

gdb python
>> run "tools/train_net.py --config-file ..."

and once it crashes, run

>> bt

and paste the result?

[ll490187880]

@fmassa yeah, as follows:
Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007fffac102424 in __gnu_cxx::new_allocator<_object*>::construct<_object*, _object*> (__p=0xb, this=0x5555565456e8) at /usr/include/c++/4.8.2/ext/new_allocator.h:120
120 { ::new((void *)__p) _Up(std::forward<_Args>(__args)...); }
(gdb) bt

0 0x00007fffac102424 in __gnu_cxx::new_allocator<_object*>::construct<_object*, _object*> (__p=0xb, this=0x5555565456e8) at /usr/include/c++/4.8.2/ext/new_allocator.h:120

1 std::allocator_traits >::_S_construct<_object*, _object*> (__p=0xb, __a=...) at /usr/include/c++/4.8.2/bits/alloc_traits.h:254

2 std::allocator_traits >::construct<_object*, _object*> (__p=0xb, __a=...) at /usr/include/c++/4.8.2/bits/alloc_traits.h:393

3 std::vector<_object*, std::allocator<_object*> >::emplace_back<_object*> (this=0x5555565456e8) at /usr/include/c++/4.8.2/bits/vector.tcc:96

4 std::vector<_object*, std::allocator<_object*> >::push_back(_object*&&) (__x=, this=0x5555565456e8)

at /usr/include/c++/4.8.2/bits/stl_vector.h:920

[youngkyoonjang]

Hi,

Could you run the python script via gdb and show the traceback?

gdb python
 >> run demo.py
.....
>> bt

Also, Does this segfault when running on the CPU?

Thanks!

@fmassa I did it before following @senlinuc 's comments (updating gcc to 4.9 / rebuilding it):
gdb python
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /home/yj18885/anaconda3/envs/MaskRCNN-PyTorch1.0/bin/python3.6...done.
(gdb) run webcam.py
Starting program: /home/yj18885/anaconda3/envs/MaskRCNN-PyTorch1.0/bin/python webcam.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Missing separate debuginfo for /home/yj18885/.local/lib/python3.6/site-packages/cv2/.libs/libz-a147dcb0.so.1.2.3
Missing separate debuginfo for /home/yj18885/.local/lib/python3.6/site-packages/numpy/core/../.libs/libgfortran-ed201abd.so.3.0.0
[New Thread 0x7fffe580e700 (LWP 19715)]
[New Thread 0x7fffe500d700 (LWP 19716)]
[New Thread 0x7fffe280c700 (LWP 19717)]
[New Thread 0x7fffe000b700 (LWP 19718)]
[New Thread 0x7fffdd80a700 (LWP 19719)]
[New Thread 0x7fffd9009700 (LWP 19720)]
[New Thread 0x7fffd6808700 (LWP 19721)]
[Thread 0x7fffe280c700 (LWP 19717) exited]
[Thread 0x7fffd6808700 (LWP 19721) exited]
[Thread 0x7fffe500d700 (LWP 19716) exited]
[Thread 0x7fffdd80a700 (LWP 19719) exited]
[Thread 0x7fffe000b700 (LWP 19718) exited]
[Thread 0x7fffe580e700 (LWP 19715) exited]
[Thread 0x7fffd9009700 (LWP 19720) exited]
Detaching after fork from child process 19722.
Detaching after fork from child process 19724.
[New Thread 0x7fffd6808700 (LWP 19726)]
[New Thread 0x7fffd9009700 (LWP 19727)]
[New Thread 0x7fffdd80a700 (LWP 19728)]
[New Thread 0x7fffe000b700 (LWP 19729)]
[New Thread 0x7fff953c1700 (LWP 19730)]
[New Thread 0x7fff94bc0700 (LWP 19731)]
[New Thread 0x7fff8eefe700 (LWP 19732)]
[New Thread 0x7fff8e6fd700 (LWP 19733)]
[New Thread 0x7fff8defc700 (LWP 19734)]
[New Thread 0x7fff8d6fb700 (LWP 19735)]
[New Thread 0x7fff8cefa700 (LWP 19736)]
[New Thread 0x7fff8c6f9700 (LWP 19737)]
[New Thread 0x7fff8bef8700 (LWP 19738)]

Program received signal SIGSEGV, Segmentation fault.
0x00007fff95425424 in construct<_object*, _object*> (__p=0xb, this=0x13259e8) at /usr/include/c++/4.8.2/ext/new_allocator.h:120
120 { ::new((void *)__p) _Up(std::forward<_Args>(__args)...); }
Missing separate debuginfos, use: debuginfo-install glibc-2.17-222.el7.x86_64
(gdb) bt

0 0x00007fff95425424 in construct<_object*, _object*> (__p=0xb, this=0x13259e8) at /usr/include/c++/4.8.2/ext/new_allocator.h:120

1 _S_construct<_object*, _object*> (__p=0xb, __a=...) at /usr/include/c++/4.8.2/bits/alloc_traits.h:254

2 construct<_object*, _object*> (__p=0xb, __a=...) at /usr/include/c++/4.8.2/bits/alloc_traits.h:393

3 emplace_back<_object*> (this=0x13259e8) at /usr/include/c++/4.8.2/bits/vector.tcc:96

4 push_back (__x=, this=0x13259e8)

at /usr/include/c++/4.8.2/bits/stl_vector.h:920

5 loader_life_support (this=0x7fffffffaa90) at /home/yj18885/anaconda3/envs/MaskRCNN-PyTorch1.0/lib/python3.6/site-packages/torch/lib/include/pybind11/cast.h:44

6 pybind11::cpp_function::dispatcher (self=, args_in=0x7fff953ca8b8, kwargs_in=0x0)

at /home/yj18885/anaconda3/envs/MaskRCNN-PyTorch1.0/lib/python3.6/site-packages/torch/lib/include/pybind11/pybind11.h:618

7 0x00007ffff796a08d in _PyCFunction_FastCallDict (func_obj=func_obj@entry=0x7fff9762ae10, args=args@entry=0x7fff95571be0, nargs=, kwargs=kwargs@entry=0x0) at Objects/methodobject.c:231

8 0x00007ffff796a315 in _PyCFunction_FastCallKeywords (func=func@entry=0x7fff9762ae10, stack=stack@entry=0x7fff95571be0, nargs=, kwnames=kwnames@entry=0x0) at Objects/methodobject.c:294

9 0x00007ffff7a0349a in call_function (pp_stack=pp_stack@entry=0x7fffffffada0, oparg=oparg@entry=3, kwnames=kwnames@entry=0x0) at Python/ceval.c:4830

10 0x00007ffff7a0771c in _PyEval_EvalFrameDefault (f=, throwflag=) at Python/ceval.c:3328

11 0x00007ffff7a0309e in _PyEval_EvalCodeWithName (_co=0x7fff9549adb0, globals=globals@entry=0x7fff9548e798, locals=locals@entry=0x0, args=, argcount=2, kwnames=0x7fff9547a920,

kwargs=0x21281a8, kwcount=2, kwstep=kwstep@entry=1, defs=0x7fff9547ab60, defcount=defcount@entry=2, kwdefs=kwdefs@entry=0x0, closure=0x0, name=name@entry=0x7fff9548f370, qualname=0x7fff9548f370)
at Python/ceval.c:4159

12 0x00007ffff7a033b2 in fast_function (kwnames=0x2, nargs=, stack=, func=0x7fff9548d6a8) at Python/ceval.c:4971

13 call_function (pp_stack=pp_stack@entry=0x7fffffffb060, oparg=oparg@entry=4, kwnames=kwnames@entry=0x7fff9547a908) at Python/ceval.c:4851

14 0x00007ffff7a08184 in _PyEval_EvalFrameDefault (f=, throwflag=) at Python/ceval.c:3344

15 0x00007ffff7a02730 in _PyFunction_FastCall (co=, args=, nargs=4, globals=) at Python/ceval.c:4912

16 0x00007ffff7a03654 in fast_function (kwnames=0x0, nargs=, stack=, func=0x7fff95524488) at Python/ceval.c:4947

17 call_function (pp_stack=pp_stack@entry=0x7fffffffb290, oparg=oparg@entry=3, kwnames=kwnames@entry=0x0) at Python/ceval.c:4851

18 0x00007ffff7a0771c in _PyEval_EvalFrameDefault (f=, throwflag=) at Python/ceval.c:3328

19 0x00007ffff7a0309e in _PyEval_EvalCodeWithName (_co=_co@entry=0x7fff955261e0, globals=globals@entry=0x7fff9548eb88, locals=locals@entry=0x0, args=args@entry=0x7fffffffb520, argcount=argcount@entry=4,

kwnames=kwnames@entry=0x7ffff7f93060, kwargs=kwargs@entry=0x7ffff7f93068, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=0x7fff9549fd48, defcount=1, kwdefs=0x0, closure=0x0, name=0x7fffd0687a78, 
qualname=0x7fff955233a0) at Python/ceval.c:4159

20 0x00007ffff7a0bd55 in _PyFunction_FastCallDict (func=func@entry=0x7fff95524510, args=args@entry=0x7fffffffb520, nargs=4, kwargs=kwargs@entry=0x7fff95507bd0) at Python/ceval.c:5063

21 0x00007ffff7910abe in _PyObject_FastCallDict (func=func@entry=0x7fff95524510, args=args@entry=0x7fffffffb520, nargs=nargs@entry=4, kwargs=kwargs@entry=0x7fff95507bd0) at Objects/abstract.c:2310

22 0x00007ffff7910bae in _PyObject_Call_Prepend (func=0x7fff95524510, obj=0x7fff954dd400, args=0x7fff95476f78, kwargs=0x7fff95507bd0) at Objects/abstract.c:2373

23 0x00007ffff791086a in PyObject_Call (func=0x7fffe48a2808, args=, kwargs=) at Objects/abstract.c:2261

24 0x00007ffff7a0808b in do_call_core (kwdict=0x7fff95507bd0, callargs=0x7fff95476f78, func=0x7fffe48a2808) at Python/ceval.c:5099

25 _PyEval_EvalFrameDefault (f=, throwflag=) at Python/ceval.c:3397

26 0x00007ffff7a0309e in _PyEval_EvalCodeWithName (_co=_co@entry=0x7fffe217f9c0, globals=globals@entry=0x7fffe217c0d8, locals=locals@entry=0x0, args=args@entry=0x7fffffffb920, argcount=argcount@entry=4,

kwnames=kwnames@entry=0x0, kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x7ffff7f96170, qualname=0x7fffe2180470)
at Python/ceval.c:4159

27 0x00007ffff7a0bd55 in _PyFunction_FastCallDict (func=func@entry=0x7fff977bbf28, args=args@entry=0x7fffffffb920, nargs=4, kwargs=kwargs@entry=0x0) at Python/ceval.c:5063

28 0x00007ffff7910abe in _PyObject_FastCallDict (func=func@entry=0x7fff977bbf28, args=args@entry=0x7fffffffb920, nargs=nargs@entry=4, kwargs=kwargs@entry=0x0) at Objects/abstract.c:2310

29 0x00007ffff7910bae in _PyObject_Call_Prepend (func=0x7fff977bbf28, obj=0x7fff954dd400, args=0x7fff953c8678, kwargs=0x0) at Objects/abstract.c:2373

30 0x00007ffff791086a in PyObject_Call (func=0x7ffff7ed3fc8, args=, kwargs=) at Objects/abstract.c:2261

31 0x00007ffff7986f11 in slot_tp_call (self=self@entry=0x7fff954dd400, args=args@entry=0x7fff953c8678, kwds=kwds@entry=0x0) at Objects/typeobject.c:6194

32 0x00007ffff79109cb in _PyObject_FastCallDict (func=0x7fff954dd400, args=, nargs=, kwargs=0x0) at Objects/abstract.c:2331

33 0x00007ffff7a03208 in call_function (pp_stack=pp_stack@entry=0x7fffffffbbc0, oparg=oparg@entry=3, kwnames=kwnames@entry=0x0) at Python/ceval.c:4854

34 0x00007ffff7a0771c in _PyEval_EvalFrameDefault (f=, throwflag=) at Python/ceval.c:3328

35 0x00007ffff7a02730 in _PyFunction_FastCall (co=, args=, nargs=4, globals=) at Python/ceval.c:4912

36 0x00007ffff7a03654 in fast_function (kwnames=0x0, nargs=, stack=, func=0x7fff955248c8) at Python/ceval.c:4947

37 call_function (pp_stack=pp_stack@entry=0x7fffffffbe00, oparg=oparg@entry=3, kwnames=kwnames@entry=0x0) at Python/ceval.c:4851

38 0x00007ffff7a0771c in _PyEval_EvalFrameDefault (f=, throwflag=) at Python/ceval.c:3328

39 0x00007ffff7a0309e in _PyEval_EvalCodeWithName (_co=_co@entry=0x7fff9547b1e0, globals=globals@entry=0x7fff9546b480, locals=locals@entry=0x0, args=args@entry=0x7fffffffc090, argcount=argcount@entry=4,

kwnames=kwnames@entry=0x7ffff7f93060, kwargs=kwargs@entry=0x7ffff7f93068, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=0x7fff95477808, defcount=1, kwdefs=0x0, closure=0x0, name=0x7fffd0687a78, 
qualname=0x7fff9548ba98) at Python/ceval.c:4159

40 0x00007ffff7a0bd55 in _PyFunction_FastCallDict (func=func@entry=0x7fff955247b8, args=args@entry=0x7fffffffc090, nargs=4, kwargs=kwargs@entry=0x7fff95507d38) at Python/ceval.c:5063

41 0x00007ffff7910abe in _PyObject_FastCallDict (func=func@entry=0x7fff955247b8, args=args@entry=0x7fffffffc090, nargs=nargs@entry=4, kwargs=kwargs@entry=0x7fff95507d38) at Objects/abstract.c:2310

42 0x00007ffff7910bae in _PyObject_Call_Prepend (func=0x7fff955247b8, obj=0x7fff954bcb00, args=0x7fff9555ce10, kwargs=0x7fff95507d38) at Objects/abstract.c:2373

43 0x00007ffff791086a in PyObject_Call (func=0x7ffff7f859c8, args=, kwargs=) at Objects/abstract.c:2261

44 0x00007ffff7a0808b in do_call_core (kwdict=0x7fff95507d38, callargs=0x7fff9555ce10, func=0x7ffff7f859c8) at Python/ceval.c:5099

---Type to continue, or q to quit---

45 _PyEval_EvalFrameDefault (f=, throwflag=) at Python/ceval.c:3397

46 0x00007ffff7a0309e in _PyEval_EvalCodeWithName (_co=_co@entry=0x7fffe217f9c0, globals=globals@entry=0x7fffe217c0d8, locals=locals@entry=0x0, args=args@entry=0x7fffffffc490, argcount=argcount@entry=4,

kwnames=kwnames@entry=0x0, kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x7ffff7f96170, qualname=0x7fffe2180470)
at Python/ceval.c:4159

47 0x00007ffff7a0bd55 in _PyFunction_FastCallDict (func=func@entry=0x7fff977bbf28, args=args@entry=0x7fffffffc490, nargs=4, kwargs=kwargs@entry=0x0) at Python/ceval.c:5063

48 0x00007ffff7910abe in _PyObject_FastCallDict (func=func@entry=0x7fff977bbf28, args=args@entry=0x7fffffffc490, nargs=nargs@entry=4, kwargs=kwargs@entry=0x0) at Objects/abstract.c:2310

49 0x00007ffff7910bae in _PyObject_Call_Prepend (func=0x7fff977bbf28, obj=0x7fff954bcb00, args=0x7fff95554f30, kwargs=0x0) at Objects/abstract.c:2373

50 0x00007ffff791086a in PyObject_Call (func=0x7ffff7f85988, args=, kwargs=) at Objects/abstract.c:2261

51 0x00007ffff7986f11 in slot_tp_call (self=self@entry=0x7fff954bcb00, args=args@entry=0x7fff95554f30, kwds=kwds@entry=0x0) at Objects/typeobject.c:6194

52 0x00007ffff79109cb in _PyObject_FastCallDict (func=0x7fff954bcb00, args=, nargs=, kwargs=0x0) at Objects/abstract.c:2331

53 0x00007ffff7a03208 in call_function (pp_stack=pp_stack@entry=0x7fffffffc730, oparg=oparg@entry=3, kwnames=kwnames@entry=0x0) at Python/ceval.c:4854

54 0x00007ffff7a0771c in _PyEval_EvalFrameDefault (f=, throwflag=) at Python/ceval.c:3328

55 0x00007ffff7a0309e in _PyEval_EvalCodeWithName (_co=_co@entry=0x7fff95ce0300, globals=globals@entry=0x7fff9546f708, locals=locals@entry=0x0, args=args@entry=0x7fffffffc9c0, argcount=argcount@entry=2,

kwnames=kwnames@entry=0x7ffff7f93060, kwargs=kwargs@entry=0x7ffff7f93068, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=0x7fff95cfcfb0, defcount=1, kwdefs=0x0, closure=0x0, name=0x7fffd0687a78, 
qualname=0x7fff9546b780) at Python/ceval.c:4159

56 0x00007ffff7a0bd55 in _PyFunction_FastCallDict (func=func@entry=0x7fff9553bea0, args=args@entry=0x7fffffffc9c0, nargs=2, kwargs=kwargs@entry=0x7fff95507c60) at Python/ceval.c:5063

57 0x00007ffff7910abe in _PyObject_FastCallDict (func=func@entry=0x7fff9553bea0, args=args@entry=0x7fffffffc9c0, nargs=nargs@entry=2, kwargs=kwargs@entry=0x7fff95507c60) at Objects/abstract.c:2310

58 0x00007ffff7910bae in _PyObject_Call_Prepend (func=0x7fff9553bea0, obj=0x7fff955829e8, args=0x7fff9550b160, kwargs=0x7fff95507c60) at Objects/abstract.c:2373

59 0x00007ffff791086a in PyObject_Call (func=0x7ffff7ea7908, args=, kwargs=) at Objects/abstract.c:2261

60 0x00007ffff7a0808b in do_call_core (kwdict=0x7fff95507c60, callargs=0x7fff9550b160, func=0x7ffff7ea7908) at Python/ceval.c:5099

61 _PyEval_EvalFrameDefault (f=, throwflag=) at Python/ceval.c:3397

62 0x00007ffff7a0309e in _PyEval_EvalCodeWithName (_co=_co@entry=0x7fffe217f9c0, globals=globals@entry=0x7fffe217c0d8, locals=locals@entry=0x0, args=args@entry=0x7fffffffcdc0, argcount=argcount@entry=2,

kwnames=kwnames@entry=0x0, kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x7ffff7f96170, qualname=0x7fffe2180470)
at Python/ceval.c:4159

63 0x00007ffff7a0bd55 in _PyFunction_FastCallDict (func=func@entry=0x7fff977bbf28, args=args@entry=0x7fffffffcdc0, nargs=2, kwargs=kwargs@entry=0x0) at Python/ceval.c:5063

64 0x00007ffff7910abe in _PyObject_FastCallDict (func=func@entry=0x7fff977bbf28, args=args@entry=0x7fffffffcdc0, nargs=nargs@entry=2, kwargs=kwargs@entry=0x0) at Objects/abstract.c:2310

65 0x00007ffff7910bae in _PyObject_Call_Prepend (func=0x7fff977bbf28, obj=0x7fff955829e8, args=0x7fff955824e0, kwargs=0x0) at Objects/abstract.c:2373

66 0x00007ffff791086a in PyObject_Call (func=0x7ffff7ee7588, args=, kwargs=) at Objects/abstract.c:2261

67 0x00007ffff7986f11 in slot_tp_call (self=self@entry=0x7fff955829e8, args=args@entry=0x7fff955824e0, kwds=kwds@entry=0x0) at Objects/typeobject.c:6194

68 0x00007ffff79109cb in _PyObject_FastCallDict (func=0x7fff955829e8, args=, nargs=, kwargs=0x0) at Objects/abstract.c:2331

69 0x00007ffff7a03208 in call_function (pp_stack=pp_stack@entry=0x7fffffffd060, oparg=oparg@entry=1, kwnames=kwnames@entry=0x0) at Python/ceval.c:4854

70 0x00007ffff7a0771c in _PyEval_EvalFrameDefault (f=, throwflag=) at Python/ceval.c:3328

71 0x00007ffff7a0309e in _PyEval_EvalCodeWithName (_co=0x7fffd05f6270, globals=globals@entry=0x7fffd0658d38, locals=locals@entry=0x0, args=, argcount=2, kwnames=0x0, kwargs=0x7fff954e8b30,

kwcount=0, kwstep=kwstep@entry=1, defs=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=0x0, name=name@entry=0x7fffd05d86a8, qualname=0x7fffd0657da0) at Python/ceval.c:4159

72 0x00007ffff7a033b2 in fast_function (kwnames=0x0, nargs=, stack=, func=0x7fff95556400) at Python/ceval.c:4971

73 call_function (pp_stack=pp_stack@entry=0x7fffffffd310, oparg=oparg@entry=1, kwnames=kwnames@entry=0x0) at Python/ceval.c:4851

74 0x00007ffff7a0771c in _PyEval_EvalFrameDefault (f=, throwflag=) at Python/ceval.c:3328

75 0x00007ffff7a02730 in _PyFunction_FastCall (co=, args=, nargs=2, globals=) at Python/ceval.c:4912

76 0x00007ffff7a03654 in fast_function (kwnames=0x0, nargs=, stack=, func=0x7fff95556378) at Python/ceval.c:4947

77 call_function (pp_stack=pp_stack@entry=0x7fffffffd550, oparg=oparg@entry=1, kwnames=kwnames@entry=0x0) at Python/ceval.c:4851

78 0x00007ffff7a0771c in _PyEval_EvalFrameDefault (f=, throwflag=) at Python/ceval.c:3328

79 0x00007ffff7a02730 in _PyFunction_FastCall (co=, args=, nargs=0, globals=) at Python/ceval.c:4912

80 0x00007ffff7a03654 in fast_function (kwnames=0x0, nargs=, stack=, func=0x7ffff7f33e18) at Python/ceval.c:4947

81 call_function (pp_stack=pp_stack@entry=0x7fffffffd790, oparg=oparg@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:4851

82 0x00007ffff7a0771c in _PyEval_EvalFrameDefault (f=, throwflag=) at Python/ceval.c:3328

83 0x00007ffff7a0309e in _PyEval_EvalCodeWithName (_co=_co@entry=0x7ffff0429a50, globals=globals@entry=0x7ffff7f4a1b0, locals=locals@entry=0x7ffff7f4a1b0, args=args@entry=0x0, argcount=argcount@entry=0,

kwnames=kwnames@entry=0x0, kwargs=kwargs@entry=0x0, kwcount=kwcount@entry=0, kwstep=kwstep@entry=2, defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, 
name=name@entry=0x0, qualname=qualname@entry=0x0) at Python/ceval.c:4159

84 0x00007ffff7a036cd in PyEval_EvalCodeEx (_co=_co@entry=0x7ffff0429a50, globals=globals@entry=0x7ffff7f4a1b0, locals=locals@entry=0x7ffff7f4a1b0, args=args@entry=0x0, argcount=argcount@entry=0,

kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0) at Python/ceval.c:4180

85 0x00007ffff7a0371b in PyEval_EvalCode (co=co@entry=0x7ffff0429a50, globals=globals@entry=0x7ffff7f4a1b0, locals=locals@entry=0x7ffff7f4a1b0) at Python/ceval.c:731

86 0x00007ffff7a3f0a2 in run_mod (arena=0x7ffff7f65288, flags=0x7fffffffdaa0, locals=0x7ffff7f4a1b0, globals=0x7ffff7f4a1b0, filename=0x7ffff01c98b0, mod=0x6bd8f8) at Python/pythonrun.c:1025

87 PyRun_FileExFlags (fp=fp@entry=0x691c50, filename_str=filename_str@entry=0x7ffff0464d10 "webcam.py", start=start@entry=257, globals=globals@entry=0x7ffff7f4a1b0, locals=locals@entry=0x7ffff7f4a1b0,

closeit=closeit@entry=1, flags=flags@entry=0x7fffffffdaa0) at Python/pythonrun.c:978

88 0x00007ffff7a3f207 in PyRun_SimpleFileExFlags (fp=fp@entry=0x691c50, filename=, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffdaa0) at Python/pythonrun.c:420

---Type to continue, or q to quit---

89 0x00007ffff7a3f6a3 in PyRun_AnyFileExFlags (fp=fp@entry=0x691c50, filename=, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffdaa0) at Python/pythonrun.c:81

90 0x00007ffff7a5b6fd in run_file (p_cf=0x7fffffffdaa0, filename=0x603330 L"webcam.py", fp=0x691c50) at Modules/main.c:340

91 Py_Main (argc=argc@entry=2, argv=argv@entry=0x602010) at Modules/main.c:810

92 0x0000000000400bbc in main (argc=2, argv=) at ./Programs/python.c:69

(gdb)

And Yes! Segmentation fault (core dumped) happens even when I run on the CPU (python webcam.py --min-image-size 300 MODEL.DEVICE cpu).

[fmassa]

Thanks for the stack trace!

This is a known issue with gcc < 4.9, see https://github.com/pytorch/pytorch/issues/6987.

While compiling the extensions you probably saw a warning like

Your compiler (g++ 4.8) may be ABI-incompatible with PyTorch!
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.

See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
for instructions on how to install GCC 4.9 or higher.

The solution is to upgrade to gcc 4.9 or higher following the instructions in https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6

I'm closing the issue, and I'll be adding a note on the TROUBLESHOOTING section mentioning this issue.

Let us know if after updating gcc and recompiling the library (after rm -rf build/ folder) you still have issues.

[fmassa]

I've just improved the README with instructions on how to address this issue in https://github.com/facebookresearch/maskrcnn-benchmark/pull/38
Please let me know if it doesn't help in your case.

[youngkyoonjang]

@

I've just improved the README with instructions on how to address this issue in #38
Please let me know if it doesn't help in your case.

Thanks. @fmassa Now it is running perfect!

[Exception-star]

Thanks for the stack trace!

This is a known issue with gcc < 4.9, see pytorch/pytorch#6987.

While compiling the extensions you probably saw a warning like

Your compiler (g++ 4.8) may be ABI-incompatible with PyTorch!
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.

See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
for instructions on how to install GCC 4.9 or higher.

The solution is to upgrade to gcc 4.9 or higher following the instructions in https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6

I'm closing the issue, and I'll be adding a note on the TROUBLESHOOTING section mentioning this issue.

Let us know if after updating gcc and recompiling the library (after rm -rf build/ folder) you still have issues.

0 0x00007ffff7de6d56 in _dl_relocate_object () from /lib64/ld-linux-x86-64.so.2

1 0x00007ffff7def7ac in dl_open_worker () from /lib64/ld-linux-x86-64.so.2

2 0x00007ffff7dea914 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2

3 0x00007ffff7deeccb in _dl_open () from /lib64/ld-linux-x86-64.so.2

4 0x00007ffff792f082 in do_dlopen () from /lib64/libc.so.6

5 0x00007ffff7dea914 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2

6 0x00007ffff792f142 in __libc_dlopen_mode () from /lib64/libc.so.6

7 0x00007ffff7bcee53 in pthread_cancel_init () from /lib64/libpthread.so.0

8 0x00007ffff7bcf01c in _Unwind_ForcedUnwind () from /lib64/libpthread.so.0

9 0x00007ffff7bcd332 in __pthread_unwind () from /lib64/libpthread.so.0

10 0x00007ffff7bc7e77 in pthread_exit () from /lib64/libpthread.so.0

11 0x000055555571d6f9 in PyThread_exit_thread ()

at /tmp/build/80754af9/python_1546130271559/work/Python/thread_pthread.h:300

12 0x0000555555763d93 in t_bootstrap () at /tmp/build/80754af9/python_1546130271559/work/Modules/_threadmodule.c:1030

13 0x00005555557182a8 in pythread_wrapper (arg=)

at /tmp/build/80754af9/python_1546130271559/work/Python/thread_pthread.h:205

14 0x00007ffff7bc6e25 in start_thread () from /lib64/libpthread.so.0

15 0x00007ffff78f0bad in clone () from /lib64/libc.so.6