Opencv_contrib: SuperpixelLSC.enforceLabelConnectivity() crashes

[mminervini]

System information (version)

• OpenCV => 3.2.0
• Python => 3.5.2
• Operating System / Platform => Ubuntu 16.10 (Linux 4.8.0-39-generic x86_64)
• Compiler => gcc version 6.2.0

Detailed description

The Python interpreter crashes while executing the function enforceLabelConnectivity() of the SuperpixelLSC class in the ximgproc module.

The function does not crash systematically all the times, yet very often (more than half of the times I try to use it).

I tired varying the value of the _min_element_size_ parameter with respect to the default value, but I still experience the crashes.

Steps to reproduce

I simply create a SuperpixelLSC on a 5 megapixel image, iterate to get the superpixel segmentation (e.g., in the order of 2k or 3k superpixels are found), and then apply enforceLabelConnectivity.
Here is a minimal working example to reproduce the bug and the test image I used.

import cv2
image = cv2.cvtColor(cv2.imread('test_image.jpg', cv2.IMREAD_UNCHANGED), cv2.COLOR_BGR2RGB)
superpixelLSC = cv2.ximgproc.createSuperpixelLSC(
        image,
        region_size=50,
        ratio=0.075)
superpixelLSC.iterate()
superpixelLSC.enforceLabelConnectivity(min_element_size=25)

When executing enforceLabelConnectivity(), Python crashes with the following errors:

*** Error in `/usr/bin/python3': free(): invalid next size (normal): 0x00005595339ee680 ***
======= Backtrace: =========
/lib/x86_64‑linux‑gnu/libc.so.6(+0x790cb)[0x7f2434d920cb]
/lib/x86_64‑linux‑gnu/libc.so.6(+0x8275a)[0x7f2434d9b75a]
/lib/x86_64‑linux‑gnu/libc.so.6(cfree+0x4c)[0x7f2434d9f18c]
/usr/local/lib/libopencv_ximgproc.so.3.2(+0xa6c4f)[0x7f23bab40c4f]
/usr/local/lib/libopencv_ximgproc.so.3.2(+0xa6edb)[0x7f23bab40edb]
/usr/local/lib/python3.5/dist‑packages/cv2.cpythonm‑x86_64‑linux‑gnu.so(+0x6bc49)[0x7f23c2377c49]
/usr/bin/python3(PyCFunction_Call+0x77)[0x55952f59a127]
/usr/bin/python3(PyEval_EvalFrameEx+0x5a0b)[0x55952f5ee99b]
/usr/bin/python3(+0x1763f6)[0x55952f5f23f6]
/usr/bin/python3(PyEval_EvalFrameEx+0x513c)[0x55952f5ee0cc]
/usr/bin/python3(+0x1763f6)[0x55952f5f23f6]
/usr/bin/python3(PyEval_EvalCode+0x1f)[0x55952f5f30af]
/usr/bin/python3(+0x180bd5)[0x55952f5fcbd5]
/usr/bin/python3(PyCFunction_Call+0x4f)[0x55952f59a0ff]
/usr/bin/python3(PyEval_EvalFrameEx+0x492a)[0x55952f5ed8ba]
/usr/bin/python3(PyEval_EvalFrameEx+0x4b7f)[0x55952f5edb0f]
/usr/bin/python3(+0x1763f6)[0x55952f5f23f6]
/usr/bin/python3(PyEval_EvalFrameEx+0x513c)[0x55952f5ee0cc]
/usr/bin/python3(+0x1763f6)[0x55952f5f23f6]
/usr/bin/python3(PyEval_EvalCode+0x1f)[0x55952f5f30af]
/usr/bin/python3(+0x180bd5)[0x55952f5fcbd5]
/usr/bin/python3(PyCFunction_Call+0x4f)[0x55952f59a0ff]
/usr/bin/python3(PyEval_EvalFrameEx+0x492a)[0x55952f5ed8ba]
/usr/bin/python3(+0x1763f6)[0x55952f5f23f6]
/usr/bin/python3(PyEval_EvalFrameEx+0x513c)[0x55952f5ee0cc]
/usr/bin/python3(+0x1763f6)[0x55952f5f23f6]
/usr/bin/python3(PyEval_EvalFrameEx+0x513c)[0x55952f5ee0cc]
/usr/bin/python3(PyEval_EvalCodeEx+0x867)[0x55952f5f3927]
/usr/bin/python3(+0x12039f)[0x55952f59c39f]
/usr/bin/python3(PyObject_Call+0x47)[0x55952f6822f7]
/usr/bin/python3(PyEval_EvalFrameEx+0x23c6)[0x55952f5eb356]
/usr/bin/python3(+0x17692f)[0x55952f5f292f]
/usr/bin/python3(PyEval_EvalFrameEx+0x555d)[0x55952f5ee4ed]
/usr/bin/python3(+0x1763f6)[0x55952f5f23f6]
/usr/bin/python3(PyEval_EvalFrameEx+0x513c)[0x55952f5ee0cc]
/usr/bin/python3(PyEval_EvalFrameEx+0x4b7f)[0x55952f5edb0f]
/usr/bin/python3(PyEval_EvalFrameEx+0x4b7f)[0x55952f5edb0f]
/usr/bin/python3(PyEval_EvalCodeEx+0x867)[0x55952f5f3927]
/usr/bin/python3(+0x12039f)[0x55952f59c39f]
/usr/bin/python3(PyObject_Call+0x47)[0x55952f6822f7]
/usr/bin/python3(PyEval_EvalFrameEx+0x23c6)[0x55952f5eb356]
/usr/bin/python3(PyEval_EvalCodeEx+0x867)[0x55952f5f3927]
/usr/bin/python3(+0x12039f)[0x55952f59c39f]
/usr/bin/python3(PyObject_Call+0x47)[0x55952f6822f7]
/usr/bin/python3(PyEval_EvalFrameEx+0x23c6)[0x55952f5eb356]
/usr/bin/python3(+0x1763f6)[0x55952f5f23f6]
/usr/bin/python3(PyEval_EvalFrameEx+0x555d)[0x55952f5ee4ed]
/usr/bin/python3(PyEval_EvalFrameEx+0x4b7f)[0x55952f5edb0f]
/usr/bin/python3(+0x17692f)[0x55952f5f292f]
/usr/bin/python3(PyEval_EvalFrameEx+0x513c)[0x55952f5ee0cc]
/usr/bin/python3(PyEval_EvalFrameEx+0x4b7f)[0x55952f5edb0f]
/usr/bin/python3(PyEval_EvalCodeEx+0x95b)[0x55952f5f3a1b]
/usr/bin/python3(+0x1201b3)[0x55952f59c1b3]
/usr/bin/python3(PyObject_Call+0x47)[0x55952f6822f7]
/usr/bin/python3(PyEval_CallObjectWithKeywords+0x30)[0x55952f63c4b0]
/usr/lib/python3/dist‑packages/PyQt5/QtCore.cpythonm‑x86_64‑linux‑gnu.so(+0x1de888)[0x7f23f7af2888]
/usr/lib/python3/dist‑packages/PyQt5/QtCore.cpythonm‑x86_64‑linux‑gnu.so(+0x1decd0)[0x7f23f7af2cd0]
/usr/lib/python3/dist‑packages/PyQt5/QtCore.cpythonm‑x86_64‑linux‑gnu.so(+0x1e1b40)[0x7f23f7af5b40]
/usr/lib/python3/dist‑packages/PyQt5/QtCore.cpythonm‑x86_64‑linux‑gnu.so(+0x1e2567)[0x7f23f7af6567]
/usr/lib/x86_64‑linux‑gnu/libQt5Core.so.5(_ZN11QMetaObject8activateEP7QObjectiiPPv+0x687)[0x7f23f76ee877]
/usr/lib/x86_64‑linux‑gnu/libQt5Core.so.5(_ZN15QSocketNotifier9activatedEiNS_14QPrivateSignalE+0x3e)[0x7f23f776b2fe]
/usr/lib/x86_64‑linux‑gnu/libQt5Core.so.5(_ZN15QSocketNotifier5eventEP6QEvent+0x52)[0x7f23f76fad12]
/usr/lib/python3/dist‑packages/PyQt5/QtCore.cpythonm‑x86_64‑linux‑gnu.so(+0x17389b)[0x7f23f7a8789b]
======= Memory map: ========
55952f47c000f869000 r‑xp 00000000 08:02 524976 /usr/bin/python3.5
55952fa68000fa6b000 r‑‑p 003ec000 08:02 524976 /usr/bin/python3.5
55952fa6b000fb01000 rw‑p 003ef000 08:02 524976 /usr/bin/python3.5
55952fb01000fb32000 rw‑p 00000000 00:00 0 
559530a31000ad16000 rw‑p 00000000 00:00 0 [heap]

And also:

*** Error in `/usr/bin/python3': double free or corruption (out): 0x0000564a2f3ef320 ***
======= Backtrace: =========
/lib/x86_64‑linux‑gnu/libc.so.6(+0x790cb)[0x7f8a6f65f0cb]
/lib/x86_64‑linux‑gnu/libc.so.6(+0x8275a)[0x7f8a6f66875a]
/lib/x86_64‑linux‑gnu/libc.so.6(cfree+0x4c)[0x7f8a6f66c18c]
/usr/local/lib/libopencv_tracking.so.3.2(_ZNSt6vectorIS_IfSaIfEESaIS1_EED1Ev+0x25)[0x7f8a04ded515]
/usr/local/lib/libopencv_ximgproc.so.3.2(+0xa5dd0)[0x7f89fef5fdd0]
/usr/local/lib/libopencv_ximgproc.so.3.2(+0xa6edb)[0x7f89fef60edb]
/usr/local/lib/python3.5/dist‑packages/cv2.cpythonm‑x86_64‑linux‑gnu.so(+0x6bc49)[0x7f8a0693cc49]
/usr/bin/python3(PyCFunction_Call+0x77)[0x564a2855d127]
/usr/bin/python3(PyEval_EvalFrameEx+0x5a0b)[0x564a285b199b]
/usr/bin/python3(+0x1763f6)[0x564a285b53f6]
/usr/bin/python3(PyEval_EvalFrameEx+0x513c)[0x564a285b10cc]
/usr/bin/python3(+0x1763f6)[0x564a285b53f6]
/usr/bin/python3(PyEval_EvalCode+0x1f)[0x564a285b60af]
/usr/bin/python3(+0x180bd5)[0x564a285bfbd5]
/usr/bin/python3(PyCFunction_Call+0x4f)[0x564a2855d0ff]
/usr/bin/python3(PyEval_EvalFrameEx+0x492a)[0x564a285b08ba]
/usr/bin/python3(PyEval_EvalFrameEx+0x4b7f)[0x564a285b0b0f]
/usr/bin/python3(+0x1763f6)[0x564a285b53f6]
/usr/bin/python3(PyEval_EvalFrameEx+0x513c)[0x564a285b10cc]
/usr/bin/python3(+0x1763f6)[0x564a285b53f6]
/usr/bin/python3(PyEval_EvalCode+0x1f)[0x564a285b60af]
/usr/bin/python3(+0x180bd5)[0x564a285bfbd5]
/usr/bin/python3(PyCFunction_Call+0x4f)[0x564a2855d0ff]
/usr/bin/python3(PyEval_EvalFrameEx+0x492a)[0x564a285b08ba]
/usr/bin/python3(+0x1763f6)[0x564a285b53f6]
/usr/bin/python3(PyEval_EvalFrameEx+0x513c)[0x564a285b10cc]
/usr/bin/python3(+0x1763f6)[0x564a285b53f6]
/usr/bin/python3(PyEval_EvalFrameEx+0x513c)[0x564a285b10cc]
/usr/bin/python3(PyEval_EvalCodeEx+0x867)[0x564a285b6927]
/usr/bin/python3(+0x12039f)[0x564a2855f39f]
/usr/bin/python3(PyObject_Call+0x47)[0x564a286452f7]
/usr/bin/python3(PyEval_EvalFrameEx+0x23c6)[0x564a285ae356]
/usr/bin/python3(+0x17692f)[0x564a285b592f]
/usr/bin/python3(PyEval_EvalFrameEx+0x555d)[0x564a285b14ed]
/usr/bin/python3(+0x1763f6)[0x564a285b53f6]
/usr/bin/python3(PyEval_EvalFrameEx+0x513c)[0x564a285b10cc]
/usr/bin/python3(PyEval_EvalFrameEx+0x4b7f)[0x564a285b0b0f]
/usr/bin/python3(PyEval_EvalFrameEx+0x4b7f)[0x564a285b0b0f]
/usr/bin/python3(PyEval_EvalCodeEx+0x867)[0x564a285b6927]
/usr/bin/python3(+0x12039f)[0x564a2855f39f]
/usr/bin/python3(PyObject_Call+0x47)[0x564a286452f7]
/usr/bin/python3(PyEval_EvalFrameEx+0x23c6)[0x564a285ae356]
/usr/bin/python3(PyEval_EvalCodeEx+0x867)[0x564a285b6927]
/usr/bin/python3(+0x12039f)[0x564a2855f39f]
/usr/bin/python3(PyObject_Call+0x47)[0x564a286452f7]
/usr/bin/python3(PyEval_EvalFrameEx+0x23c6)[0x564a285ae356]
/usr/bin/python3(+0x1763f6)[0x564a285b53f6]
/usr/bin/python3(PyEval_EvalFrameEx+0x555d)[0x564a285b14ed]
/usr/bin/python3(PyEval_EvalFrameEx+0x4b7f)[0x564a285b0b0f]
/usr/bin/python3(+0x17692f)[0x564a285b592f]
/usr/bin/python3(PyEval_EvalFrameEx+0x513c)[0x564a285b10cc]
/usr/bin/python3(PyEval_EvalFrameEx+0x4b7f)[0x564a285b0b0f]
/usr/bin/python3(PyEval_EvalCodeEx+0x95b)[0x564a285b6a1b]
/usr/bin/python3(+0x1201b3)[0x564a2855f1b3]
/usr/bin/python3(PyObject_Call+0x47)[0x564a286452f7]
/usr/bin/python3(PyEval_CallObjectWithKeywords+0x30)[0x564a285ff4b0]
/usr/lib/python3/dist‑packages/PyQt5/QtCore.cpythonm‑x86_64‑linux‑gnu.so(+0x1de888)[0x7f8a323b6888]
/usr/lib/python3/dist‑packages/PyQt5/QtCore.cpythonm‑x86_64‑linux‑gnu.so(+0x1decd0)[0x7f8a323b6cd0]
/usr/lib/python3/dist‑packages/PyQt5/QtCore.cpythonm‑x86_64‑linux‑gnu.so(+0x1e1b40)[0x7f8a323b9b40]
/usr/lib/python3/dist‑packages/PyQt5/QtCore.cpythonm‑x86_64‑linux‑gnu.so(+0x1e2567)[0x7f8a323ba567]
/usr/lib/x86_64‑linux‑gnu/libQt5Core.so.5(_ZN11QMetaObject8activateEP7QObjectiiPPv+0x687)[0x7f8a31fb2877]
/usr/lib/x86_64‑linux‑gnu/libQt5Core.so.5(_ZN15QSocketNotifier9activatedEiNS_14QPrivateSignalE+0x3e)[0x7f8a3202f2fe]
/usr/lib/x86_64‑linux‑gnu/libQt5Core.so.5(_ZN15QSocketNotifier5eventEP6QEvent+0x52)[0x7f8a31fbed12]
======= Memory map: ========
564a2843f000a2882c000 r‑xp 00000000 08:02 524976 /usr/bin/python3.5
564a28a2b000a28a2e000 r‑‑p 003ec000 08:02 524976 /usr/bin/python3.5
564a28a2e000a28ac4000 rw‑p 003ef000 08:02 524976 /usr/bin/python3.5
564a28ac4000a28af5000 rw‑p 00000000 00:00 0 
564a2a192000a34d83000 rw‑p 00000000 00:00 0 [heap]

The entire error log is available in attachment.
Please let me know if any further details are required to reproduce the bug.

enforceLabelConnectivity_crash_dump.txt

[sovrasov]

I can't reproduce the issue using a 6 MPix picture. Please, attach the image.

[mminervini]

Hi @sovrasov, thank you for your quick reaction.
I updated the original post with a test image and a code snippet, which, as I realized, is resulting systematically in a crash.
I hope this helps figure out the problem. Let me know if you need more details on my system setup or library versions.

[cbalint13]

@mminervini ,

• I am aware of the bug. On certain superpixels results it seems to fail to merge. I am able to reproduce with tiny images too, it is not size related. Will try fix but solution will be not a easy one, i will try rewrite the whole while() tree traversal. Seems that author original one suffers the same issue.

[jexner]

I encountered this problem on fairly small images (300x300). #1076 fixes this for me.

Valgrind pointed me to the erase in the while loop. When erasing an element from a vector, iterators pointing "behind" the erased element are invalidated.

Invalidates iterators and references at or after the point of the erase, including the end() iterator.

(See: http://en.cppreference.com/w/cpp/container/vector/erase)

So, I guess in some cases when S was preceding Stemp, and when Stemp was about to be erased, the crash occurred.

Additionally, valgrind complained about conditional jumps depending on uninitialized memory originating from https://github.com/opencv/opencv_contrib/blob/5d73f99753dd8a1dd33b16b3215f6e23b9523626/modules/ximgproc/src/lsc.cpp#L1168. But maybe valgrind just isn't capable of dealing with the parallel_for_. Explicitly, setting m_W to 0.0f fixes this of course.

[cbalint13]

• Made #1088 , that fix your python sample too, i believe is the most related to the issue.
• Still not sure about if valgrind complains or not anymore at that Mat init thing (will be in par_for_).
• Suggest use ( cv2.COLOR_BGR2Lab ) color space at that RGB conversion in your sample for a really good quality segmentation.

[jexner]

@cbalint13

Still not sure about if valgrind complains or not anymore at that Mat init thing (will be in par_for_).

I created a separate issue for that: https://github.com/opencv/opencv_contrib/pull/1078

[jexner]

@cbalint13 I just tried your patch locally. Running my test program standalone does not crash. But I also checked with valgrind and it reports invalid reads and finally crashes with SIGSEGV. Applying https://github.com/jexner/opencv_contrib/commit/5dc7be6a061dc6edb0cedbd347d7d99328e7de4a fixes this for me. Maybe you're able to reproduce this. Here is my code snippet:
```.cpp
cv::Mat failimage = cv::imread("./data/lsc-segfault-combined-map.png", cv::IMREAD_UNCHANGED);
EXPECT_TRUE(!failimage.empty());

// only two channels in the original image
// but when saving to PNG, third channel was added
// remove it here
cv::Mat channels[3];
cv::split(failimage, channels);
cv::merge(channels, 2, failimage);

// failing parameters
int superpixel_size = 10;
float lsc_ratio = 0.075f;
int lsc_iterations = 10;

cv::Ptr lsc = cv::ximgproc::createSuperpixelLSC(failimage, superpixel_size, lsc_ratio);
lsc->iterate(lsc_iterations);
lsc->enforceLabelConnectivity(superpixel_size); // segfaulted in here
```
And here's the failing image: https://cloud.githubusercontent.com/assets/5003917/24369113/3e696772-1323-11e7-9205-260700152fc0.png

[cbalint13]

@jexner,

• Thanks for confirming it with valgrind !

  I wasn't sure. Its clear booth (three) patches are necessary. In stuck with my test case that often keeps Label2=-1 (during L1 distance search nothing is found below threshold) thus further crashing code when accessing arrays based on Label2 index value.