Drake: "Illegal instruction (core dumped)" when running drake_visualizer

[airfield20]

$ bazel build //tools:drake_visualizer
INFO: Analysed target //tools:drake_visualizer (0 packages loaded).
INFO: Found 1 target...
Target //tools:drake_visualizer up-to-date:
bazel-bin/tools/drake_visualizer
INFO: Elapsed time: 1.000s, Critical Path: 0.01s
INFO: Build completed successfully, 1 total action

$ bazel-bin/tools/drake_visualizer
Illegal instruction (core dumped)

I also had to manually install lcm

[jwnimmer-tri]

What OS and version?

When we have seen this before, something on LD_LIBRARY_PATH or PYTHONPATH ends up loading the wrong packages. Does env PYTHONPATH= LD_LIBRARY_PATH= bazel-bin/tools/drake_visualizer get it working?

If not, try editing tools/workspace/drake_visualizer/drake_visualizer_linux.sh (assuming Ubuntu, else there's an apple version with a different name) and change the "exec" at the end to "ldd" and paste that output here.

I also had to manually install lcm

That ... is weird and shouldn't be necessary.

[airfield20]

Ubuntu 16.04

$ bazel-bin/tools/drake_visualizer
linux-vdso.so.1 => (0x00007ffd8abd6000)
libddApp.so => /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/execroot/drake/bazel-out/clang-4.0-linux-opt/bin/tools/drake_visualizer.runfiles/drake/external/drake_visualizer/bin/../lib/libddApp.so (0x00007ffa75f9d000)
libPythonQt.so => /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/execroot/drake/bazel-out/clang-4.0-linux-opt/bin/tools/drake_visualizer.runfiles/drake/external/drake_visualizer/bin/../lib/libPythonQt.so (0x00007ffa74de9000)
libQt5Widgets.so.5 => /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5 (0x00007ffa7475c000)
libQt5Core.so.5 => /usr/lib/x86_64-linux-gnu/libQt5Core.so.5 (0x00007ffa74286000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ffa73f04000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ffa73cee000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffa73924000)
libGLU.so.1 => /usr/lib/x86_64-linux-gnu/libGLU.so.1 (0x00007ffa736b5000)
libGL.so.1 => /usr/lib/x86_64-linux-gnu/mesa/libGL.so.1 (0x00007ffa73442000)
libvtkWrappingPython27Core-8.0.so.1 => external/vtk/lib/libvtkWrappingPython27Core-8.0.so.1 (0x00007ffa7320a000)
libpython2.7.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 (0x00007ffa72c7c000)
libvtkGUISupportQt-8.0.so.1 => external/vtk/lib/libvtkGUISupportQt-8.0.so.1 (0x00007ffa72a35000)
libvtkRenderingOpenGL2-8.0.so.1 => external/vtk/lib/libvtkRenderingOpenGL2-8.0.so.1 (0x00007ffa72652000)
libvtkInteractionWidgets-8.0.so.1 => external/vtk/lib/libvtkInteractionWidgets-8.0.so.1 (0x00007ffa721cb000)
libvtkRenderingAnnotation-8.0.so.1 => external/vtk/lib/libvtkRenderingAnnotation-8.0.so.1 (0x00007ffa71ea1000)
libvtkRenderingFreeType-8.0.so.1 => external/vtk/lib/libvtkRenderingFreeType-8.0.so.1 (0x00007ffa71bf6000)
libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007ffa718e5000)
libQt5Gui.so.5 => /usr/lib/x86_64-linux-gnu/libQt5Gui.so.5 (0x00007ffa7139d000)
libvtkInteractionStyle-8.0.so.1 => external/vtk/lib/libvtkInteractionStyle-8.0.so.1 (0x00007ffa7114d000)
libvtkRenderingCore-8.0.so.1 => external/vtk/lib/libvtkRenderingCore-8.0.so.1 (0x00007ffa70d48000)
libvtkFiltersSources-8.0.so.1 => external/vtk/lib/libvtkFiltersSources-8.0.so.1 (0x00007ffa70aa4000)
libvtkCommonExecutionModel-8.0.so.1 => external/vtk/lib/libvtkCommonExecutionModel-8.0.so.1 (0x00007ffa707f0000)
libvtkCommonDataModel-8.0.so.1 => external/vtk/lib/libvtkCommonDataModel-8.0.so.1 (0x00007ffa70256000)
libvtkCommonCore-8.0.so.1 => external/vtk/lib/libvtkCommonCore-8.0.so.1 (0x00007ffa6fc6c000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffa6f963000)
libQt5Multimedia.so.5 => /usr/lib/x86_64-linux-gnu/libQt5Multimedia.so.5 (0x00007ffa76746000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ffa6f746000)
libgobject-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x00007ffa6f4f3000)
libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007ffa6f1b9000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007ffa6ef9f000)
libicui18n.so.55 => /usr/lib/x86_64-linux-gnu/libicui18n.so.55 (0x00007ffa6eb3d000)
libicuuc.so.55 => /usr/lib/x86_64-linux-gnu/libicuuc.so.55 (0x00007ffa6e7a9000)
libpcre16.so.3 => /usr/lib/x86_64-linux-gnu/libpcre16.so.3 (0x00007ffa6e543000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ffa6e33f000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ffa6e137000)
/lib64/ld-linux-x86-64.so.2 (0x00007ffa76633000)
libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007ffa6df0e000)
libxcb-dri3.so.0 => /usr/lib/x86_64-linux-gnu/libxcb-dri3.so.0 (0x00007ffa6dd0b000)
libxcb-present.so.0 => /usr/lib/x86_64-linux-gnu/libxcb-present.so.0 (0x00007ffa6db08000)
libxcb-sync.so.1 => /usr/lib/x86_64-linux-gnu/libxcb-sync.so.1 (0x00007ffa6d901000)
libxshmfence.so.1 => /usr/lib/x86_64-linux-gnu/libxshmfence.so.1 (0x00007ffa6d6fe000)
libglapi.so.0 => /usr/lib/x86_64-linux-gnu/libglapi.so.0 (0x00007ffa6d4ce000)
libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007ffa6d2bc000)
libXdamage.so.1 => /usr/lib/x86_64-linux-gnu/libXdamage.so.1 (0x00007ffa6d0b9000)
libXfixes.so.3 => /usr/lib/x86_64-linux-gnu/libXfixes.so.3 (0x00007ffa6ceb3000)
libX11-xcb.so.1 => /usr/lib/x86_64-linux-gnu/libX11-xcb.so.1 (0x00007ffa6ccb1000)
libxcb-glx.so.0 => /usr/lib/x86_64-linux-gnu/libxcb-glx.so.0 (0x00007ffa6ca98000)
libxcb-dri2.so.0 => /usr/lib/x86_64-linux-gnu/libxcb-dri2.so.0 (0x00007ffa6c893000)
libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007ffa6c671000)
libXxf86vm.so.1 => /usr/lib/x86_64-linux-gnu/libXxf86vm.so.1 (0x00007ffa6c46b000)
libdrm.so.2 => /usr/lib/x86_64-linux-gnu/libdrm.so.2 (0x00007ffa6c25a000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007ffa6c057000)
libQt5X11Extras.so.5 => /usr/lib/x86_64-linux-gnu/libQt5X11Extras.so.5 (0x00007ffa76738000)
libvtkFiltersExtraction-8.0.so.1 => external/vtk/lib/libvtkFiltersExtraction-8.0.so.1 (0x00007ffa6bd86000)
libvtkglew-8.0.so.1 => external/vtk/lib/libvtkglew-8.0.so.1 (0x00007ffa6bb02000)
libXt.so.6 => /usr/lib/x86_64-linux-gnu/libXt.so.6 (0x00007ffa6b899000)
libvtkCommonSystem-8.0.so.1 => external/vtk/lib/libvtkCommonSystem-8.0.so.1 (0x00007ffa6b683000)
libvtkCommonTransforms-8.0.so.1 => external/vtk/lib/libvtkCommonTransforms-8.0.so.1 (0x00007ffa6b452000)
libvtksys-8.0.so.1 => external/vtk/lib/libvtksys-8.0.so.1 (0x00007ffa6b205000)
libvtkCommonMath-8.0.so.1 => external/vtk/lib/libvtkCommonMath-8.0.so.1 (0x00007ffa6afe1000)
libvtkFiltersHybrid-8.0.so.1 => external/vtk/lib/libvtkFiltersHybrid-8.0.so.1 (0x00007ffa6acfc000)
libvtkFiltersModeling-8.0.so.1 => external/vtk/lib/libvtkFiltersModeling-8.0.so.1 (0x00007ffa6aa85000)
libvtkImagingGeneral-8.0.so.1 => external/vtk/lib/libvtkImagingGeneral-8.0.so.1 (0x00007ffa6a797000)
libvtkRenderingVolume-8.0.so.1 => external/vtk/lib/libvtkRenderingVolume-8.0.so.1 (0x00007ffa6a05b000)
libvtkFiltersGeneral-8.0.so.1 => external/vtk/lib/libvtkFiltersGeneral-8.0.so.1 (0x00007ffa69b58000)
libvtkCommonComputationalGeometry-8.0.so.1 => external/vtk/lib/libvtkCommonComputationalGeometry-8.0.so.1 (0x00007ffa69922000)
libvtkFiltersCore-8.0.so.1 => external/vtk/lib/libvtkFiltersCore-8.0.so.1 (0x00007ffa69285000)
libvtkImagingCore-8.0.so.1 => external/vtk/lib/libvtkImagingCore-8.0.so.1 (0x00007ffa68e0e000)
libfreetype.so.6 => /usr/lib/x86_64-linux-gnu/libfreetype.so.6 (0x00007ffa68b64000)
libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007ffa688f4000)
libpng12.so.0 => /lib/x86_64-linux-gnu/libpng12.so.0 (0x00007ffa686cf000)
libharfbuzz.so.0 => /usr/lib/x86_64-linux-gnu/libharfbuzz.so.0 (0x00007ffa68471000)
libvtkCommonColor-8.0.so.1 => external/vtk/lib/libvtkCommonColor-8.0.so.1 (0x00007ffa68254000)
libvtkFiltersGeometry-8.0.so.1 => external/vtk/lib/libvtkFiltersGeometry-8.0.so.1 (0x00007ffa67fcd000)
libvtkCommonMisc-8.0.so.1 => external/vtk/lib/libvtkCommonMisc-8.0.so.1 (0x00007ffa67db3000)
libQt5Network.so.5 => /usr/lib/x86_64-linux-gnu/libQt5Network.so.5 (0x00007ffa67c59000)
libpulse.so.0 => /usr/lib/x86_64-linux-gnu/libpulse.so.0 (0x00007ffa67a09000)
libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007ffa67801000)
libicudata.so.55 => /usr/lib/x86_64-linux-gnu/libicudata.so.55 (0x00007ffa65d4a000)
libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007ffa65b46000)
libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007ffa65940000)
libvtkFiltersStatistics-8.0.so.1 => external/vtk/lib/libvtkFiltersStatistics-8.0.so.1 (0x00007ffa656ae000)
libSM.so.6 => /usr/lib/x86_64-linux-gnu/libSM.so.6 (0x00007ffa654a6000)
libICE.so.6 => /usr/lib/x86_64-linux-gnu/libICE.so.6 (0x00007ffa6528c000)
libvtkImagingSources-8.0.so.1 => external/vtk/lib/libvtkImagingSources-8.0.so.1 (0x00007ffa65044000)
libgraphite2.so.3 => /usr/lib/x86_64-linux-gnu/libgraphite2.so.3 (0x00007ffa64e1e000)
libproxy.so.1 => /usr/lib/x86_64-linux-gnu/libproxy.so.1 (0x00007ffa64bfd000)
libjson-c.so.2 => /lib/x86_64-linux-gnu/libjson-c.so.2 (0x00007ffa649f2000)
libpulsecommon-8.0.so => /usr/lib/x86_64-linux-gnu/pulseaudio/libpulsecommon-8.0.so (0x00007ffa64777000)
libdbus-1.so.3 => /lib/x86_64-linux-gnu/libdbus-1.so.3 (0x00007ffa6452b000)
libvtkalglib-8.0.so.1 => external/vtk/lib/libvtkalglib-8.0.so.1 (0x00007ffa64306000)
libvtkImagingFourier-8.0.so.1 => external/vtk/lib/libvtkImagingFourier-8.0.so.1 (0x00007ffa640e1000)
libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007ffa63edc000)
libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x00007ffa766a0000)
libwrap.so.0 => /lib/x86_64-linux-gnu/libwrap.so.0 (0x00007ffa63cd2000)
libsndfile.so.1 => /usr/lib/x86_64-linux-gnu/libsndfile.so.1 (0x00007ffa63a69000)
libasyncns.so.0 => /usr/lib/x86_64-linux-gnu/libasyncns.so.0 (0x00007ffa63863000)
libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007ffa63641000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007ffa6341f000)
libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007ffa6313e000)
libnsl.so.1 => /lib/x86_64-linux-gnu/libnsl.so.1 (0x00007ffa62f25000)
libFLAC.so.8 => /usr/lib/x86_64-linux-gnu/libFLAC.so.8 (0x00007ffa62cb0000)
libvorbisenc.so.2 => /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2 (0x00007ffa62a07000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007ffa627ec000)
libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007ffa625d8000)
libogg.so.0 => /usr/lib/x86_64-linux-gnu/libogg.so.0 (0x00007ffa623cf000)
libvorbis.so.0 => /usr/lib/x86_64-linux-gnu/libvorbis.so.0 (0x00007ffa621a3000)

[jwnimmer-tri]

That's identical to mine (which is working). I guess next I would say fire it up in gdb or valgrind and see if the backtrace highlights anything odd. Maybe its trying to initialize a GL context, and something like a NVidia driver needs a reboot or somesuch.

[airfield20]

Its a fresh install apart from installing nvidia drivers

[jwnimmer-tri]

I upgraded latest Ubuntu and the program still works fine for me. If you can get some more information by running under valgrind or gdb (change the exec to do one of those instead) maybe we can help, but at the moment I'm stumped.

[airfield20]

Running with gdb outputs the following:
Starting program: /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/execroot/drake/bazel-out/clang-4.0-linux-opt/bin/tools/drake_visualizer.runfiles/drake/external/drake_visualizer/bin/drake-visualizer
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe15f4700 (LWP 4534)]
[New Thread 0x7fffd79eb700 (LWP 4535)]
[New Thread 0x7fffd71ea700 (LWP 4536)]
[New Thread 0x7fffd69e9700 (LWP 4537)]
[New Thread 0x7fffd583e700 (LWP 4539)]
[New Thread 0x7fffd503d700 (LWP 4540)]
[Thread 0x7fffd583e700 (LWP 4539) exited]

Thread 1 "drake-visualize" received signal SIGILL, Illegal instruction.
0x00007ffff78164e6 in QtTreePropertyBrowserPrivate::init(QWidget*) ()
from /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/external/drake_visualizer/bin/../lib/libddApp.so

[airfield20]

Running with valgrind gives this output:
vex amd64->IR: unhandled instruction bytes: 0xC5 0xF9 0x6F 0x5 0xB2 0x8B 0x37 0x0
vex amd64->IR: REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
vex amd64->IR: VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=NONE
vex amd64->IR: PFX.66=0 PFX.F2=0 PFX.F3=0
==4903== valgrind: Unrecognised instruction at address 0x4f0f4e6.
==4903== at 0x4F0F4E6: QtTreePropertyBrowserPrivate::init(QWidget) (in /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/external/drake_visualizer/lib/libddApp.so)
==4903== by 0x4F0FB1F: QtTreePropertyBrowser::QtTreePropertyBrowser(QWidget) (in /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/external/drake_visualizer/lib/libddApp.so)
==4903== by 0x4ED7C66: ddPropertiesPanel::setBrowserModeToTree() (in /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/external/drake_visualizer/lib/libddApp.so)
==4903== by 0x4ED801D: ddPropertiesPanel::ddPropertiesPanel(QWidget) (in /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/external/drake_visualizer/lib/libddApp.so)
==4903== by 0x4EC32C5: ddPythonQtDecorators::qt_static_metacall(QObject, QMetaObject::Call, int, void) (in /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/external/drake_visualizer/lib/libddApp.so)
==4903== by 0x4EC5384: ddPythonQtDecorators::qt_metacall(QMetaObject::Call, int, void) (in /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/external/drake_visualizer/lib/libddApp.so)
==4903== by 0x5B06BFD: PythonQtCallSlot(PythonQtClassInfo, QObject, _object, bool, PythonQtSlotInfo, void, _object, void, PythonQtPassThisOwnershipType) (in /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/external/drake_visualizer/lib/libPythonQt.so)
==4903== by 0x5B082B1: PythonQtSlotFunction_CallImpl(PythonQtClassInfo, QObject, PythonQtSlotInfo, _object, _object, void, void, PythonQtPassThisOwnershipType) (in /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/external/drake_visualizer/lib/libPythonQt.so)
==4903== by 0x5AF4DAC: PythonQtInstanceWrapper_init(PythonQtInstanceWrapperStruct, _object, _object) (in /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/external/drake_visualizer/lib/libPythonQt.so)
==4903== by 0x832B1AE: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==4903== by 0x8374262: PyObject_Call (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==4903== by 0x831234B: PyEval_EvalFrameEx (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==4903== Your program just tried to execute an instruction that Valgrind
==4903== did not recognise. There are two possible reasons for this.
==4903== 1. Your program has a bug and erroneously jumped to a non-code
==4903== location. If you are running Memcheck and you just saw a
==4903== warning about a bad jump, it's probably your program's fault.
==4903== 2. The instruction is legitimate but Valgrind doesn't handle it,
==4903== i.e. it's Valgrind's fault. If you think this is the case or
==4903== you are not sure, please let us know and we'll try to fix it.
==4903== Either way, Valgrind will now raise a SIGILL signal which will
==4903== probably kill your program.
==4903==
==4903== Process terminating with default action of signal 4 (SIGILL)
==4903== Illegal opcode at address 0x4F0F4E6
==4903== at 0x4F0F4E6: QtTreePropertyBrowserPrivate::init(QWidget) (in /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/external/drake_visualizer/lib/libddApp.so)
==4903== by 0x4F0FB1F: QtTreePropertyBrowser::QtTreePropertyBrowser(QWidget) (in /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/external/drake_visualizer/lib/libddApp.so)
==4903== by 0x4ED7C66: ddPropertiesPanel::setBrowserModeToTree() (in /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/external/drake_visualizer/lib/libddApp.so)
==4903== by 0x4ED801D: ddPropertiesPanel::ddPropertiesPanel(QWidget) (in /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/external/drake_visualizer/lib/libddApp.so)
==4903== by 0x4EC32C5: ddPythonQtDecorators::qt_static_metacall(QObject, QMetaObject::Call, int, void) (in /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/external/drake_visualizer/lib/libddApp.so)
==4903== by 0x4EC5384: ddPythonQtDecorators::qt_metacall(QMetaObject::Call, int, void) (in /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/external/drake_visualizer/lib/libddApp.so)
==4903== by 0x5B06BFD: PythonQtCallSlot(PythonQtClassInfo, QObject, _object, bool, PythonQtSlotInfo, void, _object, void, PythonQtPassThisOwnershipType) (in /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/external/drake_visualizer/lib/libPythonQt.so)
==4903== by 0x5B082B1: PythonQtSlotFunction_CallImpl(PythonQtClassInfo, QObject, PythonQtSlotInfo, _object, _object, void, void, PythonQtPassThisOwnershipType) (in /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/external/drake_visualizer/lib/libPythonQt.so)
==4903== by 0x5AF4DAC: PythonQtInstanceWrapper_init(PythonQtInstanceWrapperStruct, _object, _object) (in /home/aaron/.cache/bazel/_bazel_aaron/957b7f10f23971776307a71dc758fa23/external/drake_visualizer/lib/libPythonQt.so)
==4903== by 0x832B1AE: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==4903== by 0x8374262: PyObject_Call (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==4903== by 0x831234B: PyEval_EvalFrameEx (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==4903==
==4903== HEAP SUMMARY:
==4903== in use at exit: 21,248,797 bytes in 53,266 blocks
==4903== total heap usage: 180,284 allocs, 127,018 frees, 68,897,486 bytes allocated
==4903==
==4903== LEAK SUMMARY:
==4903== definitely lost: 3,648 bytes in 10 blocks
==4903== indirectly lost: 13,147 bytes in 551 blocks
==4903== possibly lost: 186,874 bytes in 153 blocks
==4903== still reachable: 20,723,504 bytes in 51,060 blocks
==4903== of which reachable via heuristic:
==4903== length64 : 7,664 bytes in 113 blocks
==4903== newarray : 4,168 bytes in 53 blocks
==4903== suppressed: 0 bytes in 0 blocks
==4903== Rerun with --leak-check=full to see details of leaked memory
==4903==
==4903== For counts of detected and suppressed errors, rerun with: -v
==4903== Use --track-origins=yes to see where uninitialised values come from
==4903== ERROR SUMMARY: 1050 errors from 57 contexts (suppressed: 0 from 0)
bazel-bin/tools/drake_visualizer: line 20: 4903 Killed valgrind "external/drake_visualizer/bin/drake-visualizer" "$@"

[jwnimmer-tri]

I am stumped. I read some of the code implicated in the backtraces, and nothing stood out at me as anything risky that might differ between users. That your loaded libraries and mine matched exactly, but it works for me, makes it even stranger.

@patmarion Would you have any hints for similar errors you might have seen in the past?

[patmarion]

I don't notice any potential problems in the traces. The next output I'd ask to see is:

echo $PYTHONPATH
echo $LD_LIBRARY_PATH

If $PYTHONPATH is not empty, then you could try:

unset PYTHONPATH
bazel-bin/tools/drake_visualizer

[airfield20]

~/Drake$ echo $PYTHONPATH

~/Drake$ echo $LD_LIBRARY_PATH

The outputs of both variables are empty.

[jwnimmer-tri]

What's the output from the first paragraph of cat /proc/cpuinfo? The bytes are decoding to me as vmovdqa which is an AVX extension. ~(Though possibly they are garbage anyway; it decodes to an RIP-relative address, which seems unusual.)~

[airfield20]

~/Drake$ cat /proc/cpuinfo > cpuinfo.txt
cpuinfo.txt

[jwnimmer-tri]

So indeed that CPU does not support the avx or avx2 features (my CPU does have those). I would be surprised if Drake's pre-built drake-visualizer has or is supposed to have AVX support be required, but if it does, and if valgrind's output is to be believed, then that would explain the problem.

(I still need to think about what to try next.)

[jwnimmer-tri]

In talking with @sammy-tri apparently the PCL build system by default is too aggressive about compiling using the detected CPU, instead of a minimum-supported architecture. If we compiled drake-visualizer on a modern Intel CPU, it might have picked up AVX as a required CPU feature. We are going to try to find a computer that lacks AVX and see if we can reproduce the problem.

FYI @jamiesnape

[sammy-tri]

Tested on a machine without AVX and I reproduced the same fault in gdb:

Starting program: /tmp/drake-vis-test/repro/bin/drake-visualizer 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe15ee700 (LWP 25043)]

Thread 1 "drake-visualize" received signal SIGILL, Illegal instruction.
0x00007ffff78164e6 in QtTreePropertyBrowserPrivate::init(QWidget*) ()
   from /tmp/drake-vis-test/repro/bin/../lib/libddApp.so

The CPU is an AMD Phenom(tm) II X2 565 running Ubuntu 16.04.3:

processor   : 0
vendor_id   : AuthenticAMD
cpu family  : 16
model       : 4
model name  : AMD Phenom(tm) II X2 565 Processor
stepping    : 3
microcode   : 0x10000b6
cpu MHz     : 800.000
cache size  : 512 KB
physical id : 0
siblings    : 2
core id     : 0
cpu cores   : 2
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 5
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt hw_pstate vmmcall npt lbrv svm_lock nrip_save
bugs        : tlb_mmatch fxsave_leak sysret_ss_attrs
bogomips    : 6799.30
TLB size    : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

[jwnimmer-tri]

@jamiesnape Given the above, it seems to me like we need to recompile the drake-visualizer pre-built images on Xenial, after teaching the Director build to use a more conservative CPU architecture, or to build in a VM or cloud that doesn't claim to support AVX.

[jamiesnape]

I make that a pre-2011 machine? I would have to teach Director, so it probably won't happen quickly, unfortunately.

[patmarion]

It looks like the crash is actually in 3rd party library QtPropertyBrowser. Director has been using this library for a very long time, but after the upgrade to qt5 we grabbed a QtPropertyBrowser qt5 port that was found on Github.

The version used is here:

https://github.com/patmarion/QtPropertyBrowser/tree/qt5-port

Note the branch, qt5-port. And in that branch I see this:

https://github.com/patmarion/QtPropertyBrowser/blob/qt5-port/cmake/CompileSettings.cmake

At the bottom it does a find for OpenMP and then adds OpenMP_CXX_FLAGS. Why does it do this, I don't know. I think we could try removing that to see if it resolves the issue?

[patmarion]

Oh, and it adds -march=native to the flags. Maybe that is it?

[jwnimmer-tri]

Yup, the -march=native would definitely cause this, and would be inappropriate when making pre-built images.

[patmarion]

I will make a PR to remove these extra compiler flags. They are unnecessary for this library. They were probably added to support the use case of the upstream project where this was forked.

[jwnimmer-tri]

(Aside: TIL that in x86_64 mode, RIP-relative addressing is typical and common. So I retract my prior about the instruction looking unusual.)

[patmarion]

director/master is updated. If you need to cherry-pick a commit to a different director branch, the commit is:

https://github.com/RobotLocomotion/director/commit/6e3919d8b00e3a0f5c38ad18529559c21e769d3b

Thanks for debugging this guys, I definitely learned something. Hopefully it's resolved now. Let me know!

[airfield20]

So should I manually compile the updated director?

[jwnimmer-tri]

Drake will post updated binaries (and change Drake master to use them) before closing this issue, but there may be a delay before that's accomplished.

(Sorry for the trouble! I guess nobody has tried the this on an AMD computer in the past year or so.)

[airfield20]

Confirmed working on my system. Thanks!