Ray: Unrecognised instruction error running valgrind tests.

[robertnishihara]

I haven't tested this on the master, but I see it in the PR #1413.

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04, valgrind 3.11.0
• Ray installed from (source or binary): source
• Ray version: 0.3.0
• Python version: 3.6.2 (Anaconda)
• Exact command to reproduce:

python ./python/ray/plasma/test/test.py TestPlasmaManager.test_fetch valgrind

When running the above command, I see the following error.

$ python ./python/ray/plasma/test/test.py TestPlasmaManager.test_fetch valgrind
Using valgrind for tests
test_fetch (__main__.TestPlasmaManager) ... ==7917== Memcheck, a memory error detector
==7917== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==7917== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==7917== Command: /home/ubuntu/ray/python/ray/pyarrow_files/pyarrow/plasma_store -s /tmp/plasma_store61225481 -m 1000000000
==7917== 
Allowing the Plasma store to use up to 1GB of memory.
Starting object store with directory /dev/shm and huge page support disabled
==7918== Memcheck, a memory error detector
==7918== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==7918== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==7918== Command: /home/ubuntu/ray/python/ray/pyarrow_files/pyarrow/plasma_store -s /tmp/plasma_store96948296 -m 1000000000
==7918== 
Allowing the Plasma store to use up to 1GB of memory.
Starting object store with directory /dev/shm and huge page support disabled
7919:M 14 Jan 23:57:11.467 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
7919:M 14 Jan 23:57:11.467 # Server started, Redis version 3.9.102
7919:M 14 Jan 23:57:11.467 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
7919:M 14 Jan 23:57:11.467 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
Waiting for redis server at 127.0.0.1:33851 to respond...
7923:M 14 Jan 23:57:11.573 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
7923:M 14 Jan 23:57:11.573 # Server started, Redis version 3.9.102
7923:M 14 Jan 23:57:11.573 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
7923:M 14 Jan 23:57:11.573 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
Waiting for redis server at 127.0.0.1:61771 to respond...
==7927== Memcheck, a memory error detector
==7927== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==7927== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==7927== Command: /home/ubuntu/ray/python/ray/plasma/../core/src/plasma/plasma_manager -s /tmp/plasma_store61225481 -m /tmp/plasma_manager50006852 -h 127.0.0.1 -p 13985 -r 127.0.0.1:33851
==7927== 
vex amd64->IR: unhandled instruction bytes: 0xF 0xC7 0xF0 0x89 0x6 0xF 0x42 0xC1
vex amd64->IR:   REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=0F
vex amd64->IR:   PFX.66=0 PFX.F2=0 PFX.F3=0
==7927== valgrind: Unrecognised instruction at address 0x510eb15.
==7927==    at 0x510EB15: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==7927==    by 0x510ECB1: std::random_device::_M_getval() (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==7927==    by 0x4D3863: std::random_device::operator()() (random.h:1612)
==7927==    by 0x4EE017: ray::UniqueID::from_random() (id.cc:16)
==7927==    by 0x4ACB04: db_connect(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char const*, char const*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (redis.cc:268)
==7927==    by 0x49232F: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:473)
==7927==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7927==    by 0x48D645: main (plasma_manager.cc:1635)
==7927== Your program just tried to execute an instruction that Valgrind
==7927== did not recognise.  There are two possible reasons for this.
==7927== 1. Your program has a bug and erroneously jumped to a non-code
==7927==    location.  If you are running Memcheck and you just saw a
==7927==    warning about a bad jump, it's probably your program's fault.
==7927== 2. The instruction is legitimate but Valgrind doesn't handle it,
==7927==    i.e. it's Valgrind's fault.  If you think this is the case or
==7927==    you are not sure, please let us know and we'll try to fix it.
==7927== Either way, Valgrind will now raise a SIGILL signal which will
==7927== probably kill your program.
==7927== 
==7927== Process terminating with default action of signal 4 (SIGILL)
==7927==  Illegal opcode at address 0x510EB15
==7927==    at 0x510EB15: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==7927==    by 0x510ECB1: std::random_device::_M_getval() (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==7927==    by 0x4D3863: std::random_device::operator()() (random.h:1612)
==7927==    by 0x4EE017: ray::UniqueID::from_random() (id.cc:16)
==7927==    by 0x4ACB04: db_connect(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char const*, char const*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (redis.cc:268)
==7927==    by 0x49232F: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:473)
==7927==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7927==    by 0x48D645: main (plasma_manager.cc:1635)
==7927== 
==7927== HEAP SUMMARY:
==7927==     in use at exit: 127,568 bytes in 16 blocks
==7927==   total heap usage: 24 allocs, 8 frees, 129,058 bytes allocated
==7927== 
==7927== 16 bytes in 1 blocks are still reachable in loss record 1 of 16
==7927==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7927==    by 0x4C03B8: aeApiCreate (ae_epoll.c:40)
==7927==    by 0x4C0926: aeCreateEventLoop (ae.c:78)
==7927==    by 0x4980F7: event_loop_create() (event_loop.cc:9)
==7927==    by 0x49188A: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:457)
==7927==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7927==    by 0x48D645: main (plasma_manager.cc:1635)
==7927== 
==7927== 18 bytes in 1 blocks are still reachable in loss record 2 of 16
==7927==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7927==    by 0x491D00: _M_construct<char*> (basic_string.tcc:223)
==7927==    by 0x491D00: _M_construct_aux<char const*> (basic_string.h:195)
==7927==    by 0x491D00: _M_construct<char const*> (basic_string.h:214)
==7927==    by 0x491D00: basic_string (basic_string.h:456)
==7927==    by 0x491D00: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:467)
==7927==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7927==    by 0x48D645: main (plasma_manager.cc:1635)
==7927== 
==7927== 20 bytes in 1 blocks are still reachable in loss record 3 of 16
==7927==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7927==    by 0x491F1C: _M_construct<char*> (basic_string.tcc:223)
==7927==    by 0x491F1C: _M_construct_aux<char const*> (basic_string.h:195)
==7927==    by 0x491F1C: _M_construct<char const*> (basic_string.h:214)
==7927==    by 0x491F1C: basic_string (basic_string.h:456)
==7927==    by 0x491F1C: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:469)
==7927==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7927==    by 0x48D645: main (plasma_manager.cc:1635)
==7927== 
==7927== 26 bytes in 1 blocks are still reachable in loss record 4 of 16
==7927==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7927==    by 0x49261E: _M_construct<char*> (basic_string.tcc:223)
==7927==    by 0x49261E: _M_construct_aux<char const*> (basic_string.h:195)
==7927==    by 0x49261E: _M_construct<char const*> (basic_string.h:214)
==7927==    by 0x49261E: basic_string (basic_string.h:456)
==7927==    by 0x49261E: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:468)
==7927==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7927==    by 0x48D645: main (plasma_manager.cc:1635)
==7927== 
==7927== 28 bytes in 1 blocks are still reachable in loss record 5 of 16
==7927==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7927==    by 0x4925CE: _M_construct<char*> (basic_string.tcc:223)
==7927==    by 0x4925CE: _M_construct_aux<char const*> (basic_string.h:195)
==7927==    by 0x4925CE: _M_construct<char const*> (basic_string.h:214)
==7927==    by 0x4925CE: basic_string (basic_string.h:456)
==7927==    by 0x4925CE: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:470)
==7927==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7927==    by 0x48D645: main (plasma_manager.cc:1635)
==7927== 
==7927== 64 bytes in 1 blocks are still reachable in loss record 6 of 16
==7927==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7927==    by 0x4D0B31: __gnu_cxx::new_allocator<std::thread>::allocate(unsigned long, void const*) (new_allocator.h:104)
==7927==    by 0x4CF8B6: std::allocator_traits<std::allocator<std::thread> >::allocate(std::allocator<std::thread>&, unsigned long) (alloc_traits.h:491)
==7927==    by 0x4CDED9: std::_Vector_base<std::thread, std::allocator<std::thread> >::_M_allocate(unsigned long) (stl_vector.h:170)
==7927==    by 0x4CC44A: std::_Vector_base<std::thread, std::allocator<std::thread> >::_M_create_storage(unsigned long) (stl_vector.h:185)
==7927==    by 0x4CAA0A: std::_Vector_base<std::thread, std::allocator<std::thread> >::_Vector_base(unsigned long, std::allocator<std::thread> const&) (stl_vector.h:136)
==7927==    by 0x4C91AD: std::vector<std::thread, std::allocator<std::thread> >::vector(unsigned long, std::allocator<std::thread> const&) (stl_vector.h:278)
==7927==    by 0x4C82AE: __static_initialization_and_destruction_0(int, int) (client.cc:62)
==7927==    by 0x4C8321: _GLOBAL__sub_I_client.cc (client.cc:701)
==7927==    by 0x4F7F1C: __libc_csu_init (in /home/ubuntu/ray/python/ray/core/src/plasma/plasma_manager)
==7927==    by 0x560F7BE: (below main) (libc-start.c:247)
==7927== 
==7927== 64 bytes in 1 blocks are still reachable in loss record 7 of 16
==7927==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7927==    by 0x4D0C7B: __gnu_cxx::new_allocator<plasma::UniqueID*>::allocate(unsigned long, void const*) (new_allocator.h:104)
==7927==    by 0x4CFA08: std::allocator_traits<std::allocator<plasma::UniqueID*> >::allocate(std::allocator<plasma::UniqueID*>&, unsigned long) (alloc_traits.h:491)
==7927==    by 0x4CE31B: std::_Deque_base<plasma::UniqueID, std::allocator<plasma::UniqueID> >::_M_allocate_map(unsigned long) (stl_deque.h:615)
==7927==    by 0x4CC70F: std::_Deque_base<plasma::UniqueID, std::allocator<plasma::UniqueID> >::_M_initialize_map(unsigned long) (stl_deque.h:688)
==7927==    by 0x4CAC67: std::_Deque_base<plasma::UniqueID, std::allocator<plasma::UniqueID> >::_Deque_base() (stl_deque.h:490)
==7927==    by 0x4C92D3: std::deque<plasma::UniqueID, std::allocator<plasma::UniqueID> >::deque() (stl_deque.h:883)
==7927==    by 0x4C200E: plasma::PlasmaClient::PlasmaClient() (client.cc:78)
==7927==    by 0x4918A2: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:458)
==7927==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7927==    by 0x48D645: main (plasma_manager.cc:1635)
==7927== 
==7927== 72 bytes in 1 blocks are still reachable in loss record 8 of 16
==7927==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7927==    by 0x4C0859: aeCreateEventLoop (ae.c:67)
==7927==    by 0x4980F7: event_loop_create() (event_loop.cc:9)
==7927==    by 0x49188A: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:457)
==7927==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7927==    by 0x48D645: main (plasma_manager.cc:1635)
==7927== 
==7927== 224 bytes in 1 blocks are still reachable in loss record 9 of 16
==7927==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7927==    by 0x491897: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:458)
==7927==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7927==    by 0x48D645: main (plasma_manager.cc:1635)
==7927== 
==7927== 256 bytes in 1 blocks are still reachable in loss record 10 of 16
==7927==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7927==    by 0x4972EC: allocate (new_allocator.h:104)
==7927==    by 0x4972EC: allocate (alloc_traits.h:491)
==7927==    by 0x4972EC: _M_allocate (stl_vector.h:170)
==7927==    by 0x4972EC: void std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_emplace_back_aux<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) (vector.tcc:412)
==7927==    by 0x492A03: emplace_back<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (vector.tcc:101)
==7927==    by 0x492A03: push_back (stl_vector.h:932)
==7927==    by 0x492A03: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:471)
==7927==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7927==    by 0x48D645: main (plasma_manager.cc:1635)
==7927== 
==7927== 328 bytes in 1 blocks are still reachable in loss record 11 of 16
==7927==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7927==    by 0x4917D8: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:456)
==7927==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7927==    by 0x48D645: main (plasma_manager.cc:1635)
==7927== 
==7927== 500 bytes in 1 blocks are still reachable in loss record 12 of 16
==7927==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7927==    by 0x4D0901: __gnu_cxx::new_allocator<plasma::UniqueID>::allocate(unsigned long, void const*) (new_allocator.h:104)
==7927==    by 0x4CF63E: std::allocator_traits<std::allocator<plasma::UniqueID> >::allocate(std::allocator<plasma::UniqueID>&, unsigned long) (alloc_traits.h:491)
==7927==    by 0x4CD89F: std::_Deque_base<plasma::UniqueID, std::allocator<plasma::UniqueID> >::_M_allocate_node() (stl_deque.h:601)
==7927==    by 0x4CE398: std::_Deque_base<plasma::UniqueID, std::allocator<plasma::UniqueID> >::_M_create_nodes(plasma::UniqueID**, plasma::UniqueID**) (stl_deque.h:726)
==7927==    by 0x4CC768: std::_Deque_base<plasma::UniqueID, std::allocator<plasma::UniqueID> >::_M_initialize_map(unsigned long) (stl_deque.h:700)
==7927==    by 0x4CAC67: std::_Deque_base<plasma::UniqueID, std::allocator<plasma::UniqueID> >::_Deque_base() (stl_deque.h:490)
==7927==    by 0x4C92D3: std::deque<plasma::UniqueID, std::allocator<plasma::UniqueID> >::deque() (stl_deque.h:883)
==7927==    by 0x4C200E: plasma::PlasmaClient::PlasmaClient() (client.cc:78)
==7927==    by 0x4918A2: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:458)
==7927==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7927==    by 0x48D645: main (plasma_manager.cc:1635)
==7927== 
==7927== 8,192 bytes in 1 blocks are still reachable in loss record 13 of 16
==7927==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7927==    by 0x4C0895: aeCreateEventLoop (ae.c:69)
==7927==    by 0x4980F7: event_loop_create() (event_loop.cc:9)
==7927==    by 0x49188A: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:457)
==7927==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7927==    by 0x48D645: main (plasma_manager.cc:1635)
==7927== 
==7927== 12,288 bytes in 1 blocks are still reachable in loss record 14 of 16
==7927==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7927==    by 0x4C03EC: aeApiCreate (ae_epoll.c:43)
==7927==    by 0x4C0926: aeCreateEventLoop (ae.c:78)
==7927==    by 0x4980F7: event_loop_create() (event_loop.cc:9)
==7927==    by 0x49188A: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:457)
==7927==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7927==    by 0x48D645: main (plasma_manager.cc:1635)
==7927== 
==7927== 32,768 bytes in 1 blocks are still reachable in loss record 15 of 16
==7927==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7927==    by 0x4C0879: aeCreateEventLoop (ae.c:68)
==7927==    by 0x4980F7: event_loop_create() (event_loop.cc:9)
==7927==    by 0x49188A: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:457)
==7927==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7927==    by 0x48D645: main (plasma_manager.cc:1635)
==7927== 
==7927== 72,704 bytes in 1 blocks are still reachable in loss record 16 of 16
==7927==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7927==    by 0x50E0EFF: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==7927==    by 0x40106B9: call_init.part.0 (dl-init.c:72)
==7927==    by 0x40107CA: call_init (dl-init.c:30)
==7927==    by 0x40107CA: _dl_init (dl-init.c:120)
==7927==    by 0x4000C69: ??? (in /lib/x86_64-linux-gnu/ld-2.23.so)
==7927==    by 0xA: ???
==7927==    by 0xFFF00065A: ???
==7927==    by 0xFFF00069F: ???
==7927==    by 0xFFF0006A2: ???
==7927==    by 0xFFF0006BC: ???
==7927==    by 0xFFF0006BF: ???
==7927==    by 0xFFF0006DB: ???
==7927== 
==7927== LEAK SUMMARY:
==7927==    definitely lost: 0 bytes in 0 blocks
==7927==    indirectly lost: 0 bytes in 0 blocks
==7927==      possibly lost: 0 bytes in 0 blocks
==7927==    still reachable: 127,568 bytes in 16 blocks
==7927==         suppressed: 0 bytes in 0 blocks
==7927== 
==7927== For counts of detected and suppressed errors, rerun with: -v
==7927== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==7928== Memcheck, a memory error detector
==7928== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==7928== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==7928== Command: /home/ubuntu/ray/python/ray/plasma/../core/src/plasma/plasma_manager -s /tmp/plasma_store96948296 -m /tmp/plasma_manager11547204 -h 127.0.0.1 -p 20732 -r 127.0.0.1:33851
==7928== 
Disconnecting client on fd 5
vex amd64->IR: unhandled instruction bytes: 0xF 0xC7 0xF0 0x89 0x6 0xF 0x42 0xC1
vex amd64->IR:   REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=0F
vex amd64->IR:   PFX.66=0 PFX.F2=0 PFX.F3=0
==7928== valgrind: Unrecognised instruction at address 0x510eb15.
==7928==    at 0x510EB15: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==7928==    by 0x510ECB1: std::random_device::_M_getval() (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==7928==    by 0x4D3863: std::random_device::operator()() (random.h:1612)
==7928==    by 0x4EE017: ray::UniqueID::from_random() (id.cc:16)
==7928==    by 0x4ACB04: db_connect(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char const*, char const*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (redis.cc:268)
==7928==    by 0x49232F: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:473)
==7928==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7928==    by 0x48D645: main (plasma_manager.cc:1635)
==7928== Your program just tried to execute an instruction that Valgrind
==7928== did not recognise.  There are two possible reasons for this.
==7928== 1. Your program has a bug and erroneously jumped to a non-code
==7928==    location.  If you are running Memcheck and you just saw a
==7928==    warning about a bad jump, it's probably your program's fault.
==7928== 2. The instruction is legitimate but Valgrind doesn't handle it,
==7928==    i.e. it's Valgrind's fault.  If you think this is the case or
==7928==    you are not sure, please let us know and we'll try to fix it.
==7928== Either way, Valgrind will now raise a SIGILL signal which will
==7928== probably kill your program.
==7928== 
==7928== Process terminating with default action of signal 4 (SIGILL)
==7928==  Illegal opcode at address 0x510EB15
==7928==    at 0x510EB15: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==7928==    by 0x510ECB1: std::random_device::_M_getval() (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==7928==    by 0x4D3863: std::random_device::operator()() (random.h:1612)
==7928==    by 0x4EE017: ray::UniqueID::from_random() (id.cc:16)
==7928==    by 0x4ACB04: db_connect(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char const*, char const*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (redis.cc:268)
==7928==    by 0x49232F: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:473)
==7928==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7928==    by 0x48D645: main (plasma_manager.cc:1635)
==7928== 
==7928== HEAP SUMMARY:
==7928==     in use at exit: 127,568 bytes in 16 blocks
==7928==   total heap usage: 24 allocs, 8 frees, 129,058 bytes allocated
==7928== 
==7928== 16 bytes in 1 blocks are still reachable in loss record 1 of 16
==7928==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7928==    by 0x4C03B8: aeApiCreate (ae_epoll.c:40)
==7928==    by 0x4C0926: aeCreateEventLoop (ae.c:78)
==7928==    by 0x4980F7: event_loop_create() (event_loop.cc:9)
==7928==    by 0x49188A: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:457)
==7928==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7928==    by 0x48D645: main (plasma_manager.cc:1635)
==7928== 
==7928== 18 bytes in 1 blocks are still reachable in loss record 2 of 16
==7928==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7928==    by 0x491D00: _M_construct<char*> (basic_string.tcc:223)
==7928==    by 0x491D00: _M_construct_aux<char const*> (basic_string.h:195)
==7928==    by 0x491D00: _M_construct<char const*> (basic_string.h:214)
==7928==    by 0x491D00: basic_string (basic_string.h:456)
==7928==    by 0x491D00: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:467)
==7928==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7928==    by 0x48D645: main (plasma_manager.cc:1635)
==7928== 
==7928== 20 bytes in 1 blocks are still reachable in loss record 3 of 16
==7928==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7928==    by 0x491F1C: _M_construct<char*> (basic_string.tcc:223)
==7928==    by 0x491F1C: _M_construct_aux<char const*> (basic_string.h:195)
==7928==    by 0x491F1C: _M_construct<char const*> (basic_string.h:214)
==7928==    by 0x491F1C: basic_string (basic_string.h:456)
==7928==    by 0x491F1C: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:469)
==7928==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7928==    by 0x48D645: main (plasma_manager.cc:1635)
==7928== 
==7928== 26 bytes in 1 blocks are still reachable in loss record 4 of 16
==7928==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7928==    by 0x49261E: _M_construct<char*> (basic_string.tcc:223)
==7928==    by 0x49261E: _M_construct_aux<char const*> (basic_string.h:195)
==7928==    by 0x49261E: _M_construct<char const*> (basic_string.h:214)
==7928==    by 0x49261E: basic_string (basic_string.h:456)
==7928==    by 0x49261E: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:468)
==7928==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7928==    by 0x48D645: main (plasma_manager.cc:1635)
==7928== 
==7928== 28 bytes in 1 blocks are still reachable in loss record 5 of 16
==7928==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7928==    by 0x4925CE: _M_construct<char*> (basic_string.tcc:223)
==7928==    by 0x4925CE: _M_construct_aux<char const*> (basic_string.h:195)
==7928==    by 0x4925CE: _M_construct<char const*> (basic_string.h:214)
==7928==    by 0x4925CE: basic_string (basic_string.h:456)
==7928==    by 0x4925CE: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:470)
==7928==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7928==    by 0x48D645: main (plasma_manager.cc:1635)
==7928== 
==7928== 64 bytes in 1 blocks are still reachable in loss record 6 of 16
==7928==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7928==    by 0x4D0B31: __gnu_cxx::new_allocator<std::thread>::allocate(unsigned long, void const*) (new_allocator.h:104)
==7928==    by 0x4CF8B6: std::allocator_traits<std::allocator<std::thread> >::allocate(std::allocator<std::thread>&, unsigned long) (alloc_traits.h:491)
==7928==    by 0x4CDED9: std::_Vector_base<std::thread, std::allocator<std::thread> >::_M_allocate(unsigned long) (stl_vector.h:170)
==7928==    by 0x4CC44A: std::_Vector_base<std::thread, std::allocator<std::thread> >::_M_create_storage(unsigned long) (stl_vector.h:185)
==7928==    by 0x4CAA0A: std::_Vector_base<std::thread, std::allocator<std::thread> >::_Vector_base(unsigned long, std::allocator<std::thread> const&) (stl_vector.h:136)
==7928==    by 0x4C91AD: std::vector<std::thread, std::allocator<std::thread> >::vector(unsigned long, std::allocator<std::thread> const&) (stl_vector.h:278)
==7928==    by 0x4C82AE: __static_initialization_and_destruction_0(int, int) (client.cc:62)
==7928==    by 0x4C8321: _GLOBAL__sub_I_client.cc (client.cc:701)
==7928==    by 0x4F7F1C: __libc_csu_init (in /home/ubuntu/ray/python/ray/core/src/plasma/plasma_manager)
==7928==    by 0x560F7BE: (below main) (libc-start.c:247)
==7928== 
==7928== 64 bytes in 1 blocks are still reachable in loss record 7 of 16
==7928==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7928==    by 0x4D0C7B: __gnu_cxx::new_allocator<plasma::UniqueID*>::allocate(unsigned long, void const*) (new_allocator.h:104)
==7928==    by 0x4CFA08: std::allocator_traits<std::allocator<plasma::UniqueID*> >::allocate(std::allocator<plasma::UniqueID*>&, unsigned long) (alloc_traits.h:491)
==7928==    by 0x4CE31B: std::_Deque_base<plasma::UniqueID, std::allocator<plasma::UniqueID> >::_M_allocate_map(unsigned long) (stl_deque.h:615)
==7928==    by 0x4CC70F: std::_Deque_base<plasma::UniqueID, std::allocator<plasma::UniqueID> >::_M_initialize_map(unsigned long) (stl_deque.h:688)
==7928==    by 0x4CAC67: std::_Deque_base<plasma::UniqueID, std::allocator<plasma::UniqueID> >::_Deque_base() (stl_deque.h:490)
==7928==    by 0x4C92D3: std::deque<plasma::UniqueID, std::allocator<plasma::UniqueID> >::deque() (stl_deque.h:883)
==7928==    by 0x4C200E: plasma::PlasmaClient::PlasmaClient() (client.cc:78)
==7928==    by 0x4918A2: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:458)
==7928==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7928==    by 0x48D645: main (plasma_manager.cc:1635)
==7928== 
==7928== 72 bytes in 1 blocks are still reachable in loss record 8 of 16
==7928==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7928==    by 0x4C0859: aeCreateEventLoop (ae.c:67)
==7928==    by 0x4980F7: event_loop_create() (event_loop.cc:9)
==7928==    by 0x49188A: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:457)
==7928==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7928==    by 0x48D645: main (plasma_manager.cc:1635)
==7928== 
==7928== 224 bytes in 1 blocks are still reachable in loss record 9 of 16
==7928==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7928==    by 0x491897: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:458)
==7928==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7928==    by 0x48D645: main (plasma_manager.cc:1635)
==7928== 
==7928== 256 bytes in 1 blocks are still reachable in loss record 10 of 16
==7928==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7928==    by 0x4972EC: allocate (new_allocator.h:104)
==7928==    by 0x4972EC: allocate (alloc_traits.h:491)
==7928==    by 0x4972EC: _M_allocate (stl_vector.h:170)
==7928==    by 0x4972EC: void std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_emplace_back_aux<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) (vector.tcc:412)
==7928==    by 0x492A03: emplace_back<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (vector.tcc:101)
==7928==    by 0x492A03: push_back (stl_vector.h:932)
==7928==    by 0x492A03: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:471)
==7928==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7928==    by 0x48D645: main (plasma_manager.cc:1635)
==7928== 
==7928== 328 bytes in 1 blocks are still reachable in loss record 11 of 16
==7928==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7928==    by 0x4917D8: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:456)
==7928==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
Connection to IPC socket failed for pathname ==7928==    by 0x48D645: main (plasma_manager.cc:1635)
==7928== 
==7928== 500 bytes in 1 blocks are still reachable in loss record 12 of 16
/tmp/plasma_manager50006852, retrying ==7928==    at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
50==7928==    by 0x4D0901: __gnu_cxx::new_allocator<plasma::UniqueID>::allocate(unsigned long, void const*) (new_allocator.h:104)
 more times
==7928==    by 0x4CF63E: std::allocator_traits<std::allocator<plasma::UniqueID> >::allocate(std::allocator<plasma::UniqueID>&, unsigned long) (alloc_traits.h:491)
==7928==    by 0x4CD89F: std::_Deque_base<plasma::UniqueID, std::allocator<plasma::UniqueID> >::_M_allocate_node() (stl_deque.h:601)
==7928==    by 0x4CE398: std::_Deque_base<plasma::UniqueID, std::allocator<plasma::UniqueID> >::_M_create_nodes(plasma::UniqueID**, plasma::UniqueID**) (stl_deque.h:726)
==7928==    by 0x4CC768: std::_Deque_base<plasma::UniqueID, std::allocator<plasma::UniqueID> >::_M_initialize_map(unsigned long) (stl_deque.h:700)
==7928==    by 0x4CAC67: std::_Deque_base<plasma::UniqueID, std::allocator<plasma::UniqueID> >::_Deque_base() (stl_deque.h:490)
==7928==    by 0x4C92D3: std::deque<plasma::UniqueID, std::allocator<plasma::UniqueID> >::deque() (stl_deque.h:883)
==7928==    by 0x4C200E: plasma::PlasmaClient::PlasmaClient() (client.cc:78)
==7928==    by 0x4918A2: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:458)
==7928==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7928==    by 0x48D645: main (plasma_manager.cc:1635)
==7928== 
==7928== 8,192 bytes in 1 blocks are still reachable in loss record 13 of 16
==7928==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7928==    by 0x4C0895: aeCreateEventLoop (ae.c:69)
==7928==    by 0x4980F7: event_loop_create() (event_loop.cc:9)
==7928==    by 0x49188A: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:457)
==7928==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7928==    by 0x48D645: main (plasma_manager.cc:1635)
==7928== 
==7928== 12,288 bytes in 1 blocks are still reachable in loss record 14 of 16
==7928==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7928==    by 0x4C03EC: aeApiCreate (ae_epoll.c:43)
==7928==    by 0x4C0926: aeCreateEventLoop (ae.c:78)
==7928==    by 0x4980F7: event_loop_create() (event_loop.cc:9)
==7928==    by 0x49188A: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:457)
==7928==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7928==    by 0x48D645: main (plasma_manager.cc:1635)
==7928== 
==7928== 32,768 bytes in 1 blocks are still reachable in loss record 15 of 16
==7928==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7928==    by 0x4C0879: aeCreateEventLoop (ae.c:68)
==7928==    by 0x4980F7: event_loop_create() (event_loop.cc:9)
==7928==    by 0x49188A: PlasmaManagerState_init(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:457)
==7928==    by 0x492C39: start_server(char const*, char const*, char const*, int, char const*, int) (plasma_manager.cc:1525)
==7928==    by 0x48D645: main (plasma_manager.cc:1635)
==7928== 
==7928== 72,704 bytes in 1 blocks are still reachable in loss record 16 of 16
==7928==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7928==    by 0x50E0EFF: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==7928==    by 0x40106B9: call_init.part.0 (dl-init.c:72)
==7928==    by 0x40107CA: call_init (dl-init.c:30)
==7928==    by 0x40107CA: _dl_init (dl-init.c:120)
==7928==    by 0x4000C69: ??? (in /lib/x86_64-linux-gnu/ld-2.23.so)
==7928==    by 0xA: ???
==7928==    by 0xFFF00065A: ???
==7928==    by 0xFFF00069F: ???
==7928==    by 0xFFF0006A2: ???
==7928==    by 0xFFF0006BC: ???
==7928==    by 0xFFF0006BF: ???
==7928==    by 0xFFF0006DB: ???
==7928== 
==7928== LEAK SUMMARY:
==7928==    definitely lost: 0 bytes in 0 blocks
==7928==    indirectly lost: 0 bytes in 0 blocks
==7928==      possibly lost: 0 bytes in 0 blocks
==7928==    still reachable: 127,568 bytes in 16 blocks
==7928==         suppressed: 0 bytes in 0 blocks
==7928== 
==7928== For counts of detected and suppressed errors, rerun with: -v
==7928== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 49 more times
Disconnecting client on fd 5
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 48 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 47 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 46 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 45 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 44 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 43 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 42 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 41 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 40 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 39 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 38 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 37 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 36 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 35 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 34 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 33 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 32 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 31 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 30 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 29 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 28 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 27 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 26 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 25 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 24 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 23 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 22 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 21 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 20 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 19 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 18 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 17 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 16 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 15 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 14 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 13 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 12 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 11 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 10 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 9 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 8 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 7 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 6 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 5 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 4 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 3 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 2 more times
Connection to IPC socket failed for pathname /tmp/plasma_manager50006852, retrying 1 more times
ERROR

======================================================================
ERROR: test_fetch (__main__.TestPlasmaManager)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./python/ray/plasma/test/test.py", line 122, in setUp
    self.client1 = plasma.connect(store_name1, manager_name1, 64)
  File "plasma.pyx", line 676, in pyarrow.plasma.connect
  File "error.pxi", line 79, in pyarrow.lib.check_status
pyarrow.lib.ArrowIOError: Could not connect to socket /tmp/plasma_manager50006852

----------------------------------------------------------------------
Ran 1 test in 9.235s

FAILED (errors=1)

It seems to not recognize an instruction when calling ray::UniqueID::from_random(). This seems to be a known issue. Similar errors appear in google/protobuf#3006 and https://bugs.launchpad.net/ubuntu/+source/valgrind/+bug/1501545. Seems like upgrading to valgrind 3.12.0 should fix it, though that may require building valgrind from source.

[robertnishihara]

Note that on Travis we are currently using Ubuntu 14.04.5 and Valgrind-3.10.1 and we don't seem to see the error there.

[robertnishihara]

Workaround: I installed valgrind-3.13.0 using linuxbrew. Following http://linuxbrew.sh/, I did

sh -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)"

test -d ~/.linuxbrew && PATH="$HOME/.linuxbrew/bin:$HOME/.linuxbrew/sbin:$PATH"
test -d /home/linuxbrew/.linuxbrew && PATH="/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:$PATH"
test -r ~/.bash_profile && echo "export PATH='$(brew --prefix)/bin:$(brew --prefix)/sbin'":'"$PATH"' >>~/.bash_profile
echo "export PATH='$(brew --prefix)/bin:$(brew --prefix)/sbin'":'"$PATH"' >>~/.profile

brew update
brew install valgrind

The instructions at http://valgrind.org/downloads/repository.html probably would work as well and may be simpler.

[robertnishihara]

Closing because there's no clear action item here.

The issue hasn't shown up in Travis yet presumably because Travis's valgrind version is too old.

To use valgrind locally we need to install a newer version of valgrind as described above. Also see #1459 for how this can be done.