Rocksdb: RocksDB crash observed when we try to read a key while it is in the process of close()

[sayaji15]

RocksDB version: 5.11.3

We observed a crash when a thread was trying to read from rocksdb and another thread was trying to close the db. This ended up killing our JVM.
Shouldn't it just fail the read operation ?Is this a known issue? If yes, is it fixed in the newer version?

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000000000030, pid=41084, tid=0x00007fd5dd189700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_121-b13) (build 1.8.0_121-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  0x0000000000000030
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  T H R E A D  ---------------

Current thread (0x00007fd6689e8800):  JavaThread "kvstore-kafka-java-consumer-7" [_thread_in_native, id=43865, stack(0x00007fd5dd089000,0x00007fd5dd18a000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000030

Registers:
RAX=0x00007fd45f2478d0, RBX=0x000000000000169a, RCX=0x00007fd67ae04133, RDX=0x000000000000169a
RSP=0x00007fd5dd1855e8, RBP=0x00007fd5dd1856c0, RSI=0x0000000002814920, RDI=0x00007fd535e06e50
R8 =0x00007fd5dd185760, R9 =0x0000000002814920, R10=0x0000000005d22bc8, R11=0x0000000000000000
R12=0x0000000000000000, R13=0x00007fd535e06e50, R14=0x0000000005d24262, R15=0x0000000002815fba
RIP=0x0000000000000030, EFLAGS=0x0000000000010246, CSGSFS=0x0000000000000033, ERR=0x0000000000000014
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007fd5dd1855e8)
0x00007fd5dd1855e8:   00007fd64723795b 00007fd5dd185640
0x00007fd5dd1855f8:   00007fd5dd185760 0000000002814920
0x00007fd5dd185608:   000000000000169a 00007fd5dd185770
0x00007fd5dd185618:   00007fd6689e8800 00007fd5dd1856b0
0x00007fd5dd185628:   00007fd679e506b9 00007fd6689e8800
0x00007fd5dd185638:   00007fd5dd1856e0 00007fd5dd185670
0x00007fd5dd185648:   00007fd656bfe480 00007fd5dd185920
0x00007fd5dd185658:   00007fd647296a25 00007fd656d9e5e0
0x00007fd5dd185668:   00007fd6689e8800 00007fd656d9e5e0
0x00007fd5dd185678:   00007fd6689e8800 01007fd5dd185850
0x00007fd5dd185688:   00007fd67a9d5120 00000000000013c8
0x00007fd5dd185698:   00007fd5dd185890 0000000000000000
0x00007fd5dd1856a8:   00007fd471b3cda0 0000000000000000
0x00007fd5dd1856b8:   000000000000169a 00007fd5dd1857d0
0x00007fd5dd1856c8:   00007fd6472ce53f 00007fd67a9d5188
0x00007fd5dd1856d8:   0000000000000000 0000006600000015
0x00007fd5dd1856e8:   00007fd6476f2220 00007fd5dd1859d8
0x00007fd5dd1856f8:   0000000000000000 0000000000000000
0x00007fd5dd185708:   0000000000000001 00000000000016d0
0x00007fd5dd185718:   0000008a0000016b 0000007c00000066
0x00007fd5dd185728:   0000000002814920 0000000005d22bc8
0x00007fd5dd185738:   00007fd5dd185760 000000000000169a
0x00007fd5dd185748:   00007fd5dd185770 00007fd5dd185920
0x00007fd5dd185758:   0000000000000000 00007fd647431641
0x00007fd5dd185768:   0000000000000000 0000000000000000
0x00007fd5dd185778:   0000000000000000 00007fd5dd186e00
0x00007fd5dd185788:   0000000000000000 0000000000000000
0x00007fd5dd185798:   0000000012cb0580 00007fd5dd1857d0
0x00007fd5dd1857a8:   00007fd5dd185920 00007fd5dd186e90
0x00007fd5dd1857b8:   00007fd5dd185890 0000000000000001
0x00007fd5dd1857c8:   00007fd5dd1870a0 00007fd5dd1858f0
0x00007fd5dd1857d8:   00007fd64729772c 00007fd5dd185e70 

Instructions: (pc=0x0000000000000030)
0x0000000000000010:   
[error occurred during error reporting (printing registers, top of stack, instructions near pc), id 0xb]

Register to memory mapping:

RAX=0x00007fd45f2478d0 is an unknown value
RBX=0x000000000000169a is an unknown value
RCX=0x00007fd67ae04133: pread64+0x33 in /lib64/libpthread.so.0 at 0x00007fd67adf5000
RDX=0x000000000000169a is an unknown value
RSP=0x00007fd5dd1855e8 is pointing into the stack for thread: 0x00007fd6689e8800
RBP=0x00007fd5dd1856c0 is pointing into the stack for thread: 0x00007fd6689e8800
RSI=0x0000000002814920 is an unknown value
RDI=0x00007fd535e06e50 is an unknown value
R8 =0x00007fd5dd185760 is pointing into the stack for thread: 0x00007fd6689e8800
R9 =0x0000000002814920 is an unknown value
R10=0x0000000005d22bc8 is an unknown value
R11=0x0000000000000000 is an unknown value
R12=0x0000000000000000 is an unknown value
R13=0x00007fd535e06e50 is an unknown value
R14=0x0000000005d24262 is an unknown value
R15=0x0000000002815fba is an unknown value


Stack: [0x00007fd5dd089000,0x00007fd5dd18a000],  sp=0x00007fd5dd1855e8,  free space=1009k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  0x0000000000000030
C  [librocksdbjni3058571284681324048.so+0x49a53f]  rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x1af
C  [librocksdbjni3058571284681324048.so+0x46372c]  rocksdb::BlockFetcher::ReadBlockContents()+0x27c
C  [librocksdbjni3058571284681324048.so+0x451583]
C  [librocksdbjni3058571284681324048.so+0x453d78]  rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::FilePrefetchBuffer*, rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool, rocksdb::GetContext*)+0x318
C  [librocksdbjni3058571284681324048.so+0x4540b8]  rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::GetContext*, rocksdb::Status)+0x138
C  [librocksdbjni3058571284681324048.so+0x454633]  rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::BlockIter*, bool, rocksdb::GetContext*)+0x93
C  [librocksdbjni3058571284681324048.so+0x4575cc]  rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::GetContext*, bool)+0x4fc
C  [librocksdbjni3058571284681324048.so+0x3b0570]  rocksdb::TableCache::Get(rocksdb::ReadOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::HistogramImpl*, bool, int)+0x270
C  [librocksdbjni3058571284681324048.so+0x3cd539]  rocksdb::Version::Get(rocksdb::ReadOptions const&, rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::Status*, rocksdb::MergeContext*, rocksdb::RangeDelAggregator*, bool*, bool*, unsigned long*, rocksdb::ReadCallback*, bool*)+0x839
C  [librocksdbjni3058571284681324048.so+0x3344d5]  rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*, bool*, rocksdb::ReadCallback*, bool*)+0x695
C  [librocksdbjni3058571284681324048.so+0x3349bb]  rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*)+0x2b
C  [librocksdbjni3058571284681324048.so+0x303826]  rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, std::string*)+0x86
C  [librocksdbjni3058571284681324048.so+0x302bbd]  rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, std::string*)+0x4d
C  [librocksdbjni3058571284681324048.so+0x2b38e9]  rocksdb_get_helper(JNIEnv_*, rocksdb::DB*, rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, _jbyteArray*, int, int)+0x279
C  [librocksdbjni3058571284681324048.so+0x2b39d4]  Java_org_rocksdb_RocksDB_get__J_3BII+0x54
J 6286  org.rocksdb.RocksDB.get(J[BII)[B (0 bytes) @ 0x00007fd66d32e9c6 [0x00007fd66d32e900+0xc6]
...
J 15728 C2 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; (14 bytes) @ 0x00007fd66c9b9f70 [0x00007fd66c9b9da0+0x1d0]
J 13700 C1 java.util.concurrent.FutureTask.run()V (126 bytes) @ 0x00007fd66e57a19c [0x00007fd66e579f40+0x25c]
J 15807 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007fd66eb6bb20 [0x00007fd66eb6b640+0x4e0]
J 10720 C1 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (9 bytes) @ 0x00007fd66c8e214c [0x00007fd66c8e2040+0x10c]
J 8249 C1 java.lang.Thread.run()V (17 bytes) @ 0x00007fd66d42df0c [0x00007fd66d42ddc0+0x14c]
v  ~StubRoutines::call_stub
V  [libjvm.so+0x690dd6]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V  [libjvm.so+0x6912e1]  JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321
V  [libjvm.so+0x691787]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47
V  [libjvm.so+0x72cb00]  thread_entry(JavaThread*, Thread*)+0xa0
V  [libjvm.so+0xa75543]  JavaThread::thread_main_inner()+0x103
V  [libjvm.so+0xa7568c]  JavaThread::run()+0x11c
V  [libjvm.so+0x926268]  java_start(Thread*)+0x108
C  [libpthread.so.0+0x7aa1]  start_thread+0xd1

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 6286  org.rocksdb.RocksDB.get(J[BII)[B (0 bytes) @ 0x00007fd66d32e94c [0x00007fd66d32e900+0x4c]
J 11253 C2 com.turn.platform.profile.store.CountingKeyValueStore.get(J)[B (165 bytes) @ 0x00007fd66d0aa884 [0x00007fd66d0aa3c0+0x4c4]
...
J 15728 C2 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; (14 bytes) @ 0x00007fd66c9b9f70 [0x00007fd66c9b9da0+0x1d0]
J 13700 C1 java.util.concurrent.FutureTask.run()V (126 bytes) @ 0x00007fd66e57a19c [0x00007fd66e579f40+0x25c]
J 15807 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007fd66eb6bb20 [0x00007fd66eb6b640+0x4e0]
J 10720 C1 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (9 bytes) @ 0x00007fd66c8e214c [0x00007fd66c8e2040+0x10c]
J 8249 C1 java.lang.Thread.run()V (17 bytes) @ 0x00007fd66d42df0c [0x00007fd66d42ddc0+0x14c]
v  ~StubRoutines::call_stub

[siying]

"a thread was trying to read from rocksdb and another thread was trying to close the db" -- this is not thread safe. Users are responsible to make sure no read/write request going to RocksDB before trying to close it.