Rocksdb: optimistic transactions suffer from mutex contention

[mdcallag]

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://www.facebook.com/groups/rocksdb.dev

Expected behavior

QPS to increase with concurrency until saturation

Actual behavior

QPS decreases with concurrency

Steps to reproduce the behavior

Compare QPS with trx=p and trx=o in the results linked below. trx=p is pessimistic and trx=o is optimistic. Pessimistic does much better.

Test and results at https://gist.github.com/mdcallag/09fc94ff585297c00adef6243e6339e3
These are from Feb 2016.

Update script and results from today using RocksDB 5.14.3
script - https://gist.github.com/mdcallag/9f4d22163b3a155efbde839dba4fe1f6
results - https://gist.github.com/mdcallag/6820718a98913780ec50652b6752dfdb

With optimistic and 24 clients I see:

• in top, CPU util% is less than 200, so at most 2 cores are busy

2 PMP stack traces

--- first trace
24 __pthread_cond_timedwait,clone
20 pthread_cond_wait@@GLIBC_2.3.2,rocksdb::WriteThread::AwaitState,rocksdb::DBImpl::PipelinedWriteImpl,rocksdb::DBImpl::WriteImpl,rocksdb::DBImpl::WriteWithCallback,rocksdb::OptimisticTransaction::Commit,rocksdb::RandomTransactionInserter::DoInsert,rocksdb::Benchmark::RandomTransaction,rocksdb::Benchmark::ThreadBody,rocksdb::(anonymous,start_thread
4 pthread_cond_wait@@GLIBC_2.3.2,clone
3 pthread_cond_wait@@GLIBC_2.3.2,rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper,execute_native_thread_routine,start_thread
1 rocksdb::WriteThread::AwaitState,rocksdb::DBImpl::WriteImpl,rocksdb::DBImpl::WriteWithCallback,rocksdb::OptimisticTransaction::Commit,rocksdb::RandomTransactionInserter::DoInsert,rocksdb::Benchmark::RandomTransaction,rocksdb::Benchmark::ThreadBody,rocksdb::(anonymous,start_thread
1 rocksdb::MemTable::KeyComparator::operator(),rocksdb::InlineSkipList 1 rocksdb::MemTable::Get,rocksdb::DBImpl::Get,rocksdb::WriteBatchWithIndex::GetFromBatchAndDB,rocksdb::WriteBatchWithIndex::GetFromBatchAndDB,rocksdb::TransactionBaseImpl::Get,rocksdb::TransactionBaseImpl::GetForUpdate,rocksdb::TransactionBaseImpl::GetForUpdate,rocksdb::RandomTransactionInserter::DBGet,rocksdb::RandomTransactionInserter::DoInsert,rocksdb::Benchmark::RandomTransaction,rocksdb::Benchmark::ThreadBody,rocksdb::(anonymous,start_thread
1 pthread_cond_wait@@GLIBC_2.3.2,rocksdb::Benchmark::Run,rocksdb::db_bench_tool,_start
1 __pthread_cond_signal,rocksdb::WriteThread::ExitAsBatchGroupLeader,rocksdb::DBImpl::PipelinedWriteImpl,rocksdb::DBImpl::WriteImpl,rocksdb::DBImpl::WriteWithCallback,rocksdb::OptimisticTransaction::Commit,rocksdb::RandomTransactionInserter::DoInsert,rocksdb::Benchmark::RandomTransaction,rocksdb::Benchmark::ThreadBody,rocksdb::(anonymous,start_thread
1

--- second trace
25 __pthread_cond_timedwait,clone
23 pthread_cond_wait@@GLIBC_2.3.2,rocksdb::WriteThread::AwaitState,rocksdb::DBImpl::PipelinedWriteImpl,rocksdb::DBImpl::WriteImpl,rocksdb::DBImpl::WriteWithCallback,rocksdb::OptimisticTransaction::Commit,rocksdb::RandomTransactionInserter::DoInsert,rocksdb::Benchmark::RandomTransaction,rocksdb::Benchmark::ThreadBody,rocksdb::(anonymous,start_thread
3 pthread_cond_wait@@GLIBC_2.3.2,clone
2 pthread_cond_wait@@GLIBC_2.3.2,rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper,execute_native_thread_routine,start_thread
1 rocksdb::InlineSkipList 1 rocksdb::Hash,rocksdb::BlockBasedTableBuilder::Add,rocksdb::BuildTable,rocksdb::FlushJob::WriteLevel0Table,rocksdb::FlushJob::Run,rocksdb::DBImpl::FlushMemTableToOutputFile,rocksdb::DBImpl::BackgroundFlush,rocksdb::DBImpl::BackgroundCallFlush,rocksdb::ThreadPoolImpl::Impl::BGThread,rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper,execute_native_thread_routine,start_thread
1 pthread_cond_wait@@GLIBC_2.3.2,rocksdb::Benchmark::Run,rocksdb::db_bench_tool,_start

[wolfkdy]

During a multi-cf comparation of DB/OptTxnDB/TxnDB, I also found OptTxnDB is 5 times slower than DB/TxnDB. It seems cpu can not be fully used. here is the perf result: https://groups.google.com/forum/#!topic/rocksdb/7fgTyJa0uFo

I'll do a single cf comparation there days and try to understand where the bottleneck is.

[wolfkdy]

Well, @mdcallag , I think I've more or less undertood it. But unfortunely, Under OptTxnDB's architecture, I think it hopeless to improve OptTxnDB's performance.

At first, Reasons for short,
1) OptimisticTransactionCallback::AllowWriteBatching returns false
2) OptTxn does conflict check during commit procedure,

Explainations:
As we all know, RocksDB uses group commit mechanism to speedup the commit procedure. AllowWriteBatching==false, makes transactions commit one by one, Despite the correctness,
I turned OptTxnDB's AllowWriteBatching=true for experiment, and the throughput improved
by 30%.

It seems AllowWriteBatching is not the key point. Since in your benchmark's output, OptTxnDb
is about 5-10 times slower than db/TxnDB.

Then, I turned off OptTxnDB's conflict check by direct return Status::OK

Status OptimisticTransaction::CheckTransactionForConflicts(DB* db) {
  Status result;
  // deyukong, line below is added for experiment
  return result;
  auto db_impl = static_cast_with_check<DBImpl, DB>(db);

  // Since we are on the write thread and do not want to block other writers,
  // we will do a cache-only conflict check.  This can result in TryAgain
  // getting returned if there is not sufficient memtable history to check
  // for conflicts.
  return TransactionUtil::CheckKeysForConflicts(db_impl, GetTrackedKeys(),
                                                true /* cache_only */);
}

With AllowWriteBatching=true and no conflict checking, I run the db-bench again and found
OptTxnDB runs as fast as DB.

So it can be concluded that OptTxnDB's conflict check is a heavy operation, and it is done
during commit, and the commit operations are sequential sync point of every thread. Thus
OptTxnDB has a poor performance.

But it needs more investigation to figure out why OptTxnDB's conflict check is a heavy operation.
A simple single-thread in-memory skiplist comparation should has about 10000000-ops' throughput.
So I believe there must be deeper reasons.

TransactionDB doest not suffer from this, it does conflict check during each Put/Get. which can be
parallelized.

---

Updated: change max_write_buffer_number from the default value 4 to 1 also significantly improves
OptTxn's throughput, using your benchmark, The throughput can be improved by 50--70%. It works
because in the conflict check procedure, For each key, every immtable is looked up. In this way, reducing max_write_buffer_num or max_write_buffer_maintain_num works well.

At last, I think OptTxnDB is born slow, There woundn't be any big discovery even if anyone digs deeper. It's implementaion is restricted by its definition. Or at least, I don't
have the imagination to redesign a better architecture to make it as fast as TxnDB.

[wolfkdy]

The proposal of parallel occ for rocksdb
https://groups.google.com/forum/#!topic/rocksdb/KS047z_pQJQ