Rocksdb: operator[] delete, operator[] new in lru_cache make tcmalloc report invalid free

[mdcallag]

This is easy to reproduce. Build FB MySQL with MyRocks and tcmalloc, then run mysqld --help --verbose and tcmalloc reports an invalid free. I get this using a MyRocks build from November 3. This occurs when ~LRUCache does delete on shards_ and then LRUCacheShard::operator delete[] is called.

I don't know modern C++, but removing operator new[] and operator delete[] as in this diff fixes it. I tried this change because I was curious about calling free and new from within new[] and delete[]. At that point the code is the same as for the overloaded operator new and delete in lru_cache.cc

diff --git a/cache/lru_cache.cc b/cache/lru_cache.cc
index e4ab406..28bc29a 100644
--- a/cache/lru_cache.cc
+++ b/cache/lru_cache.cc
@@ -239,17 +239,17 @@ void* LRUCacheShard::operator new(size_t size) {
   return port::cacheline_aligned_alloc(size);
 }

-void* LRUCacheShard::operator new[](size_t size) {
-  return port::cacheline_aligned_alloc(size);
-}
+// void* LRUCacheShard::operator new[](size_t size) {
+//  return port::cacheline_aligned_alloc(size);
+//}

 void LRUCacheShard::operator delete(void *memblock) {
   port::cacheline_aligned_free(memblock);
 }

-void LRUCacheShard::operator delete[](void* memblock) {
-  port::cacheline_aligned_free(memblock);
-}
+//void LRUCacheShard::operator delete[](void* memblock) {
+//  port::cacheline_aligned_free(memblock);
+//}

 void LRUCacheShard::SetCapacity(size_t capacity) {
   autovector<LRUHandle*> last_reference_list;
diff --git a/cache/lru_cache.h b/cache/lru_cache.h
index abe78fd..2fd327b 100644
--- a/cache/lru_cache.h
+++ b/cache/lru_cache.h
@@ -205,11 +205,11 @@ class ALIGN_AS(CACHE_LINE_SIZE) LRUCacheShard : public CacheShard {
   // Overloading to aligned it to cache line size
   void* operator new(size_t);

-  void* operator new[](size_t);
+  // void* operator new[](size_t);

   void operator delete(void *);

-  void operator delete[](void*);
+  // void operator delete[](void*);

Again, I don't know modern C++ but I was confused by something. The value of shards_ is 0x2592600 but the argument to LRUCacheShard::operator delete[] (see memblock) was 0x25925c0.

From the error this is a stack trace for the LRUCache ctor including the value of this and *this

#0  rocksdb::LRUCache::LRUCache (this=0x3b140f0, capacity=8388608, num_shard_bits=4, strict_capacity_limit=false, high_pri_pool_ratio=0)
    at /data/users/mcallaghan/mysql.me/5.6/rocksdb/cache/lru_cache.cc:471
#1  0x0000000000bf9e97 in __gnu_cxx::new_allocator<rocksdb::LRUCache>::construct<rocksdb::LRUCache, unsigned long&, int&, bool&, double&> (__p=<optimized out>,
    this=<optimized out>) at /mnt/gvfs/third-party2/libgcc/76232aa2a673967fb27600d6c8d360cd6a62a6e2/5.x/gcc-5-glibc-2.23/339d858/include/c++/5.x/ext/new_allocator.h:120
#2  std::allocator_traits<std::allocator<rocksdb::LRUCache> >::construct<rocksdb::LRUCache, unsigned long&, int&, bool&, double&> (__p=<optimized out>, __a=...)
    at /mnt/gvfs/third-party2/libgcc/76232aa2a673967fb27600d6c8d360cd6a62a6e2/5.x/gcc-5-glibc-2.23/339d858/include/c++/5.x/bits/alloc_traits.h:517
#3  std::_Sp_counted_ptr_inplace<rocksdb::LRUCache, std::allocator<rocksdb::LRUCache>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<unsigned long&, int&, bool&, double&> (__a=..., this=0x3b140e0)
    at /mnt/gvfs/third-party2/libgcc/76232aa2a673967fb27600d6c8d360cd6a62a6e2/5.x/gcc-5-glibc-2.23/339d858/include/c++/5.x/bits/shared_ptr_base.h:522
#4  std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<rocksdb::LRUCache, std::allocator<rocksdb::LRUCache>, unsigned long&, int&, bool&, double&> (__a=...,
    this=<optimized out>) at /mnt/gvfs/third-party2/libgcc/76232aa2a673967fb27600d6c8d360cd6a62a6e2/5.x/gcc-5-glibc-2.23/339d858/include/c++/5.x/bits/shared_ptr_base.h:617
#5  std::__shared_ptr<rocksdb::LRUCache, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<rocksdb::LRUCache>, unsigned long&, int&, bool&, double&> (__a=...,
    __tag=..., this=<optimized out>)
    at /mnt/gvfs/third-party2/libgcc/76232aa2a673967fb27600d6c8d360cd6a62a6e2/5.x/gcc-5-glibc-2.23/339d858/include/c++/5.x/bits/shared_ptr_base.h:1097
#6  std::shared_ptr<rocksdb::LRUCache>::shared_ptr<std::allocator<rocksdb::LRUCache>, unsigned long&, int&, bool&, double&> (__a=..., __tag=..., this=<optimized out>)
    at /mnt/gvfs/third-party2/libgcc/76232aa2a673967fb27600d6c8d360cd6a62a6e2/5.x/gcc-5-glibc-2.23/339d858/include/c++/5.x/bits/shared_ptr.h:319
#7  std::allocate_shared<rocksdb::LRUCache, std::allocator<rocksdb::LRUCache>, unsigned long&, int&, bool&, double&> (__a=...)
    at /mnt/gvfs/third-party2/libgcc/76232aa2a673967fb27600d6c8d360cd6a62a6e2/5.x/gcc-5-glibc-2.23/339d858/include/c++/5.x/bits/shared_ptr.h:620
#8  std::make_shared<rocksdb::LRUCache, unsigned long&, int&, bool&, double&> ()
    at /mnt/gvfs/third-party2/libgcc/76232aa2a673967fb27600d6c8d360cd6a62a6e2/5.x/gcc-5-glibc-2.23/339d858/include/c++/5.x/bits/shared_ptr.h:636
#9  rocksdb::NewLRUCache (capacity=capacity@entry=8388608, num_shard_bits=4, num_shard_bits@entry=-1, strict_capacity_limit=strict_capacity_limit@entry=false,
    high_pri_pool_ratio=high_pri_pool_ratio@entry=0) at /data/users/mcallaghan/mysql.me/5.6/rocksdb/cache/lru_cache.cc:529
#10 0x0000000000d0c880 in rocksdb::BlockBasedTableFactory::BlockBasedTableFactory (this=0x28e8820, _table_options=...)
    at /data/users/mcallaghan/mysql.me/5.6/rocksdb/table/block_based_table_factory.cc:44
#11 0x0000000000cf52fe in rocksdb::ColumnFamilyOptions::ColumnFamilyOptions (this=0x2605918) at /data/users/mcallaghan/mysql.me/5.6/rocksdb/options/options.cc:99
#12 0x0000000000b922ea in myrocks::rocksdb_init_func (p=<optimized out>) at /data/users/mcallaghan/mysql.me/5.6/storage/rocksdb/././rdb_cf_options.h:47
#13 0x0000000000778778 in ha_initialize_handlerton (plugin=0x3afad78) at /data/users/mcallaghan/mysql.me/5.6/sql/handler.cc:655
#14 0x0000000000901e7b in plugin_initialize (plugin=plugin@entry=0x3afad78) at /data/users/mcallaghan/mysql.me/5.6/sql/sql_plugin.cc:1163
#15 0x0000000000902b28 in plugin_init (argc=argc@entry=0x1bf5368 <remaining_argc>, argv=0x265f1e0, flags=flags@entry=0)
    at /data/users/mcallaghan/mysql.me/5.6/sql/sql_plugin.cc:1465
#16 0x0000000000768375 in init_server_components () at /data/users/mcallaghan/mysql.me/5.6/sql/mysqld.cc:5851
#17 0x000000000076ba25 in mysqld_main (argc=127, argv=0x265f1e0) at /data/users/mcallaghan/mysql.me/5.6/sql/mysqld.cc:7198
#18 0x00007ffff59ac858 in __libc_start_main () from /usr/local/fbcode/gcc-5-glibc-2.23/lib/libc.so.6
#19 0x00000000007343e9 in _start () at ../sysdeps/x86_64/start.S:118

(gdb) p *this
$9 = {<rocksdb::ShardedCache> = {<rocksdb::Cache> = {_vptr.Cache = 0x12e2480 <vtable for rocksdb::LRUCache+16>}, num_shard_bits_ = 4, capacity_mutex_ = {mu_ = {__data = {
          __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}},
        __size = '\000' <repeats 39 times>, __align = 0}}, capacity_ = 8388608, strict_capacity_limit_ = false, last_id_ = {<std::__atomic_base<unsigned long>> = {
        static _S_alignment = 8, _M_i = 1}, <No data fields>}}, shards_ = 0x2592600, num_shards_ = 16}
(gdb) p this
$10 = (rocksdb::LRUCache * const) 0x3b140f0

And then this is the stack trace from the invalid free. Note the value of the memblock arg in the call to operator delete[]

#0  0x00007ffff59c17c8 in raise () from /usr/local/fbcode/gcc-5-glibc-2.23/lib/libc.so.6
#1  0x00007ffff59c3591 in abort () from /usr/local/fbcode/gcc-5-glibc-2.23/lib/libc.so.6
#2  0x0000000000738f21 in tcmalloc::Log (mode=mode@entry=tcmalloc::kCrash, filename=filename@entry=0x1181956 "src/tcmalloc.cc", line=line@entry=284, a=..., b=..., c=...,
    d=...) at src/internal_logging.cc:118
#3  0x0000000000734514 in (anonymous namespace)::InvalidFree (ptr=<optimized out>) at src/tcmalloc.cc:284
#4  0x0000000000bf8424 in rocksdb::LRUCacheShard::operator delete[] (memblock=0x25925c0) at /data/users/mcallaghan/mysql.me/5.6/rocksdb/cache/lru_cache.cc:251
#5  rocksdb::LRUCache::~LRUCache (this=0x3b140f0, __in_chrg=<optimized out>) at /data/users/mcallaghan/mysql.me/5.6/rocksdb/cache/lru_cache.cc:479
#6  0x000000000076dfe6 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x3b140e0)
    at /mnt/gvfs/third-party2/libgcc/76232aa2a673967fb27600d6c8d360cd6a62a6e2/5.x/gcc-5-glibc-2.23/339d858/include/c++/5.x/bits/shared_ptr_base.h:150
#7  0x0000000000cf6719 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x28e8848, __in_chrg=<optimized out>)
    at /mnt/gvfs/third-party2/libgcc/76232aa2a673967fb27600d6c8d360cd6a62a6e2/5.x/gcc-5-glibc-2.23/339d858/include/c++/5.x/bits/shared_ptr_base.h:659
#8  std::__shared_ptr<rocksdb::Cache, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x28e8840, __in_chrg=<optimized out>)
    at /mnt/gvfs/third-party2/libgcc/76232aa2a673967fb27600d6c8d360cd6a62a6e2/5.x/gcc-5-glibc-2.23/339d858/include/c++/5.x/bits/shared_ptr_base.h:925
#9  std::shared_ptr<rocksdb::Cache>::~shared_ptr (this=0x28e8840, __in_chrg=<optimized out>)
    at /mnt/gvfs/third-party2/libgcc/76232aa2a673967fb27600d6c8d360cd6a62a6e2/5.x/gcc-5-glibc-2.23/339d858/include/c++/5.x/bits/shared_ptr.h:93
#10 rocksdb::BlockBasedTableOptions::~BlockBasedTableOptions (this=0x28e8828, __in_chrg=<optimized out>)
    at /data/users/mcallaghan/mysql.me/5.6/rocksdb/include/rocksdb/table.h:52
#11 rocksdb::BlockBasedTableFactory::~BlockBasedTableFactory (this=0x28e8820, __in_chrg=<optimized out>)
    at /data/users/mcallaghan/mysql.me/5.6/rocksdb/table/block_based_table_factory.h:34
#12 rocksdb::BlockBasedTableFactory::~BlockBasedTableFactory (this=0x28e8820, __in_chrg=<optimized out>)
    at /data/users/mcallaghan/mysql.me/5.6/rocksdb/table/block_based_table_factory.h:34
#13 std::_Sp_counted_ptr<rocksdb::BlockBasedTableFactory*, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=<optimized out>)
    at /mnt/gvfs/third-party2/libgcc/76232aa2a673967fb27600d6c8d360cd6a62a6e2/5.x/gcc-5-glibc-2.23/339d858/include/c++/5.x/bits/shared_ptr_base.h:374
#14 0x0000000000bcf5da in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x26ad080)
    at /mnt/gvfs/third-party2/libgcc/76232aa2a673967fb27600d6c8d360cd6a62a6e2/5.x/gcc-5-glibc-2.23/339d858/include/c++/5.x/bits/shared_ptr_base.h:150
#15 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=<optimized out>, __in_chrg=<optimized out>)
    at /mnt/gvfs/third-party2/libgcc/76232aa2a673967fb27600d6c8d360cd6a62a6e2/5.x/gcc-5-glibc-2.23/339d858/include/c++/5.x/bits/shared_ptr_base.h:659
#16 std::__shared_ptr<rocksdb::TableFactory, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=<optimized out>, __in_chrg=<optimized out>)
    at /mnt/gvfs/third-party2/libgcc/76232aa2a673967fb27600d6c8d360cd6a62a6e2/5.x/gcc-5-glibc-2.23/339d858/include/c++/5.x/bits/shared_ptr_base.h:925
#17 std::__shared_ptr<rocksdb::TableFactory, (__gnu_cxx::_Lock_policy)2>::reset<rocksdb::TableFactory> (__p=<optimized out>, this=0x2605ad8)
    at /mnt/gvfs/third-party2/libgcc/76232aa2a673967fb27600d6c8d360cd6a62a6e2/5.x/gcc-5-glibc-2.23/339d858/include/c++/5.x/bits/shared_ptr_base.h:1030
#18 myrocks::Rdb_cf_options::init (this=0x26058c0, table_options=..., prop_coll_factory=...,
    default_cf_options=0x263fc95 "write_buffer_size=64m;target_file_size_base=32m;max_bytes_for_level_base=512m;level0_file_num_compaction_trigger=4;level0_slowdown_writes_trigger=20;level0_stop_writes_trigger=30;max_write_buffer_numb"..., override_cf_options=0x12d4c5f "")
    at /data/users/mcallaghan/mysql.me/5.6/storage/rocksdb/rdb_cf_options.cc:56
#19 0x0000000000b9234b in myrocks::rocksdb_init_func (p=<optimized out>) at /data/users/mcallaghan/mysql.me/5.6/storage/rocksdb/ha_rocksdb.cc:4152
#20 0x0000000000778778 in ha_initialize_handlerton (plugin=0x3afad78) at /data/users/mcallaghan/mysql.me/5.6/sql/handler.cc:655
#21 0x0000000000901e7b in plugin_initialize (plugin=plugin@entry=0x3afad78) at /data/users/mcallaghan/mysql.me/5.6/sql/sql_plugin.cc:1163
#22 0x0000000000902b28 in plugin_init (argc=argc@entry=0x1bf5368 <remaining_argc>, argv=0x265f1e0, flags=flags@entry=0)
    at /data/users/mcallaghan/mysql.me/5.6/sql/sql_plugin.cc:1465
#23 0x0000000000768375 in init_server_components () at /data/users/mcallaghan/mysql.me/5.6/sql/mysqld.cc:5851
#24 0x000000000076ba25 in mysqld_main (argc=127, argv=0x265f1e0) at /data/users/mcallaghan/mysql.me/5.6/sql/mysqld.cc:7198
#25 0x00007ffff59ac858 in __libc_start_main () from /usr/local/fbcode/gcc-5-glibc-2.23/lib/libc.so.6
#26 0x00000000007343e9 in _start () at ../sysdeps/x86_64/start.S:118
(gdb) frame 5
#5  rocksdb::LRUCache::~LRUCache (this=0x3b140f0, __in_chrg=<optimized out>) at /data/users/mcallaghan/mysql.me/5.6/rocksdb/cache/lru_cache.cc:479
479     in /data/users/mcallaghan/mysql.me/5.6/rocksdb/cache/lru_cache.cc
(gdb) p this
$16 = (rocksdb::LRUCache * const) 0x3b140f0

#3  0x0000000000734514 in (anonymous namespace)::InvalidFree (ptr=<optimized out>) at src/tcmalloc.cc:284
#4  0x0000000000bf8424 in rocksdb::LRUCacheShard::operator delete[] (memblock=0x25925c0) at /data/users/mcallaghan/mysql.me/5.6/rocksdb/cache/lru_cache.cc:251

[maysamyabandeh]

@yiwu-arbug could you please chime in?

[yiwu-arbug]

I'm not familiar with tcmalloc. The reason to override new[] and delete[] is to have it call aligned_malloc instead of normal malloc. A quick search didn't find if tcmalloc support aligned_malloc or how can we change code to support it.

[mdcallag]

My question isn't about the need to override new[] and delete[]. My question is whether the current code overrides them correctly.

[yiwu-arbug]

@mdcallag do you happen to know where I can find documents and source code of tcmalloc? I can only find an introduction article about it.

[yiwu-arbug]

AFAIK the override operation should be correct, but the implementation ofport::cacheline_aligned_alloc probably doesn't handle tcmalloc right https://github.com/facebook/rocksdb/blob/master/port/port_posix.cc#L187-L198
Maybe tcmalloc doesn't support aligned_alloc, or it support similar functionality by implementing another function. But we should at least add a macro to disable aligned_malloc, which we currently don't have.

[tulshiba]

I've not looked thoroughly into your issue but looks similar to this http://tracker.ceph.com/issues/21422 and the issue is further tracked here - https://bugzilla.redhat.com/show_bug.cgi?id=1494309

[yiwu-arbug]

@tulshiba Thanks much for the pointer. The latter link confirm it is a tcmalloc bug fixed in gperftools-2.6.1-5.fc26 and gperftools-2.6.1-5.fc27. @mdcallag Do you mind trying a newer version of tcmalloc that come with gperftools?

[mdcallag]

Thanks for saving us a lot of time @tulshiba
Yes, @yiwu-arbug, I will try that.

[yiwu-arbug]

Closing for now. Reopen if there are additional issue.