Tikv: jemalloc will fetch backtrace at every malloc

[YangKeao]

Bug Report

What version of TiKV are you using?

Release v5.0.1

Steps to reproduce

1. tiup playground v5.0.1 setups a tikv cluster
2. gdb attachs the running tikv, and set a breakpoint with break uw_frame_state_for
3. run anything like connecting with mysql
4. Then the breakpoints hit and the backtrace shows the call stack is inside the jemalloc

#0  uw_frame_state_for (context=0x7f57853d4390, fs=0x7f57853d4480) at ../../../libgcc/unwind-dw2.c:1256
#1  0x00007f578d6e31b4 in _Unwind_Backtrace (trace=trace@entry=0x55e0ee63da10 <prof_unwind_callback>, 
    trace_argument=trace_argument@entry=0x7f57853d4640) at ../../../libgcc/unwind.inc:302
#2  0x000055e0ee648b16 in _rjem_je_prof_backtrace (bt=bt@entry=0x7f57853d4680) at src/prof.c:637
#3  0x000055e0ee5ec5d5 in prof_alloc_prep (update=true, _rjem_je_prof_active=<optimized out>, usize=<optimized out>, tsd=<optimized out>)
    at include/jemalloc/internal/prof_inlines_b.h:158
#4  imalloc_body (tsd=<optimized out>, dopts=<synthetic pointer>, sopts=<synthetic pointer>) at src/jemalloc.c:2116
#5  imalloc (dopts=<synthetic pointer>, sopts=<synthetic pointer>) at src/jemalloc.c:2258
#6  _rjem_je_malloc_default (size=<optimized out>) at src/jemalloc.c:2289
#7  0x000055e0ee9c0c38 in operator new(unsigned long) ()
#8  0x000055e0ee7109f6 in rocksdb::LRUHandleTable::Resize (this=0x7f5783c47a80)
    at /rust/git/checkouts/rust-rocksdb-a9a28e74c6ead8ef/1b3a613/librocksdb_sys/rocksdb/cache/lru_cache.cc:77
#9  0x000055e0ee71157e in rocksdb::LRUCacheShard::LRUCacheShard (this=0x7f5783c47a00, capacity=524288, 
    strict_capacity_limit=<optimized out>, high_pri_pool_ratio=<optimized out>, use_adaptive_mutex=<optimized out>)
    at /rust/git/checkouts/rust-rocksdb-a9a28e74c6ead8ef/1b3a613/librocksdb_sys/rocksdb/cache/lru_cache.cc:108
#10 0x000055e0ee7116e1 in rocksdb::LRUCache::LRUCache (this=0x7f5783c21190, capacity=<optimized out>, num_shard_bits=<optimized out>, 
    strict_capacity_limit=false, high_pri_pool_ratio=0, allocator=..., use_adaptive_mutex=<optimized out>)
    at /rust/git/checkouts/rust-rocksdb-a9a28e74c6ead8ef/1b3a613/librocksdb_sys/rocksdb/cache/lru_cache.cc:460
#11 0x000055e0ee71185f in __gnu_cxx::new_allocator<rocksdb::LRUCache>::construct<rocksdb::LRUCache, unsigned long&, int&, bool&, double&, std::shared_ptr<rocksdb::MemoryAllocator>, bool&> (__p=0x7f5783c21190, this=<optimized out>)
    at /opt/rh/devtoolset-8/root/usr/include/c++/8/new:169
#12 std::allocator_traits<std::allocator<rocksdb::LRUCache> >::construct<rocksdb::LRUCache, unsigned long&, int&, bool&, double&, std::shared_ptr<rocksdb::MemoryAllocator>, bool&> (__p=0x7f5783c21190, __a=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/alloc_traits.h:475
#13 std::_Sp_counted_ptr_inplace<rocksdb::LRUCache, std::allocator<rocksdb::LRUCache>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<unsigned long&, int&, bool&, double&, std::shared_ptr<rocksdb::MemoryAllocator>, bool&> (__a=..., this=0x7f5783c21180)
    at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/shared_ptr_base.h:545

This could be the reason for #10131 . And fetching backtrace all the time will make it impossible to run pprof-rs.

What did you expect?

Don't fetch backtrace all the time :imp:

[YangKeao]

Or should I open an issue on tiup?

[BusyJay]

That's strange as the function will only be called when prof_active is set to true, which can be configured by either environment variables or API in status server.

[YangKeao]

That's strange as the function will only be called when prof_active is set to true, which can be configured by either environment variables or API in status server.

@BusyJay Yes. In the latest version of tiup, prof_active is set to true, (and that's why I opened this issue, exactly, to talk about whether it should be set to true by default):

https://github.com/pingcap/tiup/blob/52e24c2ee45e6a2acc3bd59990a4f1ebbedcd633/embed/templates/scripts/run_tikv.sh.tpl#L23

export MALLOC_CONF="prof:true,prof_active:true,prof.active:false"

[BusyJay]

No, it should not.

[BusyJay]

Instead of removing it completely, better use "prof:true,prof_active:false" to make it possible to profile memory without restarting.