Async-profiler: analyze native memory allocation

[yaoliao]

First of all, thanks for this awesome software!

I am using async-profiler to analyze memory allocation. I have some question about the result.

I find method Unsafe_AllocateMemory in jvm only use os::malloc to allocate memory. why I can see mprotect and mmap in stack.

this is my test code:

public class MemoryTest {

    public static void main(String[] args) throws Exception {
        new Thread(() -> {
            while (true) {
                ByteBuffer buffer = ByteBuffer.allocateDirect(1024 * 1024 * 10);
                try {
                    TimeUnit.MILLISECONDS.sleep(100);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            }
        }, "leak_thread").start();

        System.in.read();
    }
}

$./profiler.sh -d 10 -e malloc 30035
Started [malloc] profiling
--- Execution profile ---
Total samples       : 121

Frame buffer usage  : 0.0132%

--- 1006632960 events (99.99%), 96 samples
  [ 0] malloc
  [ 1] os::malloc(unsigned long, MemoryType)
  [ 2] Unsafe_AllocateMemory
  [ 3] sun.misc.Unsafe.allocateMemory
  [ 4] java.nio.DirectByteBuffer.<init>
  [ 5] java.nio.ByteBuffer.allocateDirect
  [ 6] NativeLeakTest.lambda$main$0
  [ 7] NativeLeakTest$$Lambda$1.834600351.run
  [ 8] java.lang.Thread.run

   .....

./profiler.sh -d 10 -e mprotect 30035
Started [mprotect] profiling
--- Execution profile ---
Total samples       : 57

Frame buffer usage  : 0.0026%

--- 53 events (92.98%), 53 samples
  [ 0] __GI___mprotect
  [ 1] sun.misc.Unsafe.allocateMemory
  [ 2] java.nio.DirectByteBuffer.<init>
  [ 3] java.nio.ByteBuffer.allocateDirect
  [ 4] NativeLeakTest.lambda$main$0
  [ 5] NativeLeakTest$$Lambda$1.834600351.run
  [ 6] java.lang.Thread.run



md5-e510eba884614cbe00fc9b2623a3a8de



./profiler.sh -d 10 -e mmap 30035
Started [mmap] profiling
--- Execution profile ---
Total samples       : 12

Frame buffer usage  : 0.0008%

--- 1006632960 events (100.00%), 10 samples
  [ 0] mmap64
  [ 1] sun.misc.Unsafe.allocateMemory
  [ 2] java.nio.DirectByteBuffer.<init>
  [ 3] java.nio.ByteBuffer.allocateDirect
  [ 4] NativeLeakTest.lambda$main$0
  [ 5] NativeLeakTest$$Lambda$1.834600351.run
  [ 6] java.lang.Thread.run



md5-16526ae816853b8e93b604893cb604be



UNSAFE_ENTRY(jlong, Unsafe_AllocateMemory(JNIEnv *env, jobject unsafe, jlong size))
  UnsafeWrapper("Unsafe_AllocateMemory");
  size_t sz = (size_t)size;
  if (sz != (julong)size || size < 0) {
    THROW_0(vmSymbols::java_lang_IllegalArgumentException());
  }
  if (sz == 0) {
    return 0;
  }
  sz = round_to(sz, HeapWordSize);
  void* x = os::malloc(sz, mtInternal);
  if (x == NULL) {
    THROW_0(vmSymbols::java_lang_OutOfMemoryError());
  }
  //Copy::fill_to_words((HeapWord*)x, sz / HeapWordSize);
  return addr_to_java(x);
UNSAFE_END

[franz1981]

Please there is another issue re this where @apangin has already answered why and how :)

[franz1981]

https://github.com/jvm-profiling-tools/async-profiler/issues/336

[krzysztofslusarski]

One hint from me, if you want to find places in your system that really consumes your RAM you can profile page-faults.

[apangin]

As I explained in this video, malloc implementation calls mmap and mprotect under the hood to extend native memory pool.

libc (which malloc is a part of) is usually compiled without frame pointers (-fomit-frame-pointer optimization), that's why async-profiler can't show that mprotect/mmap is actually called by malloc.

[yaoliao]

As I explained in this video, malloc implementation calls mmap and mprotect under the hood to extend native memory pool.

libc (which malloc is a part of) is usually compiled without frame pointers (-fomit-frame-pointer optimization), that's why async-profiler can't show that mprotect/mmap is actually called by malloc.

Thank you very much for your reply. The video is very good.