Lwjgl3: Crash with JEmalloc

[sriharshachilakapati]

I'm seeing a weird crash when using JEmalloc. The crash was EXCEPTION_ACCESS_VIOLATION and occurs when I request to allocate a memory of 4096 bytes. Dunno why that fails.

The same happens when I use MemoryUtil.memAlloc(4096), I think it is an alias for the JEmalloc functions. It must not be an issue from my system either, because if I use the BufferUtils class, it works pretty fine.

I have attached the complete log file for inspection here.

hs_err_pid4772.log

[Spasi]

I cannot reproduce this.

• Are you running the latest LWJGL nightly build?
• Is the 4096 value important? Does it also happen with other sizes?
• What's the output if you run with -Dorg.lwjgl.util.Debug=true?
• What's the output if you run with -Dorg.lwjgl.system.allocator=system?

[sriharshachilakapati]

Output with -Dorg.lwjgl.util.Debug=true

[LWJGL] Version: 3.0.0 build 78
[LWJGL]      OS: Windows 8.1 v6.3
[LWJGL]     JRE: 1.8.0_51 amd64
[LWJGL]     JVM: Java HotSpot(TM) 64-Bit Server VM v25.51-b03 by Oracle Corporation
[LWJGL] Shared library extract path: C:\Users\Harsha\AppData\Local\Temp\lwjglHarsha\7a5bd1ea
[LWJGL] Loaded library from org.lwjgl.librarypath: lwjgl
[LWJGL] ThreadLocalUtil state: UnsafeState
[LWJGL] MemoryUtil accessor: MemoryAccessorUnsafe
[LWJGL] Loaded native library: C:\Users\Harsha\AppData\Local\Temp\lwjglHarsha\7a5bd1ea\glfw.dll
[LWJGL] Loaded native library: C:\WINDOWS\system32\opengl32.dll
[LWJGL] [GL] GL_ARB_shader_subroutine was reported as available but an entry point is missing.
[LWJGL] Loaded native library: C:\Users\Harsha\AppData\Local\Temp\lwjglHarsha\7a5bd1ea\OpenAL.dll
[LWJGL] Loaded native library: C:\Users\Harsha\AppData\Local\Temp\lwjglHarsha\7a5bd1ea\jemalloc.dll
[LWJGL] MemoryUtil allocator: JEmallocAllocator

When used with -Dorg.lwjgl.system.allocator=system everything runs as expected. However, the loaded data is corrupt.

Exception in thread "Thread-1" com.shc.silenceengine.core.SilenceException: Failed to load image: Image not of any known type, or corrupt
    at com.shc.silenceengine.backend.lwjgl.LwjglImageReader.lambda$readImage$1(LwjglImageReader.java:57)
    at com.shc.silenceengine.backend.lwjgl.LwjglImageReader$$Lambda$65/1335050193.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:745)

I'm allocating this buffer to store the bytes from the File and pass them to STB image. This is not happening with BufferUtils class usage.

Should I send you the source code I was using? I can also create a runnable JAR for this.

[sriharshachilakapati]

Wait, the image issue is in my own image, the image is corrupted somehow. Using another image worked as expected. The JEmalloc issue still remains.

[Spasi]

Should I send you the source code I was using?

Yes please, a short self-contained example that reproduces the issue would be perfect.

[sriharshachilakapati]

I'm unable to reproduce this with simple example, but the issue persists in my game somewhere. Can I use JEmalloc to allocate memory from other threads?

[Spasi]

I'm unable to reproduce this with simple example, but the issue persists in my game somewhere.

You could try the opposite way: simplify your game until it's simple enough to share or the crash stops happening.

Can I use JEmalloc to allocate memory from other threads?

Yes, you can.

[sriharshachilakapati]

I just found something else, but related to this issue. This was caused indeed on my way, I forgot to deallocate some buffers (assumed that OS will deallocate memory after JVM exits). But however, I notice that the JVM (java.exe) is still running even though IDEA reported that process terminated successfully with error 0, and that is using all the memory, and force quitting it got my data back.

However, this is not happening if I use system allocator, or BufferUtils. This is the small test that I used to check the behaviour (not sure if there are any issues).

final long[] totalMemory = { 0 };
Random random = new Random();

final ByteBuffer[] buffer = { null };

IntStream.range(0, 100).forEach(i ->
{
    int size = Math.abs(random.nextInt());

    if (buffer[0] != null)
        MemoryUtil.memFree(buffer[0]);

    System.out.println("Total Memory Allocated: " + totalMemory[0] + " bytes");
    System.out.println("Requesting " + size + " bytes");

    totalMemory[0] += size;

    buffer[0] = MemoryUtil.memAlloc(size);
});

MemoryUtil.memFree(buffer[0]);

I'll be more cautious from now on this issue, and remember to free up all the unnecessary memory. However, I don't understand why the JVM keeps running. I'm using Windows 10 64 bit and Java 8 update 73. Can you kindly explain what's going on?

[Spasi]

Can you kindly explain what's going on?

The fact that it happens with jemalloc and not with the system allocator makes no sense to me.

It's a good practice, but you don't really have to deallocate everything before exiting your program. The memory is freed when the process ends, regardless of the allocator used.

If your process doesn't end, there's nothing in jemalloc that would cause that. The only way the JVM stays alive is if a non-daemon Java thread is still running.

I also tried the above program (from the command line and from inside IDEA) and can't see any issues. What do you see in your system?

[Spasi]

Hey @sriharshachilakapati,

I'm fairly sure I found and fixed the issue and it indeed was triggered by concurrent allocations. Could you please download and try the latest nightly build (87) to verify?

For those using custom jemalloc builds, make sure to build with --enable-lazy-lock=no on Windows.

[sriharshachilakapati]

Thanks spasi. I'm currently on my way home, will let you know in an hour.

On Sun, 22 May 2016 8:18 pm Ioannis Tsakpinis, [email protected]
wrote:

Hey @sriharshachilakapati https://github.com/sriharshachilakapati,

I'm fairly sure I found and fixed the issue and it indeed was triggered by
concurrent allocations. Could you please download and try the latest
nightly build (87) to verify?

For those using custom jemalloc builds, make sure to build with
--enable-lazy-lock=no on Windows.

—
You are receiving this because you were mentioned.

Reply to this email directly or view it on GitHub
https://github.com/LWJGL/lwjgl3/issues/188#issuecomment-220836320

[sriharshachilakapati]

Thank you! This is working pretty much fine now. It was tough to reproduce this. I'm curious, can you please tell me how did you find this out?

[Spasi]

Awesome, thanks.

I'm curious, can you please tell me how did you find this out?

First step was reproducing it. I recently updated jemalloc to the latest version (4.2.0) and wanted to see if there were any performance improvements. I added a malloc benchmark to the jmh-experiments branch and it was failing randomly with 3-4 or more threads allocating at the same time. I then replaced jemalloc with a debug build and inspected the crash logs; it was failing all over the library, in a different function each time. This suggested a fundamental concurrency-related problem. Going through the jemalloc issues, I finally found this. Did another build without lazy locking and everything was fine.