Julia: Memory Leak in custom system image

[vparadas]

Dear experts,
I have found a memory leak of 72B related to jl_load_dynamic_library:

To reproduce the issue, here is what I did:

• created a custom module
• run the precompilation step
• prepared the points_sysimage.jl which will be used to create the custom system image
• created the .o by using: julia --startup-file=no --cpu-target="x86_64" --output-o points_sys.o -J"$JULIA_PATH/lib/julia/sys.so" points_sysimage.jl
• create the shared library out of the above object file
• call the julia code from C++ code

Julia version is 1.5.0

Could you please look into it?

[yuyichao]

I don't think there's anything to fix here. The library is openned intentionally and since it's used thoughout the lifetime afterwards there's no need to close it.

[vtjnash]

I can't tell if it thinks that it is indicating a glibc bug, or is just confused, but the memory appears to be inside libc and the malloc calls in this code (https://code.woboq.org/userspace/glibc/elf/dl-deps.c.html) seems to probably be generally holding arrays of dependency and relocation information.

[vparadas]

That would explain why we don't see it growing..
I tried the different cases below:

• put the julia_init and jl_atexit_hook in a function and call it 10k times
• call the julia_init once, then call my custom module 10k times, then call jl_atexit_hook once

in both cases the report from valgrind was the same.

Since everything terminates nicely, I would expect to free any used memory.

[vparadas]

@vtjnash I see that you closed the ticket.
Can you please elaborate a bit more on the issue?
It would be helpful to know why this is not a Julia issue. As I phrased on my last comment, I would expect no memory leaks since Julia exits normally. Is this right or I am missing something?

[yuyichao]

No that expectation is not correct. If this is a leak in julia then it's an intentional one as I said. Otherwise if it's an leak in glibc or a bug in leak detection (and I feel like there are a lot) then it's not a julia issue.

[vparadas]

So is it an intentional or not? And how can we see if it is the one or the other?

[yuyichao]

As I said many times, it's intentional on the julia side. Whether the particular allocation you've detected is intentional or not is not a question for julia.

[el-oso]

Hi @vtjnash and @yuyichao

I asked @vparadas to submit this issue after he ran into it at work. It is indeed not a biggie and before he submitted it I already told him that sounded more like a glibc problem rather than a Julia one. However, we wanted to be diligent and contribute this finding to the community just in case it was something worse.

It seems that @vtjnash confirmed that it is inside glibc, so that is a bit reassuring because indeed is doesn't seem to be on the Julia side. However, this "leak" looks bad for Julia (whether it is it's fault or not) and just want to understand what's going on in order to explain to other people at work.

@yuyichao explanation is a bit confusing for me and I understood 4 possible explanations, so please correct me:

1. It's an intentional "leak" <- I don't understand why a leak would be intentional, doesn't sound like a good idea
2. It's not a "leak", it's memory allocated by glibc and since probably it's going to be used afterwards it's not released.
3. It's a "leak" but from glibc, therefore there's nothing we can do about it
4. It's memory allocated by glibc, not by Julia and valgrind (bug leaking detection) is getting confused. Therefore is a bug in valgrind.

You both seem to have more experience with this, so I just want to learn from you. If it's 2,3 or 4 I will be happy with a confirmation and a short explanation if it's not too much trouble. If it's 1 could you tell me why a design like this was chosen on the Julia side?

Thanks again!

[yuyichao]

I don't understand why a leak would be intentional

Because there's no point to do cleanup sometimes and as I said the handle IS intentionally leaked.

[jonas-schulze]

IIUC, since the object is used for the whole lifetime of the Julia process, it is safe to defer cleanup to the OS.

[vtjnash]

@yuyichao you might be correct in general, but this in fact likely a real bug (just not ours). This was a known bug in glibc back in 2013 (http://sourceware-org.1504.n7.nabble.com/Bug-dynamic-link-15859-New-Memory-leak-detected-in-dl-map-object-deps-td241543.html). It wasn't fixed until 2.19 (http://sourceware-org.1504.n7.nabble.com/Bug-dynamic-link-15859-New-Memory-leak-detected-in-dl-map-object-deps-tp241543p260262.html). I assume you're on Centos 7, since the stacktrace above shows you are still using glibc 2.17, while most of the rest of the distros appear to have already reached EOL on this version.

[yuyichao]

but this in fact likely a real bug (just not ours)

That's why I keep saying it can be either.

[vparadas]

@yuyichao you might be correct in general, but this in fact likely a real bug (just not ours). This was a known bug in glibc back in 2013 (http://sourceware-org.1504.n7.nabble.com/Bug-dynamic-link-15859-New-Memory-leak-detected-in-dl-map-object-deps-td241543.html). It wasn't fixed until 2.19 (http://sourceware-org.1504.n7.nabble.com/Bug-dynamic-link-15859-New-Memory-leak-detected-in-dl-map-object-deps-tp241543p260262.html). I assume you're on Centos 7, since the stacktrace above shows you are still using glibc 2.17, while most of the rest of the distros appear to have already reached EOL on this version.

Thanks for the further info @vtjnash!
Actually we are on RHEL 7 (but it is the same as CentOS 7).
The links are helping a lot!

@jonas-schulze the "defer cleanup to the OS" sounds like a solution, but are there any examples?
I am not familiar with it..

[vtjnash]

defer cleanup to the OS

This is always guaranteed. You'll never leak any memory after program exit.

[el-oso]

@vtjnash Thanks! this is very clear. Thanks for taking the time to explain it to us.