Adios2: Memory Leak

[AaronV77]

Hey Everyone,

I got ADIOS2 all implemented into our project and wanted to make sure that the memory was all correct, but I found a memory error. The memory error is occurring just in the adios2_init() function call.

Here is my snippet of simple code:

#include <stdio.h>
#include <mpi.h>
#include <stdlib.h>
#include <adios2_c.h>

int main() {

    int world_rank, world_size;
    MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
    MPI_Comm_size(MPI_COMM_WORLD, &world_size);

    adios2_adios *adios = adios2_init(MPI_COMM_WORLD, adios2_debug_mode_on);
    adios2_finalize(adios);

    return 0;
}

Here is how I am debugging the code: mpirun -np 1 valgrind --leak-check=full --show-reachable=yes ./a.out

Here is the output from said command:

==26623== Memcheck, a memory error detector
==26623== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==26623== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==26623== Command: ./a.out
==26623== 
Attempting to use an MPI routine before initializing MPICH
==26623== 
==26623== HEAP SUMMARY:
==26623==     in use at exit: 96 bytes in 2 blocks
==26623==   total heap usage: 60 allocs, 58 frees, 83,524 bytes allocated
==26623== 
==26623== 32 bytes in 1 blocks are still reachable in loss record 1 of 2
==26623==    at 0x4C3204A: calloc (vg_replace_malloc.c:762)
==26623==    by 0x61A17E4: _dlerror_run (dlerror.c:140)
==26623==    by 0x61A1165: dlsym (dlsym.c:70)
==26623==    by 0x63A5D5E: assign_function_pointers() (in /usr/local/lib/libtaustubs.so)
==26623==    by 0x63A5F18: tau_stub_initialize_simple_() (in /usr/local/lib/libtaustubs.so)
==26623==    by 0x63A5FF0: taustubs::TauTimer::TauTimer() (in /usr/local/lib/libtaustubs.so)
==26623==    by 0x63A6084: taustubs::TauTimer::get() (in /usr/local/lib/libtaustubs.so)
==26623==    by 0x63A5BA3: initialize_library() (in /usr/local/lib/libtaustubs.so)
==26623==    by 0x4010732: call_init (dl-init.c:72)
==26623==    by 0x4010732: _dl_init (dl-init.c:119)
==26623==    by 0x40010C9: ??? (in /lib/x86_64-linux-gnu/ld-2.27.so)
==26623== 
==26623== 64 bytes in 1 blocks are still reachable in loss record 2 of 2
==26623==    at 0x4C2FDFB: malloc (vg_replace_malloc.c:309)
==26623==    by 0x4017AA7: _dl_exception_create_format (dl-exception.c:130)
==26623==    by 0x400B3EB: _dl_lookup_symbol_x (dl-lookup.c:853)
==26623==    by 0x5CF4DA5: do_sym (dl-sym.c:151)
==26623==    by 0x5CF4DA5: _dl_sym (dl-sym.c:254)
==26623==    by 0x61A10E3: dlsym_doit (dlsym.c:50)
==26623==    by 0x5CF52DE: _dl_catch_exception (dl-error-skeleton.c:196)
==26623==    by 0x5CF536E: _dl_catch_error (dl-error-skeleton.c:215)
==26623==    by 0x61A1734: _dlerror_run (dlerror.c:162)
==26623==    by 0x61A1165: dlsym (dlsym.c:70)
==26623==    by 0x63A5EE2: assign_function_pointers() (in /usr/local/lib/libtaustubs.so)
==26623==    by 0x63A5F18: tau_stub_initialize_simple_() (in /usr/local/lib/libtaustubs.so)
==26623==    by 0x63A5FF0: taustubs::TauTimer::TauTimer() (in /usr/local/lib/libtaustubs.so)
==26623== 
==26623== LEAK SUMMARY:
==26623==    definitely lost: 0 bytes in 0 blocks
==26623==    indirectly lost: 0 bytes in 0 blocks
==26623==      possibly lost: 0 bytes in 0 blocks
==26623==    still reachable: 96 bytes in 2 blocks
==26623==         suppressed: 0 bytes in 0 blocks
==26623== 
==26623== For lists of detected and suppressed errors, rerun with: -s
==26623== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

The error looks to be from a third party source. Lastly, I built Valgrind and MPICH together to be able to debug MPI programs, just a fair warning if you try just using your basic Valgrind package manager install.

Thanks,

AV

[williamfgc]

@khuck @philip-davis could you take a look at this issue? I am not familiar with the underlying TAU integration, your help would be appreciated. I asked @AaronV77 to post it here. Thanks!

[pnorbert]

This is a problem in the example. You are calling MPI functions without
calling MPI_Init().

On Tue, Aug 6, 2019, 8:11 AM William F Godoy [e-mail]
wrote:

@khuck https://github.com/khuck @philip-davis
https://github.com/philip-davis could you take a look at this issue? I
am not familiar with the underlying TAU integration, your help would be
appreciated. I asked @AaronV77 https://github.com/AaronV77 to post it
here. Thanks!

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ornladios/ADIOS2/issues/1663?email_source=notifications&email_token=AAYYYLNYUK77JHNZ4Y5EKZ3QDFZ7RA5CNFSM4IJV422KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3VCUVQ#issuecomment-518662742,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAYYYLNJEHHMDHQQEO3BCDLQDFZ7RANCNFSM4IJV422A
.

[williamfgc]

@pnorbert thanks for the catch. @AaronV77 can you add MPI_Init/MPI_Finalize and check if the leak is gone (MPI leaks might show up, though).

[AaronV77]

My apologies, I tried creating a quick example program for the issue and I also forgot the MPI_Finalize() but the problem still occurs with the Init and Finalize call.

New example:

#include <stdio.h>
#include <mpi.h>
#include <stdlib.h>
#include <adios2_c.h>

int main() {
    MPI_Init(NULL, NULL);
    adios2_adios *adios = adios2_init(MPI_COMM_WORLD, adios2_debug_mode_on);
    adios2_finalize(adios);
    MPI_Finalize();
    return 0;
}

[khuck]

I'll look into it. It looks like it's just leaking function pointers, that would otherwise be freed at exit. But I'll clean it up.

[NAThompson]

Just a quick note: Valgrind's definition of a memory leak is fairly unintuitive:

==26623== LEAK SUMMARY:
==26623==    definitely lost: 0 bytes in 0 blocks
==26623==    indirectly lost: 0 bytes in 0 blocks
==26623==      possibly lost: 0 bytes in 0 blocks
==26623==    still reachable: 96 bytes in 2 blocks
==26623==         suppressed: 0 bytes in 0 blocks

Namely, you have no memory that is definitely lost, and much that "still reachable". I quote the canonical SO answer on "still reachable":

In general, there is no need to worry about "still reachable" blocks. They don't pose the sort of problem that true memory leaks can cause. For instance, there is normally no potential for heap exhaustion from "still reachable" blocks.

Not to grind this ax again, but this is one of the reasons I prefer -fsanitize=address to valgrind. Their definition of memory leak is more in line with expectations.

[khuck]

This memory leak is in libdl.so. See https://bugzilla.redhat.com/show_bug.cgi?id=1624387
and https://github.com/lattera/glibc/blob/895ef79e04a953cac1493863bcae29ad85657ee1/dlfcn/dlerror.c#L141

I don't think there's anything I can do about it - I call dlsym(), and it returns NULL if the symbol isn't found. I don't have access to a pointer that I can free...

[williamfgc]

@khuck @NAThompson @pnorbert thanks for your input, I asked to @AaronV77 to share so we can take your view on this. @AaronV77 I will run your example with -fsanitize.

[AaronV77]

Thanks everyone for the feedback, I just wanted to let everyone know. I will carry about my way! Thanks.