Ompi: Java bindings vs mpool/memkind

[ggouaillardet]

With the latest master (I did not test the release branches yet), all my Java tests ends up in a crash at the end.

I made a bit of debugging, and found the issue involves the memkind library that is linked to the mpool/memkind component.

At first glance, it seems memkind invokes pthread_key_create() with a destructor, but this destructor ends up being invoked __after__ libmemkind.so has been unmapped, resulting in a crash
(so it seems pthread_key_delete() has not been invoked by memkind)

A simple workaround is to

mpirun --mca mpool ^memkind ...

FWIW, I use memkind provided by xppsl (e.g. the Intel addons for KNL) xppsl-memkind-1.5.1-4151.x86_64

At this stage, I am not sure whether this is a bug in memkind (nor if it has been fixed since) or in how mpool/memkind uses memkind.

[hppritcha]

I think Open MPI's component model is doing its usual thing as a framework is being closed, and
component's are unloaded.

Another workaround would be to use --disable-dlopen.

I think we should at a minimum have a blurb in the readme.

@gpaulsen

[ggouaillardet]

@hppritcha I am not sure --disable-dlopen will help here (note I did not test it).

The pthread key destructor is invoked by java pretty late (and likely __after__ libmpi.so has been unmapped, so I suspect the crash will always occur (unless the pthread key is destructed, but that does not happen yet)

I will give it a try from Monday

[jsquyres]

I have no doubt that what @hppritcha says is happening is actually happening. But this usually means that something is closed down in the wrong order, and/or one component has leaked something (i.e., not cleaned it up) that another component is then later cleaning up.

In most (if not all) cases, this is a bug on our part.

[ggouaillardet]

@jsquyres the pthread key is created by libmemkind under the hood, and is not currently deleted.

that could be a bug from the library, or from Open MPI since we might miss a step that would delete that key under the hood.

I will also have to test with the latest memkind and double check that in the source code.

[ggouaillardet]

Here are the results of my tests

• the issue does not occur if Open MPI is configure'd with --disable-dlopen
• the issue occurs even with the latest version of memkind and I did issue a (proof of concept level) PR that fixes this issue