Ompi: memory hooks in UCX cause a freeze in PyFR

[robertsawko]

Hi,

This is just a placeholder for an issue I discussed with @jladd-mlnx at SC18 BoF. We're both almost certain it is a UCX bug but I want to raise it here first. Josh said it may be a known issue too. I have discussed it with app developers and they pointed to me to memory hooks. Please advise on the best way to contact UCX community.

I am running OpenMPI 3.1.2 and I compile mpi4py against it. Then I run pyfr and it freezes on my IB network. See gdb.txt for the backtrace. It cleary wants to deallocate some memory which was probably allocated before importing mpi (Python).

The cluster is ppc64le with NVLINK and P100 GPUs.

The problem does not occur if I switch off ucx memory hooks with:

mpirun \
    --mca btl_openib_warn_default_gid_prefix 0 \
    --mca pml_ucx_opal_mem_hooks 1 \
    --mca pml_ucx_verbose 9 \
    -report-bindings \
    pyfr run -b cuda mesh.pyfrm ../config.ini

When running ofed_info the UCX version I get is:

ucx:
mlnx_ofed_ucx/ucx-1.2.2947-1.src.rpm

As I understood, the community is going to adopt UCX and I want to keep up with it, so I consider this relatively important (even though I've got a workaround). Please advise.

[yosefe]

@robertsawko ucx 1.2 is pretty old, we recently release ucx 1.4 with many bugfixes, including in memory hooks area: https://github.com/openucx/ucx/releases/tag/v1.4.0
Can you give it a try?

[robertsawko]

Certainly, I am happy to try. But am I supposed to compile it from source alongside UCX which comes from OFED? Will there be no collision?

Actually, is UCX part of OFED? If so, is UCX in ppc64le OFED so old then? I thought we had one of the latest OFED installed. If any system change is necessary our change management will take a while...

[yosefe]

@robertsawko there are 2 options:

1. Upgrade to latest MLNX_OFED version - 4.4-2.0.7.0 - which contains UCX v1.4
2. Compile UCX 1.4 from source, install it to any prefix, and configure OpenMPI --with-ucx pointing to that install prefix.

[jsquyres]

@robertsawko Did this answer your question?

[robertsawko]

Sorry, @jsquyres . There was a week long outage at STFC and I couldn't verify yet the UCX update. The computers are back since yesterday so I am planning to build my own UCX today and see.

[robertsawko]

Is there any simple way to check if correct UCX is picked up at runtime? I was trying to see with ldd if ucx I built is dynamically linked but mpirun doesn't seem to have it.

I can't say for sure if it's working as the interactive nodes are inactive at the weekend... but using batch queues I built UCX, built two versions of OMPI 3.1.2+gpu and 4.0.0, built mpi4py for it (couldn't get it to build with 4.0.0), rebuild the rest of the app and run it. It is not producing output. I will be able to check gdb output when I get my interactive session on Monday...

[hoopoepg]

@robertsawko you may force loading your version of UCX using LD_PRELOAD variable:

mpirun -x LD_PRELOAD=path-to-ucx-libs/libucp.so

[yosefe]

also can check cat /proc/<pid>/maps | grep ucp
where is the MPI process running on compute node (not mpirun itself)

[robertsawko]

Thanks a lot everyone. I don't need to wait until interactive nodes are back. It's clearly producing output with ucx-1.4 although I will check the command that @yosefe posted. As always, interacting with you I got a problem solved and learnt a few new things along the way. Closing.