Ompi: OMPI + UCX segfault during MPI_Finalize

[bosilca]

Background information

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

Master with a full set of UCX flavors: 1.7, 1.8 1.9 and master. All UCX versions were configured with --with-cuda --with-avx --enable-mt

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

git pull
configure --prefix=... --enable-picky --enable-debug --disable-heterogeneous --enable-visibility \
          --enable-mpirun-prefix-by-default --with-pmix=internal --with-ucx=/opt/ucx/XXX/fast

Please describe the system on which you are running

Scientific Linux release 7.4 (Nitrogen)
Cluster of Mellanox 56gbps

hca_id: mthca0
    transport:          InfiniBand (0)
    fw_ver:             4.8.200

Details of the problem

Any app run on 2 nodes, including the most trivial benchmark, results in a failure during MPI_Finalize. Everything seems to run fine on a single node,

 0 0x0000000000053b83 ucs_debug_print_backtrace()  src/ucs/debug/debug.c:653
 1 0x000000000001cb8b uct_rc_verbs_handle_failure()  src/uct/ib/rc/verbs/rc_verbs_iface.c:63
 2 0x000000000001d195 uct_rc_verbs_iface_poll_tx()  src/uct/ib/rc/verbs/rc_verbs_iface.c:104
 3 0x000000000001d195 uct_rc_verbs_iface_progress()  src/uct/ib/rc/verbs/rc_verbs_iface.c:132
 4 0x000000000002a79a ucs_callbackq_dispatch()  src/ucs/datastruct/callbackq.h:211
 5 0x000000000002a79a uct_worker_progress()  src/uct/api/uct.h:2221
 6 0x000000000002a79a ucp_worker_progress()  src/ucp/core/ucp_worker.c:1951
 7 0x0000000000002d0c opal_common_ucx_wait_request()  ompi/build/debug/opal/mca/common/ucx/../../../../../../opal/mca/common/ucx/common_ucx.h:182
 8 0x00000000000032f8 opal_common_ucx_wait_all_requests()  ompi/build/debug/opal/mca/common/ucx/../../../../../../opal/mca/common/ucx/common_ucx.c:210
 9 0x0000000000003574 opal_common_ucx_del_procs_nofence()  ompi/build/debug/opal/mca/common/ucx/../../../../../../opal/mca/common/ucx/common_ucx.c:254
10 0x00000000000035fb opal_common_ucx_del_procs()  ompi/build/debug/opal/mca/common/ucx/../../../../../../opal/mca/common/ucx/common_ucx.c:272
11 0x0000000000006877 mca_pml_ucx_del_procs()  ompi/build/debug/ompi/mca/pml/ucx/../../../../../../ompi/mca/pml/ucx/pml_ucx.c:481
12 0x0000000000071510 ompi_mpi_finalize()  ompi/build/debug/ompi/../../../ompi/runtime/ompi_mpi_finalize.c:326
13 0x00000000000b7cb1 PMPI_Finalize()  ompi/build/debug/ompi/mpi/c/profile/pfinalize.c:54
14 0x0000000000400b35 main()  osu/osu_latency.c:305
15 0x00000000000223d5 __libc_start_main()  ???:0
16 0x0000000000400db2 _start()  ???:0

[yosefe]

"mthca0" devices are not supported by UCX
pls see more details in https://github.com/openucx/ucx/issues/5318

[bosilca]

Hold on, what exactly “wont fix” means ? I am not asking you to make UCX support old generation hardware, but you can’t leave OMPI segfaulting due to a choice made in your component. You have a choice either to implement a way to prevent UCX from selecting itself when it cannot run and/or fix the segfault after the run but we need a way to have OMPI run everywhere.

[yosefe]

@bosilca can you pls provide the full error log and full output of ibstat/ibv_devinfo?

[bosilca]

$ ibstat
CA 'mthca0'
    CA type: MT25208 (MT23108 compat mode)
    Number of ports: 2
    Firmware version: 4.8.200
    Hardware version: a0
    Node GUID: 0x00066a0098006c03
    System image GUID: 0x00066a0098006c03
    Port 1:
        State: Down
        Physical state: Polling
        Rate: 10
        Base lid: 0
        LMC: 0
        SM lid: 0
        Capability mask: 0x02590a68
        Port GUID: 0x00066a00a0006c03
        Link layer: InfiniBand
    Port 2:
        State: Active
        Physical state: LinkUp
        Rate: 10
        Base lid: 13
        LMC: 0
        SM lid: 1
        Capability mask: 0x02590a6a
        Port GUID: 0x00066a01a0006c03
        Link layer: InfiniBand

[bosilca]

For the error log it's a little more tricky, there is a lot of info from each process. I made a selection below:

local QP operation err (QPN 47040e, WQE @ 00007b03, CQN 5300b0, index 380)
  [ 0] 0047040e
  [ 4] 00000000
  [ 8] 00000000
  [ c] 00000000
  [10] 02700000
  [14] 00000000
  [18] 00007b03
  [1c] ff100000
[c05:8918 :0:8918] rc_verbs_iface.c:63   send completion with error: local QP operation error qpn 0x470411 wrid 0x29 vendor_err 0x70

      ...
       60     }
       61
       62     ucs_log(log_lvl,
==>    63             "send completion with error: %s qpn 0x%x wrid 0x%lx vendor_err 0x%x",
       64             ibv_wc_status_str(wc->status), wc->qp_num, wc->wr_id, wc->vendor_err);
       65 }
       66

[yosefe]

@bosilca https://github.com/openucx/ucx/pull/5488 should fix the issue, can you pls try it?

[bosilca]

I confirm the UCX patch fixes the issue in OMPI. Thanks @yosefe .

[bosilca]

Isn't there anything we can do in OMPI to gracefully support prior versions of UCX on older hardware?

[yosefe]

Isn't there anything we can do in OMPI to gracefully support prior versions of UCX on older hardware?

the patch adds support for older hardware

[jsquyres]

We should probably add something to the README about this (e.g., if you use older Mellanox hardware with UCX less than version X.Y.Z, Open MPI will crash). Is there a way to workaround the crash in Open MPI, or is it solely a UCX issue?

[hppritcha]

It was decided that the right place to fix this issue is in UCX itself. Use a non-zero RDMA write for flushing at the endpoint rather than a zero-byte RDMA write - see https://github.com/openucx/ucx/pull/5488. The UCX community expressed a desire to fix the problem in UCX and not patch Open MPI to work around the issue.

[abouteiller]

Issue is currently fixed with UCX 1.9.0-rc3, even on old MTHCA hardware.

[jsquyres]

Should we close this issue, then? Or should there be a configure workaround for this (e.g., avoid the UCX versions where this problem exists)...?

[jsquyres]

@abouteiller @hppritcha The README blurb that was added to Open MPI says that it will be fixed in UCX v1.10, but @abouteiller said it will be fixed in UCX v1.9.

Which is correct? Does the README need to be updated?

We need this README blurb in v4.1.x, too, BTW.

[yosefe]

the fix is part of ucx v1.9.0 release
merged in https://github.com/openucx/ucx/pull/5529

[gpaulsen]

Readme update has been merged to Open-MPI v4.0.x, v4.1.x, and master