Ompi: RMA Functionality

Created on 12 Jun 2020  路  71Comments  路  Source: open-mpi/ompi

I'm using MPI 4.0.1 installed from a system module. Our cluster is running RHEL6 with an infiniband network. The cores are Intel Xeon processors.

I鈥檓 curious about the status of RMA functionality in OpenMPI.

We鈥檝e had a MPI-RMA based runtime in Global Arrays for some time now but it does not work well with OpenMPI. It does, however, operate reasonably well with other MPI implementations. I鈥檝e run the GA test suite using the MPI-RMA runtime against different implementations of MPI-3 and the OpenMPI implementation performs significantly worse than implementations from MPICH, Intel and MVAPICH. The current test suite contains 80 programs. Using Intel MPI/5.1.3.181 we get 3 failures, with MPICH 3.3.2 we get 8 failures (MPICH 3.4a2 reduces this to 3 failures) and MVAPICH 2.3.2 reports 9 failures. In comparison, OpenMPI 4.0.1 reports 47 failures. We do have a few tests that are problematic across all implementations and we are looking at them from the GA side.

Our standard test configuration is to run on 4 processors split between two nodes. This forces at least some of the communication on to the switch. I can provide additional details about configuring and running the GA test suite as well as what tests are failing, if you are interested.

Given that we get much better results with other implementations, we believe the large number of failures we are seeing with OpenMPI are reflective of problems in the OpenMPI implementation of RMA functions in MPI-2 and 3. Are there any plans to improve these implementations? We have alternative implementations based on traditional 2-sided MPI that are robust and reasonably high performing, but we are still interested in using runtimes that sit directly on top of MPI RMA constructs. Are there plans going forward to aggressively improve the RMA implementations?

master v4.0.x v4.1.x
Source
picture bjpalmer

All 71 comments

Can you share your test suite? The RMA implementation used by Open MPI depends on your set up. There are two implementations that support infiniband and one that provides basic functionality. If the latter one (osc/pt2pt) is on use failures wouldn't be surprising as it is not as well tested. osc/rdma is well tested and provides high performance. osc/ucx still has some bugs but provides goods performance.

hjelmn picture hjelmn on 12 Jun 2020

The test suite is part of GA so if you download GA from our github repository, you will pick it up. The github repository is at

git clone https://github.com/GlobalArrays/ga.git
git checkout develop

I've been configuring with the MPI-RMA runtime using the line

./configure --enable-i4 --enable-cxx --with-mpi3\
--prefix=/my/install/directory CC=mpicc CXX=mpicxx FC=mpif90 \
CFLAGS="-g" CXXFLAG="-g" FFLAGS="-g"

I usually don't build in the top level directory, so you'll need to modify the location of configure accordingly if you are building somewhere else. You don't actually need to do 'make install' to run the test suite, just make.

After running make, run the following command in the build directory to run the test suite

make check-ga MPIEXEC="mpirun -n 4 "

Our standard configuration is 4 processors on 2 nodes.

When I run the test suite with the tests mir_perf2.x, perf2.x and thread_perf_contig.x hang. To get the full test suite to run, I've been editing the mir_perf2.F, perf2.c and thread_perf_contig.c files and adding either a "stop" (Fortran) or a "return 0;" (C) at the start of the main routine. On our platform, the remaining tests run and either pass or crash.

Let me know if you need more information.

bjpalmer picture bjpalmer on 12 Jun 2020

For got to mention, the source code for the test codes is $GA/global/testing.

bjpalmer picture bjpalmer on 12 Jun 2020

Is there a way to separate out building the tests from running them? The system I'm using doesn't support building on the compute nodes.

hppritcha picture hppritcha on 15 Jun 2020

@janjust - FYI - let's test this example on the latest greatest UCX/OSC goodies targeting OMPI v5.0 release.

jladd-mlnx picture jladd-mlnx on 15 Jun 2020
👍1

Might be a Fortran binding issue. I don't know how well those are tested.

hjelmn picture hjelmn on 15 Jun 2020
👀1

If you type

make checkprogs

in the build directory, it should build all the executables in the test directory.

bjpalmer picture bjpalmer on 15 Jun 2020

Some of the fortran tests pass and quite a few of the C tests fail. The underlying GA library is all written in C and the Fortran interface is a thin compatibility layer.

bjpalmer picture bjpalmer on 15 Jun 2020

Here's what I'm seeing on an aarch64 platform using gcc 9.3.0, ompi 4.0.4, and ucx 1.7:

PASS: ma/test-coalesce.x
PASS: ma/test-inquire.x
XFAIL: ma/testf.x
FAIL: global/testing/elempatch.x
PASS: global/testing/getmem.x
PASS: global/testing/mtest.x
PASS: global/testing/mulmatpatchc.x
PASS: global/testing/normc.x
PASS: global/testing/matrixc.x
PASS: global/testing/ntestc.x
PASS: global/testing/ntestfc.x
PASS: global/testing/packc.x
PASS: global/testing/patch_enumc.x
PASS: global/testing/print.x
PASS: global/testing/scan_addc.x
PASS: global/testing/scan_copyc.x
PASS: global/testing/testc.x
PASS: global/testing/testmatmultc.x
PASS: global/testing/testmult.x
PASS: global/testing/testmultrect.x
PASS: global/testing/gemmtest.x
PASS: global/testing/thread_perf_contig.x
PASS: global/testing/thread_perf_strided.x
PASS: global/testing/threadsafec.x
PASS: global/testing/read_only.x
PASS: global/testing/unpackc.x
PASS: global/testing/bin.x
PASS: global/testing/blktest.x
PASS: global/testing/g2test.x
PASS: global/testing/g3test.x
PASS: global/testing/ga_lu.x
PASS: global/testing/ga_shift.x
FAIL: global/testing/ghosts.x
PASS: global/testing/jacobi.x
PASS: global/testing/mir_perf2.x
PASS: global/testing/mmatrix.x
PASS: global/testing/mulmatpatch.x
PASS: global/testing/nbtest.x
PASS: global/testing/nb2test.x
PASS: global/testing/ndim.x
PASS: global/testing/patch.x
PASS: global/testing/patch2.x
PASS: global/testing/patch_enumf.x
PASS: global/testing/perfmod.x
PASS: global/testing/perform.x
PASS: global/testing/perf.x
PASS: global/testing/perf2.x
FAIL: global/testing/pg2test.x
FAIL: global/testing/pgtest.x
FAIL: global/testing/scan.x
PASS: global/testing/simple_groups.x
PASS: global/testing/sparse.x
PASS: global/testing/sprsmatmult.x
PASS: global/testing/stride.x
PASS: global/testing/testeig.x
PASS: global/testing/testmatmult.x
PASS: global/testing/testsolve.x
FAIL: global/testing/test.x
PASS: global/testing/simple_groups_comm.x
PASS: global/testing/ga-mpi.x
PASS: global/testing/lock.x
PASS: global/testing/simple_groups_commc.x
PASS: global/testing/nga-onesided.x
PASS: global/testing/nga-patch.x
PASS: global/testing/nga-periodic.x
PASS: global/testing/nga-scatter.x
PASS: global/testing/nga-util.x
PASS: global/testing/ngatest.x
PASS: global/examples/lennard-jones/lennard.x
PASS: global/examples/boltzmann/boltz.x
PASS: ga++/testing/elempatch.x
PASS: ga++/testing/mtest.x
PASS: ga++/testing/ntestc.x
PASS: ga++/testing/testc.x
PASS: ga++/testing/testmult.x
FAIL: ga++/testing/threadsafecpp.x
=================================
7 of 77 tests failed
See ./test-suite.log
Please report to [email protected]
=================================

I used the configure options given in an above comment.
I had
export OMPI_MCA_osc=ucx
export OMPI_MCA_pml=ucx
set in my shell.

hppritcha picture hppritcha on 15 Jun 2020

@hppritcha Can you run with osc/rdma as a sanity check?

hjelmn picture hjelmn on 15 Jun 2020

Out of curiosity, I ran the tests on my local shared memory machine using current master and osc rdma. I see 42 tests failing, most (all?) of which seem to end up in a similar Segfault:

Thread 1 "mtest.x" received signal SIGSEGV, Segmentation fault.
0x00007fffe76e0c3a in opal_thread_add_fetch_32 (addr=0x1, delta=-1) at ../../../../../opal/mca/threads/thread_usage.h:156
156     OPAL_THREAD_DEFINE_ATOMIC_OP(int32_t, add, +, 32)
(gdb) bt
#0  0x00007fffe76e0c3a in opal_thread_add_fetch_32 (addr=0x1, delta=-1) at ../../../../../opal/mca/threads/thread_usage.h:156
#1  0x00007fffe76e1729 in wait_sync_update (sync=0x1, updates=1, status=0) at ../../../../../opal/mca/threads/wait_sync.h:43
#2  0x00007fffe76e1888 in ompi_request_complete (request=0x555555aacd00, with_signal=true) at ../../../../../ompi/request/request.h:454
#3  0x00007fffe76e23b5 in ompi_osc_rdma_request_complete (request=0x555555aacd00, mpi_error=0) at ../../../../../ompi/mca/osc/rdma/osc_rdma_request.h:111
#4  0x00007fffe76e3e1d in ompi_osc_rdma_put_complete (btl=0x7fffebdf8220 <mca_btl_sm>, endpoint=0x555555a25e90, local_address=0x555555a59800, local_handle=0x0, context=0x555555aacd01, data=0x0, status=0)
    at ../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c:395
#5  0x00007fffebbf2f16 in mca_btl_sm_put_cma (btl=0x7fffebdf8220 <mca_btl_sm>, endpoint=0x555555a25e90, local_address=0x555555a59800, remote_address=93824997456812, local_handle=0x0, remote_handle=0x0, size=4, flags=0, order=255, 
    cbfunc=0x7fffe76e3d9a <ompi_osc_rdma_put_complete>, cbcontext=0x555555aacd01, cbdata=0x0) at ../../../../../opal/mca/btl/sm/btl_sm_put.c:87
#6  0x00007fffe76e4095 in ompi_osc_rdma_put_real (sync=0x555555a4e9b0, peer=0x555555a316f0, target_address=93824997456812, target_handle=0x0, ptr=0x555555a59800, local_handle=0x0, size=4, cb=0x7fffe76e3d9a <ompi_osc_rdma_put_complete>, 
    context=0x555555aacd01, cbdata=0x0) at ../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c:451
#7  0x00007fffe76e4342 in ompi_osc_rdma_put_contig (sync=0x555555a4e9b0, peer=0x555555a316f0, target_address=93824997456812, target_handle=0x0, source_buffer=0x555555a59800, size=4, request=0x555555aacd00)
    at ../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c:522
#8  0x00007fffe76e3651 in ompi_osc_rdma_master_noncontig (sync=0x555555a4e9b0, local_address=0x555555a59804, local_count=1, local_datatype=0x555555995e20, peer=0x555555a316f0, remote_address=93824997447616, remote_handle=0x0, remote_count=1, 
    remote_datatype=0x555555a85f10, request=0x555555aacd00, max_rdma_len=18446744073709551615, rdma_fn=0x7fffe76e414e <ompi_osc_rdma_put_contig>, alloc_reqs=false) at ../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c:264
#9  0x00007fffe76e3cdc in ompi_osc_rdma_master (sync=0x555555a4e9b0, local_address=0x555555a59804, local_count=1, local_datatype=0x555555995e20, peer=0x555555a316f0, remote_address=93824997447616, remote_handle=0x0, remote_count=1, 
    remote_datatype=0x555555a85f10, request=0x555555aacd00, max_rdma_len=18446744073709551615, rdma_fn=0x7fffe76e414e <ompi_osc_rdma_put_contig>, alloc_reqs=false) at ../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c:358
#10 0x00007fffe76e4ee9 in ompi_osc_rdma_put_w_req (sync=0x555555a4e9b0, origin_addr=0x555555a59804, origin_count=1, origin_datatype=0x555555995e20, peer=0x555555a316f0, target_disp=0, target_count=1, target_datatype=0x555555a85f10, 
    request=0x555555aacd00) at ../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c:763
#11 0x00007fffe76e53b9 in ompi_osc_rdma_rput (origin_addr=0x555555a59804, origin_count=1, origin_datatype=0x555555995e20, target_rank=0, target_disp=0, target_count=1, target_datatype=0x555555a85f10, win=0x555555a4e240, 
    request=0x7fffffffcf30) at ../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c:851
#12 0x00007ffff6794eb4 in PMPI_Rput (origin_addr=0x555555a59804, origin_count=1, origin_datatype=0x555555995e20, target_rank=0, target_disp=0, target_count=1, target_datatype=0x555555a85f10, win=0x555555a4e240, request=0x7fffffffcf30)
    at prput.c:87
#13 0x000055555566b51c in comex_putv (iov=0x555555a70500, iov_len=1, proc=0, group=0) at ../../comex/src-mpi3/comex.c:1455
#14 0x00005555556652f2 in PARMCI_PutV (darr=0x7fffffffd170, len=1, proc=0) at ../../comex/src-armci/armci.c:701
#15 0x0000555555668900 in ARMCI_PutV (darr=0x7fffffffd170, len=1, proc=0) at ../../comex/src-armci/capi.c:431
#16 0x0000555555605e28 in gai_gatscat_new (op=-99, g_a=-1000, v=0x555555a584b0, subscript=0x555555a53680, c_flag=1, nv=2500, locbytes=0x55555589dc50 <GAbytes+80>, totbytes=0x55555589dc58 <GAbytes+88>, alpha=0x0)
    at ../global/src/onesided.c:3559
#17 0x0000555555606328 in pnga_scatter (g_a=-1000, v=0x555555a584b0, subscript=0x555555a53680, c_flag=1, nv=2500) at ../global/src/onesided.c:3633
#18 0x0000555555566ad1 in NGA_Scatter (g_a=-1000, v=0x555555a584b0, subsArray=0x555555a53680, n=2500) at ../global/src/capi.c:3005
#19 0x0000555555557976 in main (argc=1, argv=0x7fffffffd6b8) at ../global/testing/mtest.c:91

Attaching a debugger, it seems that ompi_request_complete is called twice on the same request object, which leads to the Segfault in the second call:

1 invocation:

Thread 1 "mtest.x" hit Breakpoint 2, ompi_request_complete (request=0x555555aacd00, with_signal=true) at ../../../../../ompi/request/request.h:437
437     {
(gdb) bt
#0  ompi_request_complete (request=0x555555aacd00, with_signal=true) at ../../../../../ompi/request/request.h:437
#1  0x00007fffe76e23b5 in ompi_osc_rdma_request_complete (request=0x555555aacd00, mpi_error=0) at ../../../../../ompi/mca/osc/rdma/osc_rdma_request.h:111
#2  0x00007fffe76e3e1d in ompi_osc_rdma_put_complete (btl=0x7fffebdf8220 <mca_btl_sm>, endpoint=0x555555a25e90, local_address=0x555555a59804, local_handle=0x0, 
    context=0x555555aacd01, data=0x0, status=0) at ../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c:395
#3  0x00007fffebbf2f16 in mca_btl_sm_put_cma (btl=0x7fffebdf8220 <mca_btl_sm>, endpoint=0x555555a25e90, local_address=0x555555a59804, remote_address=93824997457612, 
    local_handle=0x0, remote_handle=0x0, size=4, flags=0, order=255, cbfunc=0x7fffe76e3d9a <ompi_osc_rdma_put_complete>, cbcontext=0x555555aacd01, cbdata=0x0)
    at ../../../../../opal/mca/btl/sm/btl_sm_put.c:87
#4  0x00007fffe76e4095 in ompi_osc_rdma_put_real (sync=0x555555a4e9b0, peer=0x555555a316f0, target_address=93824997457612, target_handle=0x0, ptr=0x555555a59804, local_handle=0x0, 
    size=4, cb=0x7fffe76e3d9a <ompi_osc_rdma_put_complete>, context=0x555555aacd01, cbdata=0x0) at ../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c:451
#5  0x00007fffe76e4342 in ompi_osc_rdma_put_contig (sync=0x555555a4e9b0, peer=0x555555a316f0, target_address=93824997457612, target_handle=0x0, source_buffer=0x555555a59804, size=4, 
    request=0x555555aacd00) at ../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c:522
#6  0x00007fffe76e3651 in ompi_osc_rdma_master_noncontig (sync=0x555555a4e9b0, local_address=0x555555a59804, local_count=1, local_datatype=0x555555995e20, peer=0x555555a316f0, 
    remote_address=93824997447616, remote_handle=0x0, remote_count=1, remote_datatype=0x555555a85f10, request=0x555555aacd00, max_rdma_len=18446744073709551615, 
    rdma_fn=0x7fffe76e414e <ompi_osc_rdma_put_contig>, alloc_reqs=false) at ../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c:264
#7  0x00007fffe76e3cdc in ompi_osc_rdma_master (sync=0x555555a4e9b0, local_address=0x555555a59804, local_count=1, local_datatype=0x555555995e20, peer=0x555555a316f0, 
    remote_address=93824997447616, remote_handle=0x0, remote_count=1, remote_datatype=0x555555a85f10, request=0x555555aacd00, max_rdma_len=18446744073709551615, 
    rdma_fn=0x7fffe76e414e <ompi_osc_rdma_put_contig>, alloc_reqs=false) at ../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c:358
#8  0x00007fffe76e4ee9 in ompi_osc_rdma_put_w_req (sync=0x555555a4e9b0, origin_addr=0x555555a59804, origin_count=1, origin_datatype=0x555555995e20, peer=0x555555a316f0, 
    target_disp=0, target_count=1, target_datatype=0x555555a85f10, request=0x555555aacd00) at ../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c:763
#9  0x00007fffe76e53b9 in ompi_osc_rdma_rput (origin_addr=0x555555a59804, origin_count=1, origin_datatype=0x555555995e20, target_rank=0, target_disp=0, target_count=1, 
    target_datatype=0x555555a85f10, win=0x555555a4e240, request=0x7fffffffcf30) at ../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c:851
#10 0x00007ffff6794eb4 in PMPI_Rput (origin_addr=0x555555a59804, origin_count=1, origin_datatype=0x555555995e20, target_rank=0, target_disp=0, target_count=1, 
    target_datatype=0x555555a85f10, win=0x555555a4e240, request=0x7fffffffcf30) at prput.c:87
#11 0x000055555566b51c in comex_putv (iov=0x555555a70500, iov_len=1, proc=0, group=0) at ../../comex/src-mpi3/comex.c:1455
#12 0x00005555556652f2 in PARMCI_PutV (darr=0x7fffffffd170, len=1, proc=0) at ../../comex/src-armci/armci.c:701
#13 0x0000555555668900 in ARMCI_PutV (darr=0x7fffffffd170, len=1, proc=0) at ../../comex/src-armci/capi.c:431
#14 0x0000555555605e28 in gai_gatscat_new (op=-99, g_a=-1000, v=0x555555a584b0, subscript=0x555555a53680, c_flag=1, nv=2500, locbytes=0x55555589dc50 <GAbytes+80>, 
    totbytes=0x55555589dc58 <GAbytes+88>, alpha=0x0) at ../global/src/onesided.c:3559
#15 0x0000555555606328 in pnga_scatter (g_a=-1000, v=0x555555a584b0, subscript=0x555555a53680, c_flag=1, nv=2500) at ../global/src/onesided.c:3633
#16 0x0000555555566ad1 in NGA_Scatter (g_a=-1000, v=0x555555a584b0, subsArray=0x555555a53680, n=2500) at ../global/src/capi.c:3005
#17 0x0000555555557976 in main (argc=1, argv=0x7fffffffd6b8) at ../global/testing/mtest.c:91

Second invocation:

Thread 1 "mtest.x" hit Breakpoint 2, ompi_request_complete (request=0x555555aacd00, with_signal=true) at ../../../../../ompi/request/request.h:437
437     {
(gdb) bt
#0  ompi_request_complete (request=0x555555aacd00, with_signal=true) at ../../../../../ompi/request/request.h:437
#1  0x00007fffe76e23b5 in ompi_osc_rdma_request_complete (request=0x555555aacd00, mpi_error=0) at ../../../../../ompi/mca/osc/rdma/osc_rdma_request.h:111
#2  0x00007fffe76e3e1d in ompi_osc_rdma_put_complete (btl=0x7fffebdf8220 <mca_btl_sm>, endpoint=0x555555a25e90, local_address=0x555555a59800, local_handle=0x0, 
    context=0x555555aacd01, data=0x0, status=0) at ../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c:395
#3  0x00007fffebbf2f16 in mca_btl_sm_put_cma (btl=0x7fffebdf8220 <mca_btl_sm>, endpoint=0x555555a25e90, local_address=0x555555a59800, remote_address=93824997456812, 
    local_handle=0x0, remote_handle=0x0, size=4, flags=0, order=255, cbfunc=0x7fffe76e3d9a <ompi_osc_rdma_put_complete>, cbcontext=0x555555aacd01, cbdata=0x0)
    at ../../../../../opal/mca/btl/sm/btl_sm_put.c:87
#4  0x00007fffe76e4095 in ompi_osc_rdma_put_real (sync=0x555555a4e9b0, peer=0x555555a316f0, target_address=93824997456812, target_handle=0x0, ptr=0x555555a59800, local_handle=0x0, 
    size=4, cb=0x7fffe76e3d9a <ompi_osc_rdma_put_complete>, context=0x555555aacd01, cbdata=0x0) at ../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c:451
#5  0x00007fffe76e4342 in ompi_osc_rdma_put_contig (sync=0x555555a4e9b0, peer=0x555555a316f0, target_address=93824997456812, target_handle=0x0, source_buffer=0x555555a59800, size=4, 
    request=0x555555aacd00) at ../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c:522
#6  0x00007fffe76e3651 in ompi_osc_rdma_master_noncontig (sync=0x555555a4e9b0, local_address=0x555555a59804, local_count=1, local_datatype=0x555555995e20, peer=0x555555a316f0, 
    remote_address=93824997447616, remote_handle=0x0, remote_count=1, remote_datatype=0x555555a85f10, request=0x555555aacd00, max_rdma_len=18446744073709551615, 
    rdma_fn=0x7fffe76e414e <ompi_osc_rdma_put_contig>, alloc_reqs=false) at ../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c:264
#7  0x00007fffe76e3cdc in ompi_osc_rdma_master (sync=0x555555a4e9b0, local_address=0x555555a59804, local_count=1, local_datatype=0x555555995e20, peer=0x555555a316f0, 
    remote_address=93824997447616, remote_handle=0x0, remote_count=1, remote_datatype=0x555555a85f10, request=0x555555aacd00, max_rdma_len=18446744073709551615, 
    rdma_fn=0x7fffe76e414e <ompi_osc_rdma_put_contig>, alloc_reqs=false) at ../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c:358
#8  0x00007fffe76e4ee9 in ompi_osc_rdma_put_w_req (sync=0x555555a4e9b0, origin_addr=0x555555a59804, origin_count=1, origin_datatype=0x555555995e20, peer=0x555555a316f0, 
    target_disp=0, target_count=1, target_datatype=0x555555a85f10, request=0x555555aacd00) at ../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c:763
#9  0x00007fffe76e53b9 in ompi_osc_rdma_rput (origin_addr=0x555555a59804, origin_count=1, origin_datatype=0x555555995e20, target_rank=0, target_disp=0, target_count=1, 
    target_datatype=0x555555a85f10, win=0x555555a4e240, request=0x7fffffffcf30) at ../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c:851
#10 0x00007ffff6794eb4 in PMPI_Rput (origin_addr=0x555555a59804, origin_count=1, origin_datatype=0x555555995e20, target_rank=0, target_disp=0, target_count=1, 
    target_datatype=0x555555a85f10, win=0x555555a4e240, request=0x7fffffffcf30) at prput.c:87
#11 0x000055555566b51c in comex_putv (iov=0x555555a70500, iov_len=1, proc=0, group=0) at ../../comex/src-mpi3/comex.c:1455
#12 0x00005555556652f2 in PARMCI_PutV (darr=0x7fffffffd170, len=1, proc=0) at ../../comex/src-armci/armci.c:701
#13 0x0000555555668900 in ARMCI_PutV (darr=0x7fffffffd170, len=1, proc=0) at ../../comex/src-armci/capi.c:431
#14 0x0000555555605e28 in gai_gatscat_new (op=-99, g_a=-1000, v=0x555555a584b0, subscript=0x555555a53680, c_flag=1, nv=2500, locbytes=0x55555589dc50 <GAbytes+80>, 
    totbytes=0x55555589dc58 <GAbytes+88>, alpha=0x0) at ../global/src/onesided.c:3559
#15 0x0000555555606328 in pnga_scatter (g_a=-1000, v=0x555555a584b0, subscript=0x555555a53680, c_flag=1, nv=2500) at ../global/src/onesided.c:3633
#16 0x0000555555566ad1 in NGA_Scatter (g_a=-1000, v=0x555555a584b0, subsArray=0x555555a53680, n=2500) at ../global/src/capi.c:3005
#17 0x0000555555557976 in main (argc=1, argv=0x7fffffffd6b8) at ../global/testing/mtest.c:91
(gdb) print *request
$1 = {super = {super = {super = {obj_magic_id = 16046253926196952813, obj_class = 0x7fffe7911b80 <ompi_osc_rdma_request_t_class>, obj_reference_count = 1, 
        cls_init_file_name = 0x7fffe7705fb8 "../../../../../ompi/mca/osc/rdma/osc_rdma_comm.c", cls_init_lineno = 847}, opal_list_next = 0x0, opal_list_prev = 0x0, item_free = 1, 
      opal_list_item_refcount = 0, opal_list_item_belong_to = 0x0}, registration = 0x0, ptr = 0x0}, req_type = OMPI_REQUEST_WIN, req_status = {MPI_SOURCE = 0, MPI_TAG = 0, 
    MPI_ERROR = 0, _cancelled = 0, _ucount = 0}, req_complete = 0x1, req_state = OMPI_REQUEST_ACTIVE, req_persistent = false, req_f_to_c_index = -32766, req_start = 0x0, 
  req_free = 0x7fffe76f2f2e <request_free>, req_cancel = 0x7fffe76f2f1c <request_cancel>, req_complete_cb = 0x0, req_complete_cb_data = 0x0, req_mpi_object = {comm = 0x555555a4e240, 
    file = 0x555555a4e240, win = 0x555555a4e240}}

Note the in the second case req_complete is 0x1, which eventually is treated as sync object and derefernced.

I haven't tried to understand why this happens, might dig into that later today.

devreal picture devreal on 16 Jun 2020

We have vanilla builds of UCX 1.4.0 and 1.8.0 on our machine. I can try running with that to see if I can reproduce @hppritcha's results. What exactly do you need to do to run with UCX? Put the libraries in your LD_LIBRARY_PATH and set the environment variables?

bjpalmer picture bjpalmer on 16 Jun 2020

@bjpalmer you need to add --with-ucx=path_to_ucx_1.8.0_install to the OMPI configure line and rebuild. Then just to be certain set then env. variables in your shell before running the make check

export OMPI_MCA_osc=ucx
export OMPI_MCA_pml=ucx
hppritcha picture hppritcha on 16 Jun 2020

@bjpalmer Can these tests potentially hang? Or is there a mechanism to kill jobs that are taking too long - mine seem to be stuck on ga_shift running ompi-master with UCX-v1.8 (both ompi and ucx in debug), but there were other tests that took a while but eventually passed.

janjust picture janjust on 16 Jun 2020

Some of them can definitely hang. On our system running 4.0.1, the mir_perf2.x, perf2.x and thread_perf_contig.x tests were hanging. I haven't seen any hangs in ga_shift.x. Some of the tests take a significant amount of time before finishing. I don't know of a way to kill tests that hang while the test suite is running. I usually just go edit the source file (they are in GA_DIR/global/testing) and put in a stop or return 0 to get the test to exit before starting. You might be able to get a running test to fail by logging on to one of the nodes that it is executing on and killing one of the processes manually.

bjpalmer picture bjpalmer on 16 Jun 2020
👍1

@bjpalmer, I made a mistake, it's not _ga_shift.x_ that hangs, it's _ghosts.x_ which seems to be a multi-threaded code. The same test failed for @hppritcha.

janjust picture janjust on 16 Jun 2020

I ran using openmpi 4.0.4 with built with UCX and with the OMPI_MCA_osc and OMPI_MCA_pml environment variables set to ucx. I had two hung processes and a total of 27 failures. I'm not doing as well as @hppritcha and while this is still a lot of failures, it is a considerable improvement over the previous run using Open MPI 4.0.1. I'm compiling with gcc/7.3.0 and using UCX 1.8.0.

bjpalmer picture bjpalmer on 16 Jun 2020

With the patch in #7829 all tests run successfully on my local shared memory system using osc/rdma. Can someone test osc/rdma on multiple nodes? (I don't have access to a non-IB machine anymore)

devreal picture devreal on 16 Jun 2020

I ran the test suite using osc/ucx on my local system and see a number of tests failing. It seems that UCX 1.8.0 is complaining about misaligned variables in all failed tests, e.g.:

> Checking disjoint put ... 
[1592329233.441781] [beryl:9227 :0]       amo_send.c:128  UCX  ERROR atomic variable must be naturally aligned (remote address 0x55d5ab64370c, size 8)
[1592329233.441839] [beryl:9227 :0]       amo_send.c:128  UCX  ERROR atomic variable must be naturally aligned (remote address 0x55d5ab643744, size 8)
[1592329233.441868] [beryl:9227 :0]       amo_send.c:128  UCX  ERROR atomic variable must be naturally aligned (remote address 0x55d5ab64377c, size 8)
[1592329233.441895] [beryl:9227 :0]       amo_send.c:128  UCX  ERROR atomic variable must be naturally aligned (remote address 0x55d5ab6437b4, size 8)
[1592329233.441923] [beryl:9227 :0]       amo_send.c:128  UCX  ERROR atomic variable must be naturally aligned (remote address 0x55d5ab6437ec, size 8)
[1592329233.441964] [beryl:9227 :0]       amo_send.c:128  UCX  ERROR atomic variable must be naturally aligned (remote address 0x559bb8e6db04, size 8)

AFAICS, the MPI standard does not mandate any alignment for user-provided input buffers and only mentions alignment in the context of performance (aligned memory may lead to better performance). This looks like a bug in osc/ucx to me.

devreal picture devreal on 16 Jun 2020

This gets more and more interesting: the UCX error is not caused by the application using misaligned target offsets but by the osc/ucx issuing a 64bit atomic fetch-add inside an Rget to acquire a request: 

(gdb) bt
#0  ucs_log_dispatch (file=file@entry=0x7fffeb19da70 "../../../src/ucp/rma/amo_send.c", line=line@entry=128, 
    function=function@entry=0x7fffeb19de20 <__func__.16080> "ucp_atomic_fetch_nb", level=level@entry=UCS_LOG_LEVEL_ERROR, 
    format=format@entry=0x7fffeb19db18 "atomic variable must be naturally aligned (remote address 0x%lx, size %zu)")
    at ../../../src/ucs/debug/log.c:181
#1  0x00007fffeb15800f in ucp_atomic_fetch_nb (ep=<optimized out>, opcode=<optimized out>, value=<optimized out>, 
    result=0x55555659a258, op_size=8, remote_addr=93825006787100, rkey=0x555555d2d380, 
    cb=0x7fffeacd9715 <opal_common_ucx_req_completion>) at ../../../src/ucp/rma/amo_send.c:126
#2  0x00007fffe7d2a48f in opal_common_ucx_atomic_fetch_nb (ep=0x7fffe3d38ba8, opcode=UCP_ATOMIC_FETCH_OP_FADD, value=0, 
    result=0x55555659a258, op_size=8, remote_addr=93825006787100, rkey=0x555555d2d380, 
    req_handler=0x7fffeacd9715 <opal_common_ucx_req_completion>, worker=0x555555cc1350)
    at ../../../../../opal/mca/common/ucx/common_ucx.h:199
#3  0x00007fffe7d2b419 in opal_common_ucx_wpmem_fetch_nb (mem=0x5555565b0600, opcode=UCP_ATOMIC_FETCH_OP_FADD, value=0, 
    target=1, buffer=0x55555659a258, len=8, rem_addr=93825006787100, user_req_cb=0x7fffe7d30e54 <req_completion>, 
    user_req_ptr=0x5555559b5400) at ../../../../../opal/mca/common/ucx/common_ucx_wpool.h:519
#4  0x00007fffe7d2dfd1 in ompi_osc_ucx_rget (origin_addr=0x55555642a090, origin_count=4, 
    origin_dt=0x555555825ba0 <ompi_mpi_char>, target=1, target_disp=199996, target_count=4, 
    target_dt=0x555555825ba0 <ompi_mpi_char>, win=0x5555565f5470, request=0x7fffffffce30)
    at ../../../../../ompi/mca/osc/ucx/osc_ucx_comm.c:868
#5  0x00007ffff6793f1b in PMPI_Rget (origin_addr=0x55555642a090, origin_count=4, 
    origin_datatype=0x555555825ba0 <ompi_mpi_char>, target_rank=1, target_disp=199996, target_count=4, 
    target_datatype=0x555555825ba0 <ompi_mpi_char>, win=0x5555565f5470, request=0x7fffffffce30) at prget.c:84
#6  0x00005555555fce2c in comex_nbget (src=src@entry=0x555556335a1c, dst=0x55555642a090, dst@entry=0x205d349e1eb45f00, 
    bytes=4, proc=proc@entry=1, group=group@entry=0, hdl=hdl@entry=0x7fffffffcf04) at ../../comex/src-mpi3/comex.c:2212
#7  0x00005555555f6a22 in PARMCI_NbGetS (src_ptr=0x555556335a1c, src_stride_arr=src_stride_arr@entry=0x7fffffffcf44, 
    dst_ptr=0x205d349e1eb45f00, dst_ptr@entry=0x55555642a090, dst_stride_arr=dst_stride_arr@entry=0x7fffffffcf64, 
    count=count@entry=0x7fffffffcf84, stride_levels=stride_levels@entry=0, proc=1, nb_handle=0x7fffffffcf04)
    at ../../comex/src-armci/armci.c:579
#8  0x00005555555f96e5 in ARMCI_NbGetS (src_ptr=<optimized out>, src_stride_arr=src_stride_arr@entry=0x7fffffffcf44, 
    dst_ptr=dst_ptr@entry=0x55555642a090, dst_stride_arr=dst_stride_arr@entry=0x7fffffffcf64, 
    count=count@entry=0x7fffffffcf84, stride_levels=stride_levels@entry=0, proc=1, nb_handle=0x7fffffffcf04)
    at ../../comex/src-armci/capi.c:314
#9  0x00005555555b8646 in ngai_nbgets (nbhandle=0x7fffffffcf04, type_size=4, field_size=-1, field_off=0, 
    proc=<optimized out>, nstrides=0, count=0x7fffffffcf84, stride_loc=0x7fffffffcf64, pbuf=0x55555642a090 "", 
    stride_rem=0x7fffffffcf44, prem=<optimized out>, loc_base_ptr=0x55555642a090 "") at ../global/src/onesided.c:443
---Type <return> to continue, or q <return> to quit---
#10 ngai_gets (type_size=4, field_size=-1, field_off=0, proc=<optimized out>, nstrides=0, count=0x7fffffffcf84, 
    stride_loc=0x7fffffffcf64, pbuf=0x55555642a090 "", stride_rem=0x7fffffffcf44, prem=<optimized out>, 
    loc_base_ptr=0x55555642a090 "") at ../global/src/onesided.c:496
#11 ngai_get_common (g_a=g_a@entry=-1000, lo=lo@entry=0x7fffffffd46c, hi=hi@entry=0x7fffffffd46c, 
    buf=buf@entry=0x55555642a090, ld=ld@entry=0x0, field_off=field_off@entry=0, field_size=-1, nbhandle=0x0)
    at ../global/src/onesided.c:1022
#12 0x00005555555b8dd8 in pnga_get (g_a=g_a@entry=-1000, lo=lo@entry=0x7fffffffd46c, hi=hi@entry=0x7fffffffd46c, 
    buf=buf@entry=0x55555642a090, ld=ld@entry=0x0) at ../global/src/onesided.c:1166
#13 0x00005555555d9de9 in pnga_scan_add (g_src=-1000, g_dst=-999, g_msk=-998, lo=<optimized out>, hi=<optimized out>, 
    excl=<optimized out>) at ../global/src/types2.xh:1
#14 0x000055555555786f in test_scan_add_C_INT_C_INT (q=2, excl=<optimized out>, lhi=<optimized out>, llo=<optimized out>)
    at ../global/testing/scan_addc.c:156
#15 main (argc=<optimized out>, argv=<optimized out>) at ../global/testing/scan_addc.c:246

The problematic code line is this: https://github.com/open-mpi/ompi/blob/master/ompi/mca/osc/ucx/osc_ucx_comm.c#L868

The target offset is the offset the user provided but that may not be 64-bit aligned, e.g., for get with MPI_INT.

@janjust @artpol84 Is there no other way to get a request object from UCX than through an atomic operation? Maybe that atomic operation should target an address we know has correct alignment?

devreal picture devreal on 16 Jun 2020

@bjpalmer Out of curiosity, is your UCX compiled in multi-threaded mode?
./ucx_info -v | grep enable-mt you should see the --enable-mt configure flag set. I think that's the reason behind the hangs and subsequently failures.

I ran the tests with ompi-v4.0.4rc3 ucx-v1.7.x and ucx-v1.8.x and I see:
7/80 and 3/80 failures, respectively.

The common failures were:

953:FAIL: global/testing/scan.x
961:FAIL: global/testing/test.x
962:FAIL: global/testing/overlay.x

ucx-v1.7.x failed in: 

980:FAIL: ga++/testing/threadsafecpp.x
981:FAIL: global/testing/thread_perf_contig.x
982:FAIL: global/testing/thread_perf_strided.x
983:FAIL: global/testing/threadsafec.x

but ucx-v1.8.x passed.

Out of curiosity, what were the 3 failures in intel, and mpich?

janjust picture janjust on 17 Jun 2020

I tried the benchmark with the modified osc/ucx in PR #6980 and I see the same three tests failing after applying the following patch to fix the rput/rget issue:

@@ -1074,7 +1075,7 @@ int ompi_osc_ucx_rput(const void *origin_addr, int origin_count,
     mca_osc_ucx_component.num_incomplete_req_ops++;
     ret = opal_common_ucx_wpmem_fetch_nb(module->mem, UCP_ATOMIC_FETCH_OP_FADD,
                                          0, target, &(module->req_result),
-                                         sizeof(uint64_t), remote_addr,
+                                         sizeof(uint64_t), remote_addr & ~0x7UL,
                                          req_completion, ucx_req);
     if (ret != OMPI_SUCCESS) {
         OMPI_OSC_UCX_REQUEST_RETURN(ucx_req);
@@ -1126,7 +1127,7 @@ int ompi_osc_ucx_rget(void *origin_addr, int origin_count,
     mca_osc_ucx_component.num_incomplete_req_ops++;
     ret = opal_common_ucx_wpmem_fetch_nb(module->mem, UCP_ATOMIC_FETCH_OP_FADD,
                                          0, target, &(module->req_result),
-                                         sizeof(uint64_t), remote_addr,
+                                         sizeof(uint64_t), remote_addr & ~0x7UL,
                                          req_completion, ucx_req);

I dug into the test.x case it a bit and found that osc/ucx does not correctly handle indexed datatypes if the datatype has overlapping entries. In the benchmark, some values may be the target of multiple updates in the same accumulate operation. That does not work well with the get-update-put mechanism used in the osc/ucx as every value in the local copy is updated once and the target values will be overwritten instead of updated (i.e., incremented) twice.

devreal picture devreal on 17 Jun 2020

@devreal do you mean that master with #6980 fails on 3 tests?

artpol84 picture artpol84 on 17 Jun 2020

@devreal do you mean that master with #6980 fails on 3 tests?

I would be shocked, I'm getting consistent 27/80 failures with master.

janjust picture janjust on 17 Jun 2020

Sorry, that wasn't entirely clear: both master and #6980 fail for the same reason that I described above. The problematic part didn't change in #6980.

Also: this was on my local shared memory system, there may be more failures on multiple nodes.

devreal picture devreal on 17 Jun 2020

Can anyone comment on the scale of these tests? I run these across 2 nodes, ppn2. @devreal how many ppns do you run on a single node?

janjust picture janjust on 17 Jun 2020

I ran them with both a single and four processes. The three failed test cases caused by the indexed data type occur even when running on a single process.

devreal picture devreal on 17 Jun 2020

A bunch of questions to catch up on:
1) I looked at the output of ./ucx_info -v for our UCX 1.8.0 build. The only option listed was --prefix.

[d3g293@constance01 bin]$ ucx_info -v

UCT version=1.8.0 revision c30b7da

configured with: --prefix=/share/apps/ucx/1.8.0

2) The results on Intel MPI/5.1.3.181
cache_test.x hangs
sprsmatmult.x hangs
ga++/threadsafecpp.x fails

Results for MPICH 3.4a2
ga++/threadsafecpp.x fails
threadsafec.x runs but errors present
nbtest.x runs but errors present

Results for MVAPICH 2.3.2
nbtestc.x hangs
testmult.x hangs
sprsmatmult.x fails
ga++/testmult.x fails
ga++/threadsafecpp.x fails

The remaining tests run but report errors
nbtest.x
thread_perf_contig.x
thread_perf_strided.x
threadsafec.x

It looks like nbtest.x has a bug in it, so I would ignore that test. The thread tests are also a bit sketchy, but they appear to pass on a regular basis when run with the progress ranks runtime.

3) the test suite is designed to run on 4 processors total and should be run on 2 nodes with 2 processors per node. Some tests will fail if run on anything other than 4 processors. For these tests there should be an error message in the output saying that they must be run on 4 processors.

bjpalmer picture bjpalmer on 17 Jun 2020

@bjpalmer it appears your UCX is not configured with multithreaded support.

jladd-mlnx picture jladd-mlnx on 17 Jun 2020

Should it be? I can try building it on my own and rebuild 4.0.4.

bjpalmer picture bjpalmer on 17 Jun 2020

@bjpalmer yes, the hangs you are seeing are due to UCX not being configured with multi-threaded support. With UCX v1.8.x with multi-threaded support, I'm seeing 3/80 failures.

janjust picture janjust on 17 Jun 2020

@bjpalmer You don't need to rebuild OMPI, just UCX. Please add --enable-mt to your UCX configure command.

jladd-mlnx picture jladd-mlnx on 17 Jun 2020

alternatively, UCX configure option: ./contrib/configure-release-mt will include it.

janjust picture janjust on 17 Jun 2020
👍1

I built UCX with --enable-mt and used that in the open MPI 4.0.4 build. I reran the test suite but still saw 26 failures.

bjpalmer picture bjpalmer on 17 Jun 2020

@bjpalmer - so something is amiss, I able to reproduce @hppritcha results with ucx v1.7, and with v1.8 I see only 3 failures. I reran to make sure it's not a coincidence.

My commits are: 56470b4aba18c886fe843daffd80e09ac3fb4bf0(ompi v4.0.x) and 49982d4dcf2a4a9ff4a39dc4c962d9612f673301(ucx v1.8.x).
my gcc version is 8.3.0, running on x86 and rhel 7.6
I'm not sure if any of this matters; however. Are you sure the correct UCX is used? If I try to rebuild my ucx without mt, I see the same ~27 failures out of 80.

janjust picture janjust on 17 Jun 2020

Also, @bjpalmer could you share the logs or check whether you are seeing crashes in them?

devreal picture devreal on 17 Jun 2020

I'm running 4.0.4 which I built from the download tarball. I can checkout the master and try that instead, if that is what your are using. Can you tell me exactly how you configure UCX? I just used

./configure --enable-mt --prefix=/people/d3g293/mpich/ucx-1.8.0

and then built MPI with

../configure --prefix=/qfs/people/d3g293/openmpi-4.0.4/build-ucx --with-ucx=/people/d3g293/ucx-1.8.0

Most of the tests that are failing are failing because an internal check on the answer fails so the code calls MPI_Abort. Almost everyone produces the following error on exiting, even if the test passed

[node008.local:14107] 3 more processes have sent help message help-mpi-btl-openib.txt / error in device init
[node008.local:14107] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

I've assumed in the past that this is just some deficiency in our ancient cluster and ignored it, but perhaps it suggests something to you.

I can try running a few of the tests in some other mode to see if I can extract more information if you have any suggestions.

bjpalmer picture bjpalmer on 18 Jun 2020

I'm not using master, I tested it but it I'm seeing about ~28 failed tests. I'm using the 4.0.x branch, which is 4.0.4rc3, basically I cloned both repos and checked out the latest 4.0.x and 1.8.x branches for ompi, and ucx respectively.

My config and build commands are:
for ucx:
./autogen.sh && ./contrib/configure-release-mt --prefix=$MTR_SCRAP/builds/ucx-v1.8.x-mt-rel --without-java && make -j install

for ompi
git clean -fdx && ./autogen.pl && ./configure --with-ucx=/hpc/mtr_scrap/users/tomislavj/builds/ucx-v1.8.x-mt-rel --without-verbs --prefix=$MTR_SCRAP/builds/ompi-v4.0.x --enable-mca-no-build=coll-ml,btl-uct --enable-orterun-prefix-by-default=yes && make -j install

janjust picture janjust on 18 Jun 2020

Just talked with @artpol84 offline about the osc/ucx issues. So far there are two issues to address:

1) The way rget/rput is implemented right now needs fixing. A quick fix would be for the fetch-and-op to work on aligned target memory (see https://github.com/open-mpi/ompi/issues/7813#issuecomment-645354174). However, maybe the non-blocking flushes introduced with UCX recently could be a better match here?

2) The handling of complex datatypes with atomics needs fixing. We agreed that keeping the get-reduce-put protocol to stay one-sided would be beneficial (as opposed to falling back to some active message protocol). However, this needs some investigation into how to use the datatype engine to treat overlapping entries properly.

devreal picture devreal on 19 Jun 2020

I tried the above configurations (I think) but all tests failed. I got rid of the --without-verbs option on the ompi build and got back to 27 failures. I'm still using the 4.0.4 tarball but I will try the 4.0.x branch to see if that makes a difference. At this point, given the difference between my results and yours, I'm inclined to believe that there are deficiencies in the environment of our cluster.

bjpalmer picture bjpalmer on 19 Jun 2020

@devreal I think you synced with @artpol84, not artemry-mlnx

@bjpalmer quite possible, fwiw I'm running on x86 system, with ConnectX-5 (I tested 6 too), RHEL 7.6, Mellanox OFED 5.0 - so all in all relatively newer setup.

janjust picture janjust on 19 Jun 2020
👍1

Having checked the MPI 3.1 standard, I believe that the test (test.x) is incorrect. Section 11.3.4 (Accumulate Functions) states on page 425 that:

The parameter target_datatype must not specify overlapping entries

The test case eventually passes an indexed datatype to MPI_Accumulate that contains overlapping entries. The same seems to be true for overlay.x. @bjpalmer can you confirm that?

I thus now believe that osc/ucx is correct in that regard.

The test case contains a commented out line (958) that seems to be meant to ensure that unique indices are used. If I uncomment that line (and comment out the #define NGA_GATSCAT line) the test succeeds on my local machine.

devreal picture devreal on 22 Jun 2020

I will check the test. I'm not sure if having unique indices is a requirement from the GA side, but if indices are repeated, that would probably be an error for most algorithms, especially accumulate.

bjpalmer picture bjpalmer on 22 Jun 2020

@devreal, to be clear are you saying that you believe osc/ucx is correct in this context WITH https://github.com/open-mpi/ompi/pull/7852 or without it?

gpaulsen picture gpaulsen on 22 Jun 2020

@gpaulsen #7852 addresses a bug in osc/rdma, it's independent of osc/ucx.

devreal picture devreal on 22 Jun 2020
👍1

@jladd-mlnx Is there a reason why --enable-mt is still not the default? I find that a little confusing.

Doesn't impact btl/uct as it does its own locking if UCX doesn't have MT enabled but that isn't the default path.

hjelmn picture hjelmn on 23 Jun 2020

It might be worthwhile for Open MPI to fail configure if UCX is being used and OpenUCX doesn't have multi-threading enabled.

hjelmn picture hjelmn on 23 Jun 2020

@hjelmn: https://github.com/open-mpi/ompi/pull/6209

jladd-mlnx picture jladd-mlnx on 23 Jun 2020

@jladd-mlnx Right. But this is about RMA. What happens with RMA? Does osc/ucx disqualify itself? if not it should.

hjelmn picture hjelmn on 23 Jun 2020

The calls to MPI_Accumulate uses either an MPI SubArray or MPI Struct as the datatype argument. I'm assuming that there is no chance that a SubArray can specify overlapping entries.

bjpalmer picture bjpalmer on 24 Jun 2020

OK, I will have to check what datatype is used in that case then (I was assuming indexed as that would be my choice for the random index pairs in test.F). I guess structs can contain overlapping entries.

devreal picture devreal on 24 Jun 2020

I pushed a fix for the overlapping entries problem to the develop branch.

bjpalmer picture bjpalmer on 24 Jun 2020

@bjpalmer thanks, both test.x and overlay.x now succeed locally (though I wonder whether that should be addressed inside GA's MPI-RMA adapter as users may still provide overlapping entries in applications...)

I am still seeing scan.x fail with osc/ucx (Scan_copy failed.) but it's not entirely clear what happens in this test and how to debug that as there seem to be multiple operations involved (GA_scan_copy, nga_get, nga_scatter_acc, ga_ddot) and it's hard to discern which of them produces a wrong result. Any idea where to start looking?

devreal picture devreal on 25 Jun 2020

It looks like the scan.x test is failing because an underlying MPI_Accumulate is trying to use overlapping entries. The accumulate operation is producing incorrect results, which then lead to an MPI_Abort. I will modify the test so that it does not do this (the accumulates are used as a check on the results of scan_copy, so they are not integral to the test).

I've also tried running Open MPI 4.0.4 on some newer hardware running RHEL7 instead of RHEL6 but I'm still seeing about 25 failures in the test suite. This new hardware is managed as part of the same center as the machine I was using previously and may suffer the same quirks. I will see if I can get access to a new machine they are standing up in our EMSL computing center.

bjpalmer picture bjpalmer on 25 Jun 2020

I pushed a change in scan.x that should get rid of the failure you are seeing.

bjpalmer picture bjpalmer on 25 Jun 2020

@bjpalmer I tried the fix and a run with a single process succeeds but the test hangs on 4 processes, with one process stuck in MPI_Allgather and three in MPI_Barrier:

Output:

$ mpirun -n 4 --map-by :OVERSUBSCRIBE --mca osc ucx ./global/testing/scan.x 
  GA initialized 
 using            4  process(es)
 testing pack/unpack
me:    0 lo:        1 hi:    20000
me:    1 lo:    20001 hi:    40000
me:    2 lo:    40001 hi:    60000
me:    3 lo:    60001 hi:    80000
me:    0 num:    20000 lo:        1 hi:    20000
me:    1 num:        0 lo:    20001 hi:    20000
me:    2 num:        0 lo:    40001 hi:    20000
me:    3 num:        0 lo:    60001 hi:    20000
 Pack successful
 Unpack successful
me:    0 lo:        1 hi:    20000
me:    1 lo:    20001 hi:    40000
me:    2 lo:    40001 hi:    60000
me:    3 lo:    60001 hi:    80000
 Too many elements 
me:    1 num:        0 lo:    20001 hi:    20000

The one in MPI_Allgather:

#12 0x00007ffff64dc02c in PMPI_Allgather (sendbuf=0x7fffffffd238, sendcount=8, 
    sendtype=0x555555895b60 <ompi_mpi_byte>, recvbuf=0x555555e750f0, 
    recvcount=8, recvtype=0x555555895b60 <ompi_mpi_byte>, comm=0x555555bd02c0)
    at pallgather.c:125
#13 0x00005555556746ba in comex_free (ptr=0x555555e75120, group=0)
    at ../../comex/src-mpi3/comex.c:3471
#14 0x00005555556687bc in PARMCI_Free (ptr=0x555555e75120)
    at ../../comex/src-armci/armci.c:308
#15 0x000055555565935e in pnga_destroy (g_a=-1000) at ../global/src/base.c:3977
#16 0x0000555555659481 in pnga_terminate () at ../global/src/base.c:4024
#17 0x000055555557d444 in ga_terminate_ () at ../global/src/fapi.c:1094
#18 0x000055555555a7a5 in MAIN__ () at ../global/testing/scan.F:54

The ones in MPI_Barrier:

#11 0x00007ffff64e26c0 in PMPI_Barrier (comm=0x555555bd02c0) at pbarrier.c:66
#12 0x0000555555670a73 in comex_barrier (group=0)
    at ../../comex/src-mpi3/comex.c:1920
#13 0x000055555566b4c3 in parmci_msg_barrier ()
    at ../../comex/src-armci/message.c:538
#14 0x0000555555570d40 in pnga_msg_sync () at ../global/src/collect.c:149
#15 0x00005555555f3560 in pnga_sync () at ../global/src/onesided.c:168
#16 0x000055555557e470 in ga_sync_ () at ../global/src/fapi.c:1528
#17 0x000055555555837c in test_nga_scan () at ../global/testing/scan.F:392
#18 0x000055555555a79b in MAIN__ () at ../global/testing/scan.F:48

It looks like the check for if(num.gt.nelements_max) only triggers on one process.

devreal picture devreal on 26 Jun 2020

I ran the test suite on 4 Haswell-based IB nodes (as far as I got within the 30 minute limit for interactive jobs on our system) and see tests failing that didn't fail when I ran them locally:

  • FAIL: global/testing/testmatmultc.x: this test fails with incorrect results from ga_dgemm:
Matrix Multiplication on C = A[512,512]xB[512,512]
           Run# 0:       0.1974 seconds        340.0 mflops/proc  n n
Values are = 38405696.000000 127905412.000000
Values are = 40244796.000000 89499716.000000
[0] Received an Error in Communication: (1) 0:verify ga_dgemm failed:
  • FAIL: global/testing/testmult.x:
[0] Received an Error in Communication: (1) 0:GA_Sgemm, NGA_Matmul_patch Failed:
Single Precision: Testing GA_Sgemm,NGA_Matmul_patch for 2-Dimension
abs(result) = 16555640832.000000 > 0.000001
[1] Received an Error in Communication: (1) 1:GA_Sgemm, NGA_Matmul_patch Failed:

abs(result) = 16555640832.000000 > 0.000001
[3] Received an Error in Communication: (1) 3:GA_Sgemm, NGA_Matmul_patch Failed:
[2] Received an Error in Communication: (1) 2:GA_Sgemm, NGA_Matmul_patch Failed:

abs(result) = 16555640832.000000 > 0.000001

abs(result) = 16555640832.000000 > 0.000001
  • FAIL: global/testing/unpackc.x: fails with unpack mismatch:
NELEM=200000
testing lo=0 hi=199999 q=1 C_INT C_INT
testing lo=0 hi=199999 q=1 C_LONG C_INT
[1] Received an Error in Communication: (-1) 1:unpack mismatch in C_LONG/C_INT:
[0] Received an Error in Communication: (-1) 0:unpack mismatch in C_LONG/C_INT:
[2] Received an Error in Communication: (-1) 2:unpack mismatch in C_LONG/C_INT:
[3] Received an Error in Communication: (-1) 3:unpack mismatch in C_LONG/C_INT:
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 1 in communicator MPI COMMUNICATOR 3 DUP FROM 0
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
  • FAIL: global/testing/ndim.x: fails filling array A:
           4  nodes
  Testing 3-dimensional Arrays
2: array section [1:19,20:37,1:37]
1: array section [20:37,1:19,1:37]
3: array section [20:37,20:37,1:37]
 3-dimensional Array A:    37    37    37
 distribution information for all processors
 -------------------------------------------
0: array section [1:19,1:19,1:37]

 Filling array A
ERROR (proc=2): a [0,32,34] =2.000000,  b [0,32,34] =-1.000000
[2] Received an Error in Communication: (-1) 2:bye:
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 2 in communicator MPI COMMUNICATOR 3 DUP FROM 0
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
  • FAIL: global/testing/pg2test.x and global/testing/pgtest.x: failure in disjoint put:
> Checking disjoint put ...
  put            1          65          73   9280.0000000000000        0.0000000000000000
[3] Received an Error in Communication: (-1) 3:... exiting:
  • FAIL: global/testing/blktest.x: fails in GA_symmetrize:
 Testing GA_symmetrize...

[0] Received an Error in Communication: (1) 0:exiting:

 GA_Symmetrize is not OK

Note that ndim.x seems to suffer from timing issues as the reason for failure is not consistent (sometimes fails after filling, sometimes at random get). Similarly, unpackc.x succeeds if I break at all the GA_unpack calls.

devreal picture devreal on 26 Jun 2020

@devreal How is osc/rdma looking for these tests after your fix?

hjelmn picture hjelmn on 26 Jun 2020

@hjelmn is there a chance to test osc/rdma without the openib btl? I tried --mca osc rdma --mca btl uct but the uct (or ucx) btl is not found.

devreal picture devreal on 26 Jun 2020

I have updated my summary above, the unaligned AMO errors and errors in MPI_Finalize are gone after merging https://github.com/open-mpi/ompi/pull/7871 into master. The tests that fail now are not trivial to debug, some seem to trigger race conditions as stepping through them lets them succeed. Plus, some of them are generated from macros, which makes debugging a challenge. @bjpalmer can you try the latest master with UCX 1.8.0 and try to narrow down which operations are actually failing?

devreal picture devreal on 26 Jun 2020

I will download master and see if what kind of luck I have.

bjpalmer picture bjpalmer on 26 Jun 2020

@devreal You need to enable it used --mca btl_uct_memory_domains=. You can find the ib memory domain with ucx_info.

hjelmn picture hjelmn on 26 Jun 2020

Is that the correct syntax? My configure doesn't seem to have heard of it. It tried adding

--mca btl_uct_memory_domains=ml4_0

but I just get a message saying --mca is an unknown option. There doesn't seem to be anything similar showing up when I type configure -help.

bjpalmer picture bjpalmer on 26 Jun 2020

@hjelmn I found a second instance of the fragmented rput issue and posted a PR for master: https://github.com/open-mpi/ompi/pull/7882

So far, I see a bunch of tests failing with osc/rdma and btl/uct on 4 nodes using the patch above:

FAIL: global/testing/patch_enumc.x
FAIL: global/testing/testmatmultc.x
FAIL: global/testing/testmult.x
FAIL: global/testing/read_only.x
FAIL: global/testing/cache_test.x
FAIL: global/testing/blktest.x
FAIL: global/testing/g2test.x
FAIL: global/testing/g3test.x
FAIL: global/testing/ghosts.x
FAIL: global/testing/mmatrix.x
FAIL: global/testing/ndim.x
FAIL: global/testing/pg2test.x
FAIL: global/testing/pgtest.x

I will try to investigate that next week.

devreal picture devreal on 26 Jun 2020

@bjpalmer Could you please try to use the UCX one-sided component? The btl_uct_memory_domains parameter is only needed when using the RDMA one-sided component with the UCX byte-transfer layer (yeah, my head keeps spinning, too).

You should only need to pass --mca osc ucx to use the UCX one-sided component.

devreal picture devreal on 26 Jun 2020

I must be missing something here, but I'm configuring with

../configure --prefix=/qfs/people/d3g293/ompi/build-ucx \
--with-ucx=/people/d3g293/ucx-1.8.0 --disable-man-pages --mca osc ucx\
--enable-mca-no-build=coll-ml,btl-uct --enable-orterun-prefix-by-default=yes

but I'm getting

[d3g293@constance03 build-ucx]$ ./build_csh
configure: error: unrecognized option: --mca' Try../configure --help' for more information

Am I adding this directive to the right spot? (The two enable directives come from a previous post, so let me know if I should get rid of them). If I leave out the --mca option, the configure goes through but I get the following when I try and compile I get the following error on 'make install'

make install-exec-hook
make[4]: Entering directory /qfs/people/d3g293/mpich/ompi/build-ucx/prrte' make[4]: Leaving directory/qfs/people/d3g293/mpich/ompi/build-ucx/prrte'
make[3]: Nothing to be done for install-data-am'. make[3]: Leaving directory/qfs/people/d3g293/mpich/ompi/build-ucx/prrte'
make[2]: Leaving directory /qfs/people/d3g293/mpich/ompi/build-ucx/prrte' make[1]: Leaving directory/qfs/people/d3g293/mpich/ompi/build-ucx/prrte'
make[1]: Entering directory /qfs/people/d3g293/mpich/ompi/build-ucx' make[2]: Entering directory/qfs/people/d3g293/mpich/ompi/build-ucx'
make install-exec-hook
make[3]: Entering directory /qfs/people/d3g293/mpich/ompi/build-ucx' WARNING! Common symbols found: rmaps_rank_file_lex.o: 0000000000000004 C prte_rmaps_rank_file_leng rmaps_rank_file_lex.o: 0000000000000008 C prte_rmaps_rank_file_text hostfile_lex.o: 0000000000000004 C prte_util_hostfile_leng hostfile_lex.o: 0000000000000008 C prte_util_hostfile_text show_help_lex.o: 0000000000000004 C prte_show_help_yyleng show_help_lex.o: 0000000000000008 C prte_show_help_yytext keyval_lex.o: 0000000000000004 C prte_util_keyval_yyleng keyval_lex.o: 0000000000000008 C prte_util_keyval_yytext make[3]: [install-exec-hook] Error 1 (ignored) make[3]: Leaving directory/qfs/people/d3g293/mpich/ompi/build-ucx'
make[2]: Nothing to be done for install-data-am'. make[2]: Leaving directory/qfs/people/d3g293/mpich/ompi/build-ucx'
make[1]: Leaving directory `/qfs/people/d3g293/mpich/ompi/build-ucx'

bjpalmer picture bjpalmer on 26 Jun 2020

@bjpalmer Sorry, I slightly misread your first comment. The --mca configuration parameters are a runtime configuration mechanism passed to mpirun, not configure. The "common symbol warnings" at the end of the build process are OK, your build seems to succeed.

devreal picture devreal on 27 Jun 2020

Just fyi, with the added updates in ga tests.
I now see only 1/80 tests failing using ompi v4.0.x and ucx-v1.8.x (I haven't checked master yet.)
scan.x failed, but it was either a hang (in collectives), or just took too long.
I had to kill it - I'll rerun again with added time.

janjust picture janjust on 29 Jun 2020
👍1

I discovered an array overflow in scan.F after I bumped up the size of the test. I've fixed it and checked it in so it may work better now. I'll also try compiling the fortran-based tests with the array bound checker on and see if anything else shows up. I will try running with --mca option and see what happens.

bjpalmer picture bjpalmer on 30 Jun 2020

I tested the latest tests with ompi master branch (410596e), and with ucx-v1.8.x-mt
5/80 failures, significantly lower than what I previously measured.

getmem.x
packc.x
blktest.x
overlay.x
ngatest.x

v4.0.x with ucx-v1.8.x-mt 80/80 passed, and 1 expected failure, I'm guessing that's what XFAIL means.

=====================================================
All 80 tests behaved as expected (1 expected failure)
=====================================================
janjust picture janjust on 1 Jul 2020

@janjust Can you test again with latest UCX (getting near 1.9.0?) and master and 4.1.x?

jsquyres picture jsquyres on 16 Oct 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ax3l picture ax3l  路  6Comments
yoshihingis picture yoshihingis  路  3Comments
abeltre1 picture abeltre1  路  4Comments
ashwinvis picture ashwinvis  路  5Comments
egorgam picture egorgam  路  5Comments