Ompi: connectX5 100gbps yields low speed in openmpi-4.0.1

[oneZeroOneZeroOne]

Thank you for taking the time to submit an issue!

Background information

What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)

4.0.1

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

source tarball
compiler: tried both GNU-6 and intel-18
transport: tried both UCX and ibverbs

./configure --prefix=/opt/openmpi/openmpi-4.0.1-GNU6 --disable-dependency-tracking --enable-shared --enable-static --disable-heterogeneous --enable-mpi-thread-multiple --without-verbs --without-mxm --without-psm --without-psm2 --with-libfabric=/usr/ --with-io-romio-flags= --with-ucx=/opt/UCX/1.7 --enable-mca-no-build=btl-uct --with-hwloc=/usr/ --with-hwloc-libdir=/usr/lib/x86_64-linux-gnu/ --enable-mpi1-compatibility --without-slurm --with-tm=/opt/pbspro/18.1.0-git/ CC=/usr/bin/gcc-6 CXX=/usr/bin/g++-6 FC=/usr/bin/gfortran-6

Please describe the system on which you are running

• Operating system/version: Debian 9.9
• Computer hardware: Intel Xeon head-node & AMD-EPYC compute nodes
• Network type: Infiniband Mellanox ConnectX5 (head-node: MT26428; compute nodes: MT27800)

---

Details of the problem

I am not getting the expected bandwidth or latency when compiling performance tests (like OSU microbenchmarks) with openmpi, although the low-level tests (ib_read_bw) do show the full speed of the network (i.e. 100gbps).

1. low-level test (ib_read_bw) - looks good:

># ib_read_bw 192.168.3.1
---------------------------------------------------------------------------------------
                    RDMA_Read BW Test
 Dual-port       : OFF          Device         : mlx5_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x04 QPN 0x0087 PSN 0x9a0a45 OUT 0x10 RKey 0x0069dd VAddr 0x007f6213d78000
 remote address: LID 0x01 QPN 0x0090 PSN 0x494b03 OUT 0x10 RKey 0x007af0 VAddr 0x007fe4743b1000
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 65536      1000             11791.86            11791.58                  0.188665
---------------------------------------------------------------------------------------

1. OSU benchmark, compiled with openmpi-4.0.1 - yields roughly 32gbps:

>~/osu> mpirun -mca pml ucx -H 192.168.3.1,192.168.3.2 -np 2 osu_bw
# OSU MPI Bandwidth Test v5.6
# Size      Bandwidth (MB/s)
1                       3.05
2                       5.04
4                       7.14
8                      13.58
16                     28.70
32                     51.82
64                    102.67
128                   162.31
256                   399.78
512                   701.49
1024                 1450.57
2048                 2636.50
4096                 3221.63
8192                 3853.71
16384                3942.07
32768                4081.85
65536                4270.52
131072               3989.46
262144               4060.83
524288               4068.83
1048576              4020.62
2097152              4040.20
4194304              4057.03

I was expecting something closer to 100gbps for larger packet sizes, similar to this URL: MVAPICH . What am I doing wrong?

[yosefe]

@oneZeroOneZeroOne maybe the issue is with CPU affinity set by OpenMPI, can you pls try running
taskset -c 0 ib_read_bw ... (on both sides) and check if it still gets 100Gbs?

[oneZeroOneZeroOne]

indeed, the bandwidth goes down to about half.
without taskset:

~> ib_read_bw
---------------------------------------------------------------------------------------
                    RDMA_Read BW Test
 Dual-port       : OFF          Device         : mlx5_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x04 QPN 0x02d7 PSN 0xb03acf OUT 0x10 RKey 0x100702 VAddr 0x002b618a451000
 remote address: LID 0x01 QPN 0x0109 PSN 0x5c4d33 OUT 0x10 RKey 0x06052f VAddr 0x002b3202d24000
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 65536      1000             11793.36            11792.73                  0.188684
---------------------------------------------------------------------------------------

versus:

>:~> taskset -c 0 ib_read_bw
---------------------------------------------------------------------------------------
                    RDMA_Read BW Test
 Dual-port       : OFF          Device         : mlx5_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x04 QPN 0x02d8 PSN 0x72b4e5 OUT 0x10 RKey 0x100d07 VAddr 0x002af8fb4a3000
 remote address: LID 0x01 QPN 0x010a PSN 0xb9d3d4 OUT 0x10 RKey 0x061643 VAddr 0x002acb31cb9000
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 65536      1000             10393.20            6446.26                   0.103140
---------------------------------------------------------------------------------------

[yosefe]

possibly the BW between NUMA domains on that server is sub-optimal..
can you try adding -mca rmaps_dist_device mlx5_0 -mca rmaps_base_mapping_policy dist:span --report-bindings to mpirun, and report the output?

[oneZeroOneZeroOne]

Looks like I am getting the expected full speed (100 gbps) in this way:

taifun1:~/osu/gnu-ucx> mpirun -mca pml ucx -mca rmaps_dist_device mlx5_0 -mca rmaps_base_mapping_policy dist:span --report-bindings -H 192.168.3.1,192.168.3.2 -np 2 ./osu_bw
[taifun1:05353] MCW rank 0 bound to socket 0[core 12[hwt 0-1]]: [../../../../../../../../../../../../BB/../../..]
[taifun2:17315] MCW rank 1 bound to socket 0[core 12[hwt 0]]: [././././././././././././B/././.]
# OSU MPI Bandwidth Test v5.6
# Size      Bandwidth (MB/s)
1                       3.47
2                       7.38
4                      14.55
8                      29.49
16                     58.29
32                    114.78
64                    225.17
128                   398.20
256                   757.13
512                  1346.01
1024                 2468.93
2048                 3697.60
4096                 4958.72
8192                 5966.11
16384                6509.58
32768               11645.72
65536               11936.86
131072              12168.79
262144              12277.94
524288              12334.98
1048576             12363.70
2097152             12377.99
4194304             12385.08

[jsquyres]

Looks like the issue here was just setting the locality correctly (i.e., getting the MPI process in the same NUMA locality as the NIC).

Closing the issue, since it appears resolved.