Ompi: How to compile OpenMPI to support FDR over SR-IOV
Thank you for taking the time to submit an issue!
Background information
I'm using Mellanox 56G FDR with SRIOV on KVM virtualization, and I want to use the RDMA to communicate between VM with FDR Virtual Function.
What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)
openmpi-3.0.0
Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)
./configure
make
make install
Please describe the system on which you are running
- Operating system/version: CentsOS 7.3
- Computer hardware: KVM Virtualization
- Network type: 56G FDR -- Virtual Function
Details of the problem
Please describe, in detail, the problem that you are having, including the behavior you expect to see, the actual behavior that you are seeing, steps to reproduce the problem, etc. It is most helpful if you can attach a small program that a developer can use to reproduce your problem.
Note: If you include verbatim output (or a code block), please use a GitHub Markdown code block like below:
```[root@vcn01 C]# mpirun --allow-run-as-root -np 2 ./IOR
WARNING: No preset parameters were found for the device that Open MPI
detected:
Local host: vcn01
Device name: mlx5_0
Device vendor ID: 0x02c9
Device vendor part ID: 4114
Default device parameters will be used, which may result in lower
performance. You can edit any of the files specified by the
btl_openib_device_param_files MCA parameter to set values for your
device.
NOTE: You can turn off this warning by setting the MCA parameter
btl_openib_warn_no_device_params_found to 0.
[vcn01][[60572,1],0][connect/btl_openib_connect_udcm.c:1235:udcm_rc_qp_to_rtr] error modifing QP to RTR errno says Invalid argument
[vcn01][[60572,1],1][connect/btl_openib_connect_udcm.c:1235:udcm_rc_qp_to_rtr] error modifing QP to RTR errno says Invalid argument
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
mpirun noticed that process rank 0 with PID 0 on node vcn01 exited on signal 11 (Segmentation fault).
[vcn01:14783] 1 more process has sent help message help-mpi-btl-openib.txt / no device params found
[vcn01:14783] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[root@vcn01 C]#
```
pharthiphan
All 7 comments
@pharthiphan, sorry for the late response.
On behalf of Mellanox I can recommends to build Open MPI with UCX support and use UCX PML
(-mca pml ucx).
We support this configuration and if you'll have any issues with it we will make sure that it will be fixed.
@jladd-mlnx @yosefe @jsquyres FYI
artpol84
on 22 Mar 2018
Hi artpol84
I have build OpenMPI with UCX support using the following link
https://github.com/openucx/ucx/wiki/OpenMPI-and-OpenSHMEM-installation-with-UCX
System command using ompi works
> [root@vsn01 lustre]# mpirun --allow-run-as-root -np 2 --host vsn01,vsn02 -mca pml ucx hostname
> vsn01
> vsn02
> [root@vsn01 lustre]#
mpi hello world program compiled using ompi, works but it throws warnings, what does this mean.
> [root@vsn01 lustre]# mpirun --allow-run-as-root -np 2 --host vsn01,vsn02 -mca pml ucx ./a.out
> --------------------------------------------------------------------------
> WARNING: No preset parameters were found for the device that Open MPI
> detected:
>
> Local host: vsn01
> Device name: mlx5_0
> Device vendor ID: 0x02c9
> Device vendor part ID: 4114
>
> Default device parameters will be used, which may result in lower
> performance. You can edit any of the files specified by the
> btl_openib_device_param_files MCA parameter to set values for your
> device.
>
> NOTE: You can turn off this warning by setting the MCA parameter
> btl_openib_warn_no_device_params_found to 0.
> --------------------------------------------------------------------------
> [1524462245.763684] [vsn01:29368:0] cpu.c:52 UCX WARN CPU does not support invariant TSC, time may be unstable
> [1524462245.831433] [vsn02:27232:0] cpu.c:52 UCX WARN CPU does not support invariant TSC, time may be unstable
> Hello world from processor vsn02, rank 1 out of 2 processors
> Hello world from processor vsn01, rank 0 out of 2 processors
> [vsn01:29358] 1 more process has sent help message help-mpi-btl-openib.txt / no device params found
> [vsn01:29358] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
> [root@vsn01 lustre]#
I tried to IOR which is compiled using ompi with UCX support. compilation was smooth but while running on multiple nodes crashes
> [root@vsn01 C]# mpirun --allow-run-as-root -np 2 --host vsn01,vsn02 -mca pml ucx ./IOR -a POSIX -w -r -t 1m -b 1m -o /lustre/test.out
> --------------------------------------------------------------------------
> WARNING: No preset parameters were found for the device that Open MPI
> detected:
>
> Local host: vsn01
> Device name: mlx5_0
> Device vendor ID: 0x02c9
> Device vendor part ID: 4114
>
> Default device parameters will be used, which may result in lower
> performance. You can edit any of the files specified by the
> btl_openib_device_param_files MCA parameter to set values for your
> device.
>
> NOTE: You can turn off this warning by setting the MCA parameter
> btl_openib_warn_no_device_params_found to 0.
> --------------------------------------------------------------------------
> [1524462454.619292] [vsn01:29584:0] cpu.c:52 UCX WARN CPU does not support invariant TSC, time may be unstable
> [1524462454.680336] [vsn02:27310:0] cpu.c:52 UCX WARN CPU does not support invariant TSC, time may be unstable
> mlx5: vsn02: got completion with error:
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 78006802 0a000041 000074d2
> [vsn02:27310:0:27310] ud_verbs.c:305 Fatal: Send completion (wr_id=0xFAAFFAAF with error: local QP operation error
> ==== backtrace ====
> 0 /lustre/sw/ucx/lib/libucs.so.0(ucs_fatal_error+0xec) [0x7f907b4bde0c]
> 1 /lustre/sw/ucx/lib/libuct.so.0(+0x7d1c6) [0x7f907b95b1c6]
> 2 /lustre/sw/ucx/lib/libucs.so.0(+0x160ba) [0x7f907b4b80ba]
> 3 /lustre/sw/ucx/lib/libucp.so.0(ucp_worker_progress+0x32) [0x7f907bbb36e2]
> 4 /lustre/sw/ompi/lib/openmpi/mca_pml_ucx.so(mca_pml_ucx_send+0x150) [0x7f907bdfa580]
> 5 /lustre/sw/ompi/lib/libmpi.so.0(PMPI_Send+0xf2) [0x7f908dc35c72]
> 6 ./IOR() [0x404d1b]
> 7 ./IOR() [0x407a2a]
> 8 ./IOR() [0x402f93]
> 9 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f908d5f8b35]
> 10 ./IOR() [0x402b99]
> ===================
> [vsn02:27310] *** Process received signal ***
> [vsn02:27310] Signal: Aborted (6)
> [vsn02:27310] Signal code: (-6)
> [vsn02:27310] [ 0] /lib64/libpthread.so.0(+0xf370)[0x7f908d9a7370]
> [vsn02:27310] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x7f908d60c1d7]
> [vsn02:27310] [ 2] /lib64/libc.so.6(abort+0x148)[0x7f908d60d8c8]
> [vsn02:27310] [ 3] /lustre/sw/ucx/lib/libucs.so.0(ucs_fatal_error+0xf1)[0x7f907b4bde11]
> [vsn02:27310] [ 4] /lustre/sw/ucx/lib/libuct.so.0(+0x7d1c6)[0x7f907b95b1c6]
> [vsn02:27310] [ 5] /lustre/sw/ucx/lib/libucs.so.0(+0x160ba)[0x7f907b4b80ba]
> [vsn02:27310] [ 6] /lustre/sw/ucx/lib/libucp.so.0(ucp_worker_progress+0x32)[0x7f907bbb36e2]
> [vsn02:27310] [ 7] /lustre/sw/ompi/lib/openmpi/mca_pml_ucx.so(mca_pml_ucx_send+0x150)[0x7f907bdfa580]
> [vsn02:27310] [ 8] /lustre/sw/ompi/lib/libmpi.so.0(PMPI_Send+0xf2)[0x7f908dc35c72]
> [vsn02:27310] [ 9] ./IOR[0x404d1b]
> [vsn02:27310] [10] ./IOR[0x407a2a]
> [vsn02:27310] [11] ./IOR[0x402f93]
> [vsn02:27310] [12] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f908d5f8b35]
> [vsn02:27310] [13] ./IOR[0x402b99]
> [vsn02:27310] *** End of error message ***
> -------------------------------------------------------
> Primary job terminated normally, but 1 process returned
> a non-zero exit code. Per user-direction, the job has been aborted.
> -------------------------------------------------------
> --------------------------------------------------------------------------
> mpirun noticed that process rank 1 with PID 27310 on node vsn02 exited on signal 6 (Aborted).
> --------------------------------------------------------------------------
> [vsn01:29574] 1 more process has sent help message help-mpi-btl-openib.txt / no device params found
> [vsn01:29574] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
> [root@vsn01 C]#
Note:- IOR run on a single machine goes through
> [root@vsn01 C]# mpirun --allow-run-as-root -np 1 --host vsn01 -mca pml ucx ./IOR -a POSIX -w -r -t 1m -b 1m -o /lustre/test.out
> --------------------------------------------------------------------------
> WARNING: No preset parameters were found for the device that Open MPI
> detected:
>
> Local host: vsn01
> Device name: mlx5_0
> Device vendor ID: 0x02c9
> Device vendor part ID: 4114
>
> Default device parameters will be used, which may result in lower
> performance. You can edit any of the files specified by the
> btl_openib_device_param_files MCA parameter to set values for your
> device.
>
> NOTE: You can turn off this warning by setting the MCA parameter
> btl_openib_warn_no_device_params_found to 0.
> --------------------------------------------------------------------------
> [1524462746.099137] [vsn01:29624:0] cpu.c:52 UCX WARN CPU does not support invariant TSC, time may be unstable
> IOR-2.10.3: MPI Coordinated Test of Parallel I/O
>
> Run began: Mon Apr 23 01:52:26 2018
> Command line used: ./IOR -a POSIX -w -r -t 1m -b 1m -o /lustre/test.out
> Machine: Linux vsn01
>
> Summary:
> api = POSIX
> test filename = /lustre/test.out
> access = single-shared-file
> ordering in a file = sequential offsets
> ordering inter file= no tasks offsets
> clients = 1 (1 per node)
> repetitions = 1
> xfersize = 1 MiB
> blocksize = 1 MiB
> aggregate filesize = 1 MiB
>
> Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)
> --------- --------- --------- ---------- ------- --------- --------- ---------- ------- --------
> write 369.49 369.49 369.49 0.00 369.49 369.49 369.49 0.00 0.00271 EXCEL
> read 1179.84 1179.84 1179.84 0.00 1179.84 1179.84 1179.84 0.00 0.00085 EXCEL
>
> Max Write: 369.49 MiB/sec (387.44 MB/sec)
> Max Read: 1179.84 MiB/sec (1237.16 MB/sec)
>
> Run finished: Mon Apr 23 01:52:26 2018
> [root@vsn01 C]#
Please help !
pharthiphan
on 23 Apr 2018
I was able to get rid of the warnings by adding Vendor ID 4114 to the file mca-btl-openib-device-params.ini
but still couldn't run IOR on multiple machines, Please help!
pharthiphan
on 23 Apr 2018
@pharthiphan when using UCX, can you please add "-x UCX_IB_ADDR_TYPE=ib_global" to mpirun command line?
The error is "malformed wqe" which happens because UCX uses LID-based addressing by default and the SRIOV you have is probably configured (in opensm) to be GID-based
yosefe
on 23 Apr 2018
@yosefe worked thankyou
> [root@vsn01 C]# mpirun --allow-run-as-root -np 4 --host vsn01,vsn01,vsn02,vsn02 -mca pml ucx -x UCX_IB_ADDR_TYPE=ib_global ./IOR -a POSIX -w -r -t 1m -b 1m -o /lustre/test.out
> [1524473977.321253] [vsn01:19448:0] cpu.c:52 UCX WARN CPU does not support invariant TSC, time may be unstable
> [1524473977.339176] [vsn01:19449:0] cpu.c:52 UCX WARN CPU does not support invariant TSC, time may be unstable
> [1524473977.343667] [vsn02:7582 :0] cpu.c:52 UCX WARN CPU does not support invariant TSC, time may be unstable
> [1524473977.362494] [vsn02:7583 :0] cpu.c:52 UCX WARN CPU does not support invariant TSC, time may be unstable
> IOR-2.10.3: MPI Coordinated Test of Parallel I/O
>
> Run began: Mon Apr 23 04:59:37 2018
> Command line used: ./IOR -a POSIX -w -r -t 1m -b 1m -o /lustre/test.out
> Machine: Linux vsn01
>
> Summary:
> api = POSIX
> test filename = /lustre/test.out
> access = single-shared-file
> ordering in a file = sequential offsets
> ordering inter file= no tasks offsets
> clients = 4 (2 per node)
> repetitions = 1
> xfersize = 1 MiB
> blocksize = 1 MiB
> aggregate filesize = 4 MiB
>
> Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)
> --------- --------- --------- ---------- ------- --------- --------- ---------- ------- --------
> write 72.88 72.88 72.88 0.00 72.88 72.88 72.88 0.00 0.05489 EXCEL
> read 105.79 105.79 105.79 0.00 105.79 105.79 105.79 0.00 0.03781 EXCEL
>
> Max Write: 72.88 MiB/sec (76.42 MB/sec)
> Max Read: 105.79 MiB/sec (110.93 MB/sec)
>
> Run finished: Mon Apr 23 04:59:37 2018
> [root@vsn01 C]#
>
@yosefe, thank you!
artpol84
on 24 Apr 2018
@artpol84 @yosefe Should you guys add a UCX section to the Open MPI FAQ on the web site? Stuff like this could be quite useful.
jsquyres
on 24 Apr 2018
Related issues
bodgerer
路
8Comments
ax3l
路
6Comments
sagitter
路
5Comments
yoshihingis
路
3Comments
ax3l
路
7Comments
Most helpful comment
@yosefe worked thankyou