Aws-parallelcluster: Abnormal in-node latency with EFA enabled

[jimmielin]

Environment:

• AWS ParallelCluster / CfnCluster version: pcluster 2.4.0
• OS: ubuntu1604
• Scheduler: slurm
• Master instance type: c5n.large
• Compute instance type: c5n.18xlarge

Bug description and how to reproduce:
When EFA is enabled in ~/.parallelcluster/config using enable_efa = compute and using built-in openMPI & libfabric, in-node latency as benchmarked using osu-micro-benchmarks is extremely high compared to without-EFA levels.

Steps to reproduce:

1. Create a HPC cluster with EFA enabled (enable_efa = compute).
2. Build osu-micro-benchmarks from http://mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-5.6.1.tar.gz and run osu_latency test on one single node:

srun -N 1 -n 2 --pty /bin/bash
mpirun --mca btl self,vader,tcp ./osu_latency

Results show > 10 us latency in one single node:

# OSU MPI Latency Test v5.6.1
# Size          Latency (us)
0                      14.78
1                      15.76
2                      15.70
4                      15.05
8                      15.02
16                     15.04
32                     15.01
64                     15.05
128                    15.02
256                    15.08
512                    15.14
1024                   15.83
2048                   16.09
4096                   17.64
8192                   20.23
16384                  54.93
32768                  60.09
65536                  70.12
131072                 85.27
262144                153.30
524288                294.98
1048576               597.51
2097152              1200.84
4194304              2371.78

Which is extremely high compared to when EFA is disabled, where latencies are ~1 us scale.

0                       0.31
1                       0.34
2                       0.33
4                       0.33
8                       0.33
16                      0.37
32                      0.37
64                      0.41
128                     0.43
256                     0.48
512                     0.63
1024                    0.73
2048                    0.90
4096                    1.62
8192                    1.86
16384                   2.54
32768                   3.33
65536                   5.05
131072                  8.57
262144                 15.75
524288                 33.36
1048576                83.37
2097152               209.42
4194304               407.54

Additional context:
This issue results in extremely degraded performance in MPI when enabling EFA. It seems like MPI shared memory has something wrong. We suspect that enabling EFA messed up OpenMPI's Vader BTL for shared memory data transfer.

[sean-smith]

This is expected as the libfabric provider for EFA does not currently detect a local process and defer to the shm provider to accelerate in-node traffic. It is on our roadmap this year to add this capability and we will follow up on this ticket when we have more information.

[JiaweiZhuang]

Seems like Intel-MPI does not have this problem. Tested with pcluster 2.4.1 on a single c5n.18xlarge node (ubuntu1604 + slurm). Here're the reproducible steps and results:

Both OpenMPI and Intel MPI are pre-installed:

$ module avail
------------------------------------- /usr/share/modules/versions --------------------------------------
3.2.10

------------------------------------ /usr/share/modules/modulefiles ------------------------------------
dot                 module-git          modules             openmpi/3.1.4-2
intelmpi/2019.4.243 module-info         null                use.own
$ module load intelmpi/2019.4.243
$ which mpicc
/opt/intel/compilers_and_libraries_2019.4.243/linux/mpi/intel64/bin/mpicc

Grab OSU:

wget http://mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-5.6.1.tar.gz
tar zxf osu-micro-benchmarks-5.6.1.tar.gz
cd osu-micro-benchmarks-5.6.1
./configure CC=mpicc CXX=mpicxx --prefix=$HOME/osu
make
make install

Measure point-to-point latency on a single node:

$ cd $HOME/osu/libexec/osu-micro-benchmarks/mpi/pt2pt
$ # srun -N 1 -n 2 ./osu_latency  # this doesn't work as the MPI is not built properly with Slurm
$ srun -N 1 -n 2 --pty /bin/bash  # interactive session works though
$ mpirun ./osu_latency
# OSU MPI Latency Test v5.6.1
# Size          Latency (us)
0                       0.45
1                       0.42
2                       0.42
4                       0.42
8                       0.42
16                      0.42
32                      0.44
64                      0.44
128                     0.55
256                     0.61
512                     0.81
1024                    0.90
2048                    1.06
4096                    1.50
8192                    1.94
16384                   3.31
32768                   5.08
65536                   9.46
131072                 18.43
262144                 35.54
524288                 84.90
1048576               171.69
2097152               341.89
4194304               683.29

which is normal in-node latency.

OpenMPI is still having the problem shown by @jimmielin (Just replace module load intelmpi with module load openmpi in the above steps)

[JiaweiZhuang]

The in-node latency problem has disastrous impact on the performance of MPI_Bcast and other collectives, which heavily rely on shared-memory message passing.

For broadcasting large messages, OpenMPI is 10x slower than Intel MPI (15000 us vs 1500 us) :

It is even slower than OpenMPI without EFA (falls back to TCP):

This could kill the performance of I/O-heavy models (for example WRF by default uses serial read + broadcasting). I can stick to Intel MPI for now, but fixing OpenMPI would be much more ideal, given its popularity... Is it possible to fix this by an ad-hoc patch to openmpi or libfabric? I don't mind applying some hacks by myself if that's doable.

[rajachan]

Jiawei - Thanks for sharing the results. Neither the Open MPI libfabric MTL (OFI MTL) nor the libfabric EFA provider optimize for intra-instance communication with shared-memory today. LIke Sean mentioned, adding shared-memory support to the EFA provider is something on our roadmap and is being actively worked on, but we do not have a date for when that will be available yet. It is more involved than a quick patch to either of these libraries, so I don't think we can suggest a workaround/hack at this time.

[JiaweiZhuang]

@rajachan Thanks, this is useful to know! I will use Intel MPI before the fix.

[sean-smith]

Thanks @JiaweiZhuang !

Can you clarify MPI is not built properly with Slurm ? Usually I run with:

module load intelmpi
export I_MPI_PMI_LIBRARY=/opt/slurm/lib/libpmi.so
srun -N 1 -n 2 /shared/intelmpi/osu-micro-benchmarks-5.4/mpi/pt2pt/osu_latency

I think all you're missing is export I_MPI_PMI_LIBRARY=/opt/slurm/lib/libpmi.so line. See https://slurm.schedmd.com/mpi_guide.html#intel_mpi

[JiaweiZhuang]

@sean-smith Yes I was missing I_MPI_PMI_LIBRARY -- adding so fixes the Slurm problem. Thanks!

[jimmielin]

Thanks @JiaweiZhuang for testing Intel MPI results. I've been trying to use intel-mpi with gfortran and I've ran into this issue with a handful of legacy code (e.g. in hdf5 fortran bindings) where use mpi is used instead of include "mpif.h" in modern fortran.

This results in a unexpected EOF when reading the mpi module, as Intel MPI's mpi.mod is compiled with Intel compilers (ifort) instead of gfortran, resulting in module file incompatibility. This is shown by

strings mpi.mod | grep 17
17.0.3

Thus it will be impossible to use use mpi in gfortran-compiled fortran code linked with Intel MPI. The following hack will fix this issue:

• Remove mpi.mod compiled with Intel ifort from the Intel-MPI directory
• ln -sf mpif90 mpifort to avoid existing amazon EFA-enabled openmpi from interfering with the intel fortran compilers (Intel MPI does not have mpifort by default, so some code that calls mpifort will use Open-MPI compiler while mpif90 is the right intel-MPI compiler)
• Create a file hack.f90 with the following:

module mpi
    include "mpif.h"
end module mpi

• Compile using mpif90 -c hack.f90 and rm hack.o, keeping mpi.mod that is binary compatible with gfortran

Additionally if installing any code via spack (e.g. hdf5/fortran) remember to source Intel MPI variables:

source $(spack location -i intel-mpi%[email protected])/compilers_and_libraries/linux/mpi/intel64/bin/mpivars.sh
spack install --dirty hdf5+fortran+hl+mpi %[email protected] ^intel-mpi

And pass them onto the spack compiler with the spack install --dirty option.

This should fix the issues. I will be reporting this to Intel too but I am posting it here in case anyone is stuck with a similar issue.

[JiaweiZhuang]

I can confirm the use mpi problem reported by @jimmielin . The default compiler (GCC 4.8.5 on CentOS 7) works fine, but a very new version (GCC 9.2.0) has this problem. Just put the steps here for record.

$ spack -v install [email protected]
$ spack compiler add $(spack location -i [email protected])
$ export PATH=$(spack location -i gcc)/bin:$PATH
$ module load intelmpi 
$ mpicc --version   # wraps a new version of gcc
gcc (GCC) 9.2.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Test program test.f90:

program test
use mpi
integer ierr

call mpi_init(ierr)
print *, "Hello world"
call mpi_finalize(ierr)

stop
end

Error:

$ mpif90 test.f90
test.f90:2.4:

use mpi
    1
Fatal Error: File 'mpi.mod' opened at (1) is not a GNU Fortran module file

This is indeed an Intel-MPI problem and should be raised upstream instead of here. Just in case people hit the same issue. For example WRF has a ton of use mpi statements.

[lukeseawalker]

@jimmielin and @JiaweiZhuang thanks for sharing!

[rajachan]

As an update to this issue, shared-memory support has been directly added to the EFA provider in libfabric. Open MPI can now leverage this to optimize node-local communication without any modifications to the OFI MTL. This is available as part of libfabric 1.9 that was released a couple days back.

[JiaweiZhuang]

I can confirm that the latency problem is now solved by libfabric 1.9 (installable via Spack spack/spack#13964).

spack -v install osu-micro-benchmarks ^[email protected]+pmi schedulers=slurm fabrics=libfabric ^[email protected] fabrics=efa
cd $(spack location -i osu-micro-benchmarks)/libexec/osu-micro-benchmarks/mpi/pt2pt/
srun -N 1 -n 2 ./osu_latency

which gives:

# OSU MPI Latency Test v5.6.1
# Size          Latency (us)
0                       1.10
1                       1.99
2                       1.00
4                       0.99
8                       1.00
16                      1.10
32                      1.14
64                      1.14
128                     1.14
256                     1.17
512                     1.24
1024                    1.37
2048                    1.58
4096                    3.03
8192                    5.64
16384                   6.87
32768                   7.46
65536                   8.85
131072                 11.59
262144                 16.95
524288                 31.36
1048576                85.78
2097152               218.69
4194304               420.02

[jimmielin]

Thanks @rajachan and thanks @JiaweiZhuang for verifying this. The issue can be closed now.