Ompi: SLURM integration appears broken

Created on 14 Oct 2017  Â·  32Comments  Â·  Source: open-mpi/ompi

I have checked master, v3.1.x, and v3.0.x. Only master appears to be broken, though the reason doesn't seem obvious as the code that is failing doesn't appear to be different on master. I definitely configured --with-pmi and confirmed that the s2 and s1 components were built. I set pmix_base_verbose and confirmed that the proper component is being selected.

Here is the error:

$ srun -N 2 --mpi=pmi2  ./ring
[rhc001:111766] PMI_Get_universe_size [pmix_s2.c:288:s2_init]: Operation failed
--------------------------------------------------------------------------
The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM's PMI support and therefore cannot
execute. There are several options for building PMI support under
SLURM, depending upon the SLURM version you are using:

  version 16.05 or later: you can use SLURM's PMIx support. This
  requires that you configure and build SLURM --with-pmix.

  Versions earlier than 16.05: you must use either SLURM's PMI-1 or
  PMI-2 support. SLURM builds PMI-1 by default, or you can manually
  install PMI-2. You must then build Open MPI using --with-pmi pointing
  to the SLURM PMI library location.

Please configure as appropriate and try again.
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[rhc001:111766] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: rhc001: task 0: Exited with exit code 1
[rhc002.cluster:27298] PMI_Get_universe_size [pmix_s2.c:288:s2_init]: Operation failed
--------------------------------------------------------------------------
The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM's PMI support and therefore cannot
execute. There are several options for building PMI support under
SLURM, depending upon the SLURM version you are using:

  version 16.05 or later: you can use SLURM's PMIx support. This
  requires that you configure and build SLURM --with-pmix.

  Versions earlier than 16.05: you must use either SLURM's PMI-1 or
  PMI-2 support. SLURM builds PMI-1 by default, or you can manually
  install PMI-2. You must then build Open MPI using --with-pmi pointing
  to the SLURM PMI library location.

Please configure as appropriate and try again.
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[rhc002.cluster:27298] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: rhc002: task 1: Exited with exit code 1

@artpol84 Can you folks please take a look?

RTE master v3.0.x v3.1.x bug
Source
picture rhc54
👍1

Most helpful comment

@pllopis @henkela Count a +1 to the number of people hit by --enable-static that disables OpenMPI's support for slurm (using pmi2). Thanks for the hint, without that it would have been a long fight to debug the issue.

picture alalazo on 3 May 2018
👍4

All 32 comments

Never mind - I found the problem. I had enabled installation of libpmix and that also installed a libpmi2 in the OMPI prefix location. Since that came first, the procs were picking it up and failing when attempting to connect to slurm's pmi daemon.

rhc54 picture rhc54 on 14 Oct 2017

@rhc54

I also encounter similar trouble, with configure: 

$ROOT_DIR/autogen.pl && \
  ./configure \
  --prefix=$ROOT_DIR/output \
  --with-slurm \
  --with-pmix \
  --with-hwloc

then, the following log: 

--------------------------------------------------------------------------
The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM's PMI support and therefore cannot
execute. There are several options for building PMI support under
SLURM, depending upon the SLURM version you are using:

  version 16.05 or later: you can use SLURM's PMIx support. This
  requires that you configure and build SLURM --with-pmix.

  Versions earlier than 16.05: you must use either SLURM's PMI-1 or
  PMI-2 support. SLURM builds PMI-1 by default, or you can manually
  install PMI-2. You must then build Open MPI using --with-pmi pointing
  to the SLURM PMI library location.

Please configure as appropriate and try again.

also be found.

Have some more guide to kick this issue?

backyes picture backyes on 28 Dec 2017

I'm not sure how to advise you without knowing what version of SLURM and OMPI you are talking about. In general, the problem is that you configured OMPI without specifying --with-pmi and pointing to the location of the SLURM PMI libraries. If you are using a new enough version of SLURM, _and_ you configured it with PMIx support, then that would also solve the problem.

There is some more info here about the latter option.

rhc54 picture rhc54 on 2 Jan 2018

Dear @rhc54

recently I obtained the very same error message when running srun with openmpi 2.1.2 and 3.0.0.
I'm using slurm 17.02 on centos 7.3. Slurm is built with pmi2 but without pmix.
An old version of openmpi - 1.10.4 - works using srun --mpi=pmi2 but when I install 2.1.2 or 3.0.0 with the same configure-line srun --mpi=pmi2 shows

--------------------------------------------------------------------------
PMI2_Init failed to intialize.  Return code: 1

--------------------------------------------------------------------------
--------------------------------------------------------------------------
The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM's PMI support and therefore cannot
execute. There are several options for building PMI support under
SLURM, depending upon the SLURM version you are using:

  version 16.05 or later: you can use SLURM's PMIx support. This
  requires that you configure and build SLURM --with-pmix.

  Versions earlier than 16.05: you must use either SLURM's PMI-1 or
  PMI-2 support. SLURM builds PMI-1 by default, or you can manually
  install PMI-2. You must then build Open MPI using --with-pmi pointing
  to the SLURM PMI library location.

Please configure as appropriate and try again.
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[x0286:240629] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: x0286: task 0: Exited with exit code 1
srun: Terminating job step 950183.0

Mpirun works.
Is it mandatory for openmpi 2.1.2 and 3.0.0 to built slurm with pmix?
The configure-line looks like
./configure --prefix= --enable-shared --enable-mpi-thread-multiple --with-psm2 --enable-mpirun-prefix-by-default --disable-dlopen --with-pmi=/usr --with-slurm
And the compiler is GCC 6.3.0.

Any hints welcome.

Best regards,

henkela picture henkela on 24 Jan 2018
👍2

No, you don't have to build slurm with pmix. However, you may be picking up the pmix-based PMI2 library by mistake. Check the OMPI install/lib location - if there is a libpmi2.* there, delete it.

rhc54 picture rhc54 on 25 Jan 2018

Thank you for your quick response.

That's the strange thing about it - there is no libpmi2 in OMPI/lib. ls shows

libmpi.la               libmpi.so.40.0.0             libmpi_usempi_ignore_tkr.so.40      libopen-pal.so         liboshmem.la           mpi_f08_interfaces_callbacks.mod  pmpi_f08_interfaces.mod
libmpi_mpifh.la         libmpi_usempif08.la          libmpi_usempi_ignore_tkr.so.40.0.0  libopen-pal.so.40      liboshmem.so           mpi_f08_interfaces.mod
libmpi_mpifh.so         libmpi_usempif08.so          libompitrace.la                     libopen-pal.so.40.0.0  liboshmem.so.40        mpi_f08.mod
libmpi_mpifh.so.40      libmpi_usempif08.so.40       libompitrace.so                     libopen-rte.la         liboshmem.so.40.0.0    mpi_f08_types.mod
libmpi_mpifh.so.40.0.0  libmpi_usempif08.so.40.0.0   libompitrace.so.40                  libopen-rte.so         mpi_ext.mod            mpi.mod
libmpi.so               libmpi_usempi_ignore_tkr.la  libompitrace.so.40.0.0              libopen-rte.so.40      mpi_f08_callbacks.mod  openmpi
libmpi.so.40            libmpi_usempi_ignore_tkr.so  libopen-pal.la                      libopen-rte.so.40.0.0  mpi_f08_ext.mod        pkgconfig

I did a strace of a hello world mpi program. and extracted the successful opens from the trace. This is how it looks for OpenMPI 3.0.0 

open("/cluster/easybuild/broadwell/software/mpi/OpenMPI/3.0.0-GCC-6.3.0/lib/libmpi.so.40", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/liblustreapi.so", O_RDONLY|O_CLOEXEC) = 3
open("/cluster/easybuild/broadwell/software/mpi/OpenMPI/3.0.0-GCC-6.3.0/lib/libopen-rte.so.40", O_RDONLY|O_CLOEXEC) = 3
open("/cluster/easybuild/broadwell/software/mpi/OpenMPI/3.0.0-GCC-6.3.0/lib/libopen-pal.so.40", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libfabric.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libnuma.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libpmi2.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libpmi.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libpsm2.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libutil.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/librdmacm.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libibverbs.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libnl-route-3.so.200", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libpsm_infinipath.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libnl-3.so.200", O_RDONLY|O_CLOEXEC) = 3
open("/cluster/easybuild/broadwell/software/compiler/GCCcore/6.3.0/lib64/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libslurm.so.31", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libinfinipath.so.4", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libuuid.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/proc/net/psched", O_RDONLY) = 3
read(3, "000003e8 00000040 000f4240 3b9ac"..., 1024) = 36
open("/etc/libnl/classid", O_RDONLY) = 3
open("/proc/self/status", O_RDONLY) = 3
openat(AT_FDCWD, "/sys/devices/system/node", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
getdents(3, {{d_ino=8349, d_off=1, d_reclen=24, d_name=".", d_type=DT_DIR} {d_ino=9, d_off=312848657, d_reclen=24, d_name="..", d_type=DT_DIR} {d_ino=8405, d_off=312848833, d_reclen=32, d_name="node0", d_type=DT_DIR} {d_ino=8934, d_off=318580955, d_reclen=32, d_name="node1", d_type=DT_DIR} {d_ino=8356, d_off=900028725, d_reclen=32, d_name="power", d_type=DT_DIR} {d_ino=8351, d_off=906194608, d_reclen=32, d_name="possible", d_type=DT_REG} {d_ino=8353, d_off=1266072600, d_reclen=40, d_name="has_normal_memory", d_type=DT_REG} {d_ino=8355, d_off=1321717208, d_reclen=32, d_name="has_cpu", d_type=DT_REG} {d_ino=8352, d_off=1472641949, d_reclen=32, d_name="online", d_type=DT_REG} {d_ino=8350, d_off=1771120378, d_reclen=32, d_name="uevent", d_type=DT_REG} {d_ino=8354, d_off=2147483647, d_reclen=32, d_name="has_memory", d_type=DT_REG}}, 32768) = 344
openat(AT_FDCWD, "/sys/devices/system/cpu", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
open("/proc/self/status", O_RDONLY) = 3
open("/dev/urandom", O_RDONLY|O_NOFOLLOW) = 3
open("/cluster/easybuild/broadwell/software/mpi/OpenMPI/3.0.0-GCC-6.3.0/etc/openmpi-mca-params.conf", O_RDONLY) = 3
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 3
open("/proc/cpuinfo", O_RDONLY) = 3
read(10, "# -*- text -*-\n#\n# Copyright (c)"..., 1048576) = 3358
write(2, "*** An error occurred in MPI_Ini"..., 34) = 34
write(2, "***    and potentially your MPI "..., 37) = 37

When I use our OpenMPI 1.10.4 the same operation shows

open("/cluster/easybuild/broadwell/software/mpi/OpenMPI/1.10.4-GCC-6.3.0/lib/libmpi.so.12", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libpsm2.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libfabric.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/cluster/easybuild/broadwell/software/mpi/OpenMPI/1.10.4-GCC-6.3.0/lib/libopen-rte.so.12", O_RDONLY|O_CLOEXEC) = 3
open("/cluster/easybuild/broadwell/software/mpi/OpenMPI/1.10.4-GCC-6.3.0/lib/libopen-pal.so.13", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libpmi.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libpmi2.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libnuma.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libutil.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/cluster/easybuild/broadwell/software/compiler/GCCcore/6.3.0/lib64/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/librdmacm.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libibverbs.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libnl-route-3.so.200", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libpsm_infinipath.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libnl-3.so.200", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libslurm.so.31", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libinfinipath.so.4", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libuuid.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/proc/net/psched", O_RDONLY) = 3
read(3, "000003e8 00000040 000f4240 3b9ac"..., 1024) = 36
open("/etc/libnl/classid", O_RDONLY) = 3
open("/proc/self/status", O_RDONLY) = 3
openat(AT_FDCWD, "/sys/devices/system/node", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
getdents(3, {{d_ino=8349, d_off=1, d_reclen=24, d_name=".", d_type=DT_DIR} {d_ino=9, d_off=312848657, d_reclen=24, d_name="..", d_type=DT_DIR} {d_ino=8405, d_off=312848833, d_reclen=32, d_name="node0", d_type=DT_DIR} {d_ino=8934, d_off=318580955, d_reclen=32, d_name="node1", d_type=DT_DIR} {d_ino=8356, d_off=900028725, d_reclen=32, d_name="power", d_type=DT_DIR} {d_ino=8351, d_off=906194608, d_reclen=32, d_name="possible", d_type=DT_REG} {d_ino=8353, d_off=1266072600, d_reclen=40, d_name="has_normal_memory", d_type=DT_REG} {d_ino=8355, d_off=1321717208, d_reclen=32, d_name="has_cpu", d_type=DT_REG} {d_ino=8352, d_off=1472641949, d_reclen=32, d_name="online", d_type=DT_REG} {d_ino=8350, d_off=1771120378, d_reclen=32, d_name="uevent", d_type=DT_REG} {d_ino=8354, d_off=2147483647, d_reclen=32, d_name="has_memory", d_type=DT_REG}}, 32768) = 344
openat(AT_FDCWD, "/sys/devices/system/cpu", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
open("/proc/self/status", O_RDONLY) = 3
open("/cluster/easybuild/broadwell/software/mpi/OpenMPI/1.10.4-GCC-6.3.0/etc/openmpi-mca-params.conf", O_RDONLY) = 3
read(8, "11:blkio:/\n10:freezer:/slurm/uid"..., 1024) = 321
read(8, "...............................\n", 4096) = 32
read(8, "SE5C610.86B.01.01.0019.101220160"..., 4096) = 36

To me it looks like the same sys-libs are used and the very same libpmi2.so but OpenMpi 3.0.0 uses additionally liblustreapi and libz and the order of opening the libs is different. Maybe you have an idea what I missed or could check?
(openmpi-mca-params.conf is identical)

For completeness, here's my srun call
srun --mpi=pmi2 -n1 strace -t -v -o local.trace ./dummy

Thanks in advance

henkela picture henkela on 25 Jan 2018

Thanks for the info! Looks like you are indeed pulling the correct libs, so it sounds like there is a bug in OMPI itself. I'm afraid I can't take this any further right now, but hopefully can point the right folks towards it.

@artpol84 @hppritcha @jsquyres @bwbarrett It appears that the Slurm native PMI support in the OMPI v3.x series (the s1 and s2 components in the opal/pmix framework) is broken. I would suggest someone fix it prior to the next releases.

rhc54 picture rhc54 on 25 Jan 2018

I'd like to add: I see a similar problem when I try with OMPI 2.1.2...
Here's the trace of that: 

open("/cluster/easybuild/broadwell/software/mpi/OpenMPI/2.1.2-GCC-6.3.0/lib/libmpi.so.20", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/liblustreapi.so", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libpsm2.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/cluster/easybuild/broadwell/software/mpi/OpenMPI/2.1.2-GCC-6.3.0/lib/libopen-rte.so.20", O_RDONLY|O_CLOEXEC) = 3
open("/cluster/easybuild/broadwell/software/mpi/OpenMPI/2.1.2-GCC-6.3.0/lib/libopen-pal.so.20", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libfabric.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libnuma.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libpmi.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libpmi2.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libutil.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/cluster/easybuild/broadwell/software/compiler/GCCcore/6.3.0/lib64/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/librdmacm.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libibverbs.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libnl-route-3.so.200", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libpsm_infinipath.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libnl-3.so.200", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libslurm.so.31", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libinfinipath.so.4", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/libuuid.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/proc/net/psched", O_RDONLY) = 3
read(3, "000003e8 00000040 000f4240 3b9ac"..., 1024) = 36
open("/etc/libnl/classid", O_RDONLY) = 3
open("/proc/self/status", O_RDONLY) = 3
openat(AT_FDCWD, "/sys/devices/system/node", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
getdents(3, {{d_ino=8349, d_off=1, d_reclen=24, d_name=".", d_type=DT_DIR} {d_ino=9, d_off=312848657, d_reclen=24, d_name="..", d_type=DT_DIR} {d_ino=8405, d_off=312848833, d_reclen=32, d_name="node0", d_type=DT_DIR} {d_ino=8934, d_off=318580955, d_reclen=32, d_name="node1", d_type=DT_DIR} {d_ino=8356, d_off=900028725, d_reclen=32, d_name="power", d_type=DT_DIR} {d_ino=8351, d_off=906194608, d_reclen=32, d_name="possible", d_type=DT_REG} {d_ino=8353, d_off=1266072600, d_reclen=40, d_name="has_normal_memory", d_type=DT_REG} {d_ino=8355, d_off=1321717208, d_reclen=32, d_name="has_cpu", d_type=DT_REG} {d_ino=8352, d_off=1472641949, d_reclen=32, d_name="online", d_type=DT_REG} {d_ino=8350, d_off=1771120378, d_reclen=32, d_name="uevent", d_type=DT_REG} {d_ino=8354, d_off=2147483647, d_reclen=32, d_name="has_memory", d_type=DT_REG}}, 32768) = 344
openat(AT_FDCWD, "/sys/devices/system/cpu", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
open("/proc/self/status", O_RDONLY) = 3
open("/dev/urandom", O_RDONLY|O_NOFOLLOW) = 3
open("/cluster/easybuild/broadwell/software/mpi/OpenMPI/2.1.2-GCC-6.3.0/etc/openmpi-mca-params.conf", O_RDONLY) = 3
open("/proc/cpuinfo", O_RDONLY) = 3
shmget(IPC_PRIVATE, 4096, IPC_CREAT|IPC_EXCL|0700) = 303398912
read(7, "-67803@z0092_0/43179/pmix_dstor_"..., 255) = 38
sendto(19, "\377\377\377\377\377\377\377\377\25\0\0\0\0\0\0\0002829778945\0\0\0\0\0001"..., 37, 0, NULL, 0) = 37
read(20, "# -*- text -*-\n#\n# Copyright (c)"..., 1048576) = 3846
write(2, "*** An error occurred in MPI_Ini"..., 34) = 34
write(2, "***    and potentially your MPI "..., 37) = 37

Here's STDERR for OMPI2.1.2:

--------------------------------------------------------------------------
PMI2_Init failed to intialize.  Return code: 1

--------------------------------------------------------------------------
 PMI2_Info_GetJobAttr [pmix_s2.c:311:s2_init]: Operation failed
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  init pmix failed
  --> Returned value Error (-1) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  orte_ess_init failed
  --> Returned value Error (-1) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  ompi_mpi_init: ompi_rte_init failed
  --> Returned "Error" (-1) instead of "Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)

and for OMPI 3.0.0

--------------------------------------------------------------------------
PMI2_Init failed to intialize.  Return code: 1

--------------------------------------------------------------------------
--------------------------------------------------------------------------
The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM's PMI support and therefore cannot
execute. There are several options for building PMI support under
SLURM, depending upon the SLURM version you are using:

  version 16.05 or later: you can use SLURM's PMIx support. This
  requires that you configure and build SLURM --with-pmix.

  Versions earlier than 16.05: you must use either SLURM's PMI-1 or
  PMI-2 support. SLURM builds PMI-1 by default, or you can manually
  install PMI-2. You must then build Open MPI using --with-pmi pointing
  to the SLURM PMI library location.

Please configure as appropriate and try again.
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
henkela picture henkela on 25 Jan 2018
👍1

Just as a quick sanity check: did you by any chance do an rpm install of pmix to the default locations (i.e., /usr)? We have seen problems where the packagers included libpmi and libpmi2 from PMIx in their distros, and they overwrite the ones from Slurm - causing a similar problem to what you report.

If so, you might want to reinstall the Slurm package to restore their libpmi and libpmi2.

rhc54 picture rhc54 on 25 Jan 2018

I checked yum provides */libpmi2.so, which gave me only our self-build rpms.
yum info pmix show Available Packages only.

henkela picture henkela on 25 Jan 2018

Hi,
I have also run into this issue on OpenMPI 3.0.0. I can't comment on other versions since I do not have them at hand to try, but I am pretty sure the same issue was present on 2.x.

In my case, I am building --with-slurm and --with-pmi=/usr.

First: According to the docs at https://www.open-mpi.org/faq/?category=slurm#slurm-direct-srun-mpi-apps I would have expected to supply --with-pmi=/usr/include/slurm, but in that case pmi2.h is not found. It is only found if I specify the /usr prefix due to the configure logic. Should I report this as a bug?

Second: I can work around this issue by not using --enable-static. It does not make sense to me that using --enable-static, which also implies --disable-mca-dso, would disable slurm support, and I also did not find this documented, so it is probably a bug. After removing --enable-static from my config flags, srun finally works as expected.

@rhc54 @henkela could you confirm whether you also experience this behaviour?

Would there be a way of using --enable-static and having slurm support?

Thanks

pllopis picture pllopis on 20 Apr 2018

Thanks for the hint. I'll test it ASAP and get back to you.

Am 20.04.2018 um 16:25 schrieb Pablo Llopis <[email protected]notifications@github.com>:

Hi,
I have also run into this issue on OpenMPI 3.0.0. I can't comment on other versions since I do not have them at hand to try, but I am pretty sure the same issue was present on 2.x.

In my case, I am building --with-slurm and --with-pmi=/usr.

First: According to the docs at https://www.open-mpi.org/faq/?category=slurm#slurm-direct-srun-mpi-apps I would have expected to supply --with-pmi=/usr/include/slurm, but in that case pmi2.h is not found. It is only found if I specify the /usr prefix due to the configure logic. Should I report this as a bug?

Second: I can work around this issue by not using --enable-static. It does not make sense to me that using --enable-static, which also implies --disable-mca-dso would disable slurm support, and I also did not find this documented, so it is probably a bug. After removing --enable-static from my config flags, srun finally works as expected.

@rhc54https://github.com/rhc54 @henkelahttps://github.com/henkela could you confirm whether you also experience this behaviour?

Would there be a way of using --enable-static and having slurm support?

Thanks

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/open-mpi/ompi/issues/4338#issuecomment-383112852, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AIQ31-5uoeEeejPySczXtOjhGMGh9HRZks5tqe-ngaJpZM4P5OkF.

henkela picture henkela on 24 Apr 2018
👍1

I've checked it. We didn't use enable-static but --disable-dlopen. When I removed --disable-dlopen and rebuild srun started working. I'm just noch sure if it's about the static or about the implied disable-mca-dso...

Von: Pablo Llopis notifications@github.com
Gesendet: Freitag, 20. April 2018 16:24:40
An: open-mpi/ompi
Cc: Henkel, Andreas; Mention
Betreff: Re: [open-mpi/ompi] SLURM integration appears broken (#4338)

Hi,
I have also run into this issue on OpenMPI 3.0.0. I can't comment on other versions since I do not have them at hand to try, but I am pretty sure the same issue was present on 2.x.

In my case, I am building --with-slurm and --with-pmi=/usr.

First: According to the docs at https://www.open-mpi.org/faq/?category=slurm#slurm-direct-srun-mpi-apps I would have expected to supply --with-pmi=/usr/include/slurm, but in that case pmi2.h is not found. It is only found if I specify the /usr prefix due to the configure logic. Should I report this as a bug?

Second: I can work around this issue by not using --enable-static. It does not make sense to me that using --enable-static, which also implies --disable-mca-dso would disable slurm support, and I also did not find this documented, so it is probably a bug. After removing --enable-static from my config flags, srun finally works as expected.

@rhc54https://github.com/rhc54 @henkelahttps://github.com/henkela could you confirm whether you also experience this behaviour?

Would there be a way of using --enable-static and having slurm support?

Thanks

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/open-mpi/ompi/issues/4338#issuecomment-383112852, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AIQ31-5uoeEeejPySczXtOjhGMGh9HRZks5tqe-ngaJpZM4P5OkF.

henkela picture henkela on 26 Apr 2018

I'm not using --disable-dlopen, but it does sound that this --disable-mca-dso flag that is implied by both --disable-dlopen and --enable-static might be the cause of this issue.

For the record, I'm building OpenMPI by invoking ./configure as follows, straight from my specfile:

./configure --prefix=%{install_prefix} \
            --libdir=%{install_prefix}/lib64 \
            --enable-mpi-cxx --enable-cxx-exceptions --enable-mpi-thread-multiple --enable-orterun-prefix-by-default \
            --with-slurm --with-pmi=/usr
            CFLAGS='-m64 -O2 -pipe -Wall -Wshadow' \
            CXXFLAGS='-m64 -O2 -pipe -Wall -Weffc++ -Wshadow' \
            FCFLAGS='-m64 -O2 -pipe -Wall' \
            FFLAGS='-m64 -O2 -pipe -Wall'

And as I said, srun stops working just by adding --enable-static. I have not yet tried adding just --disable-mca-dso to the above flags to confirm if that's really the issue, although it certainly seems like it.

pllopis picture pllopis on 26 Apr 2018

@pllopis @henkela Count a +1 to the number of people hit by --enable-static that disables OpenMPI's support for slurm (using pmi2). Thanks for the hint, without that it would have been a long fight to debug the issue.

alalazo picture alalazo on 3 May 2018
👍4

@rhc54 @hjelmn Is this issue addressed by #5103?

jsquyres picture jsquyres on 4 May 2018

Afraid not - this is probably something to do with the internal PMIx support not turning off the build of the backward compatibility libraries, thus creating duplicate PMI2 symbols. You might check the configure.m4 in opal/mca/pmix/pmix2x to ensure those libraries aren't being built.

rhc54 picture rhc54 on 4 May 2018

We turned off the pmi1/2 compatibility libraries in the internal build, so I think this is fixed

rhc54 picture rhc54 on 1 Oct 2018

Reading through the thread here I don't understand if this is a bug or undocumented behaviour. But I can still reproduce the issue with OpenMPI 2.1.5 and 3.1.3. If I build OpenMPI with --enable-static Slurm/PMI2 support is broken. I've confirmed this with Slurm 17.11.2 and the latest 18.08.3. OS is Scientific Linux 7 with gcc 4.8.5.

[gpu14:07175] PMI_Get_universe_size [../../../../../openmpi-2.1.5/opal/mca/pmix/s2/pmix_s2.c:276:s2_init]: Operation failed
[gpu14:07176] PMI_Get_universe_size [../../../../../openmpi-2.1.5/opal/mca/pmix/s2/pmix_s2.c:276:s2_init]: Operation failed
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  getting local rank failed
  --> Returned value Error (-1) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
[...]

Full error messages here: https://pastebin.com/raw/gnggxWv3

Configure flags used:

../openmpi-2.1.5/configure --prefix=/sw/openmpi/2.1.5 --with-hwloc --with-munge --with-pmi --with-slurm --with-verbs=no --enable-shared --enable-static --enable-orterun-prefix-by-default
stiobhan picture stiobhan on 30 Nov 2018

@jsquyres This is a config issue in OPAL, I think. The issue here is that building static breaks the Slurm PMI2 component (opal/mca/pmix/s2). I suspect it breaks the s1 component as well. Someone will need to look at those, I guess - it's outside my purview.

rhc54 picture rhc54 on 30 Nov 2018

I thought I checked this more thoroughly before posting, but I must have mixed something up. I've check again now with all recent versions, and I can only reproduce it with 2.1.5.

Versions 3.0.3, 3.1.3 and 4.0.0 seem to work fine with --enable-shared --enable-static.

stiobhan picture stiobhan on 3 Dec 2018

@rhc54 is this still relevant?

artpol84 picture artpol84 on 30 May 2019

Don't know - I don't believe we still support the OMPI v2.x series any more, so I guess if someone can confirm this works with the v3.x and v4.x releases, I imagine it could be closed. However, I would suggest someone check those releases before closing this as I see references to this ticket since @stiobhan last post.

rhc54 picture rhc54 on 31 May 2019

Don't know - I don't believe we still support the OMPI v2.x series any more, so I guess if someone can confirm this works with the v3.x and v4.x releases, I imagine it could be closed.

If it matters, when I was hitting the issue above it was with version 3.0.1

alalazo picture alalazo on 13 Jun 2019

@karasevb please have a look

artpol84 picture artpol84 on 13 Jun 2019

@alalazo thank you for checking

artpol84 picture artpol84 on 13 Jun 2019
👍1

Hello guys, I came into the same problem, with openmpi3.1.0, SLURM 16.05.3. I want to run the CESM climate model with openmpi, but got the running error:

srun.sw: job 45954906 queued and waiting for resources
srun.sw: job 45954906 has been allocated resources
[cn060952:28261] OPAL ERROR: Not initialized in file pmix2x_client.c at line 109

The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM's PMI support and therefore cannot
execute. There are several options for building PMI support under
SLURM, depending upon the SLURM version you are using:
  version 16.05 or later: you can use SLURM's PMIx support. This
  requires that you configure and build SLURM --with-pmix.
  Versions earlier than 16.05: you must use either SLURM's PMI-1 or
  PMI-2 support. SLURM builds PMI-1 by default, or you can manually
  install PMI-2. You must then build Open MPI using --with-pmi pointing
  to the SLURM PMI library location.

Please configure as appropriate and try again.
 An error occurred in MPI_Init
 on a NULL communicator
 MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, and potentially your MPI job)
[cn060952:28261] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

The technician on the supercomputer server told me to configure the openmpi again with the address to their SLURM lib. So I configured the openmpi3.1.0 again as follows:

./configure --prefix=/home/export/online1/zhub/openmpi-3_1_0 --with-pmi=/usr/sw-cluster/slurm-16.05.3 FC=pgf90 CC=pgcc

But I failed in the make step :

make[3]: Leaving directory `/home/export/online1/zhub/openmpi-3.1.0/opal/mca/pmix/pmix2x/pmix'
make[3]: Entering directory `/home/export/online1/zhub/openmpi-3.1.0/opal/mca/pmix/pmix2x'
  CC       mca_pmix_pmix2x_la-pmix2x_component.lo
  CC       mca_pmix_pmix2x_la-pmix2x.lo
  CC       mca_pmix_pmix2x_la-pmix2x_client.lo
  CC       mca_pmix_pmix2x_la-pmix2x_local.lo
  CC       mca_pmix_pmix2x_la-pmix2x_server_south.lo
  CC       mca_pmix_pmix2x_la-pmix2x_server_north.lo
  CCLD     mca_pmix_pmix2x.la
make[3]: Leaving directory `/home/export/online1/zhub/openmpi-3.1.0/opal/mca/pmix/pmix2x'
make[2]: Leaving directory `/home/export/online1/zhub/openmpi-3.1.0/opal/mca/pmix/pmix2x'
Making all in mca/pmix/s1
make[2]: Entering directory `/home/export/online1/zhub/openmpi-3.1.0/opal/mca/pmix/s1'
  CC       mca_pmix_s1_la-pmix_s1_component.lo
  CC       mca_pmix_s1_la-pmix_s1.lo
  CCLD     mca_pmix_s1.la
pgcc-Error-Unknown switch: -pthread
make[2]: *** [mca_pmix_s1.la] wrong 1
make[2]: Leaving directory `/home/export/online1/zhub/openmpi-3.1.0/opal/mca/pmix/s1'
make[1]: *** [all-recursive] wrong 1
make[1]: Leaving directory `/home/export/online1/zhub/openmpi-3.1.0/opal'
make: *** [all-recursive] wrong 1

So how can I build openmpi3.1.0 and link it to SLURM 16.05.3 without error? It seems like the supercomputer system I'm using only has pmi with its SLURM without pmix, but why did this pmix error come out in the openmpi make-step as mentioned above? here's what the SLRUM16.05.3 looking like:

[zhub@psn004` ~]$ cd /usr/sw-cluster/slurm-16.05.3/
[zhub@psn004 slurm-16.05.3]$ ls
include  lib  share
[zhub@psn004 slurm-16.05.3]$ cd include/
[zhub@psn004 include]$ ls
slurm
[zhub@psn004 include]$ cd slurm/
[zhub@psn004 slurm]$ ls
pmi.h  slurmdb.h  slurm_errno.h  slurm.h  smd_ns.h  spank.h
[zhub@psn004 slurm]$ cd ../../lib/
[zhub@psn004 lib]$ ls
libpmi.a   libpmi.so    libpmi.so.0.0.0  libslurmdb.a   libslurmdb.so     libslurmdb.so.30.0.0  libslurm.so     libslurm.so.30.0.0
libpmi.la  libpmi.so.0  libslurm.a       libslurmdb.la  libslurmdb.so.30  libslurm.la           libslurm.so.30  slurm

Would there be any advice or support?
Thanks in advance
@rhc54

dadada15 picture dadada15 on 3 Jul 2019

@dadada15 have you tried with the latest patch release in the 3.1 series? That would be 3.1.4.

stiobhan picture stiobhan on 3 Jul 2019

@dadada15 have you tried with the latest patch release in the 3.1 series? That would be 3.1.4.

Yes, I tried openmpi3.1.4 but got the same problem. Is this configure command somewhere wrong?

./configure --prefix=/home/export/online1/zhub/openmpi-3_1_4 --with-pmi=/usr/sw-cluster/slurm-16.05.3 FC=pgf90 CC=pgcc
dadada15 picture dadada15 on 4 Jul 2019

The SLURM in the supercomputer system seems only has pmi lib withou pmix, so I configure my openmpi with "--with-pmi=/usr/sw-cluster/slurm-16.05.3". However, when I finished configure step, I found this info

Open MPI configuration:
-----------------------
Version: 3.1.4
Build MPI C bindings: yes
Build MPI C++ bindings (deprecated): no
Build MPI Fortran bindings: mpif.h, use mpi
MPI Build Java bindings (experimental): no
Build Open SHMEM support: yes
Debug build: no
Platform file: (none)

Miscellaneous
-----------------------
CUDA support: no
PMIx support: internal

It said that pmix is internal supported, while the "./cofigure -help" said that

--with-pmix(=DIR)       Build PMIx support. DIR can take one of three
                          values: "internal", "external", or a valid directory
                          name. "internal" (or no DIR value) forces Open MPI
                          to use its internal copy of PMIx. "external" forces
                          Open MPI to use an external installation of PMIx.
                          Supplying a valid directory name also forces Open
                          MPI to use an external installation of PMIx, and
                          adds DIR/include, DIR/lib, and DIR/lib64 to the
                          search path for headers and libraries. Note that
                          Open MPI does not support --without-pmix.

My understanding is that openmpi will find the pmix when the pmix is internal supported, but the truth is there's no pmix in the SLURM.
So how can I build openmpi and link it to SLURM 16.05.3(only has pmi) without error? In other words, how can I organize the "./configure" command?

dadada15 picture dadada15 on 4 Jul 2019

@dadada15 "internal" pmix is the one that is built-in into the OMPI tarball. It will be always found.
"internal" version is always used when you are launching with mpirun.
OMPI can support PMI1/2 and PMIx in parallel and select between them at runtime. I.e. it will select PMI1 when you launch with srun and use PMIx when launced with mpirun.

My comment doesn't address the main issue though.

artpol84 picture artpol84 on 6 Jul 2019

@dadada15 "internal" pmix is the one that is built-in into the OMPI tarball. It will be always found.
"internal" version is always used when you are launching with mpirun.
OMPI can support PMI1/2 and PMIx in parallel and select between them at runtime. I.e. it will select PMI1 when you launch with srun and use PMIx when launced with mpirun.

My comment doesn't address the main issue though.

Thanks for your answer. Now I have a general understanding of the relationship between ompi,pmi and pmix。

dadada15 picture dadada15 on 7 Jul 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

sagitter picture sagitter  Â·  5Comments
dirk-schubert-arm picture dirk-schubert-arm  Â·  4Comments
yoshihingis picture yoshihingis  Â·  3Comments
bodgerer picture bodgerer  Â·  8Comments
oneZeroOneZeroOne picture oneZeroOneZeroOne  Â·  5Comments