Ompi: hwloc Error while building OpenMPI v. 4.0.1

[moravveji]

Dear Great MPI guys,

I am trying to build a CUDA-aware OpenMPI/4.0.1 on our compute cluster. We are running CentOS 7.4 on all nodes (SkyLake processors with IB interconnect and 4xP100 Nvidia GPUs). I am installing from the tarball. I use GCC v. 6.4, hwloc v. 2.0.2 and CUDA v. 10.1

I have attached the script that I use building OpenMPI, together with the redirected stdout/stderr. I get the following hwloc errors:

../../../opal/.libs/libopen-pal.so: error: undefined reference to 'hwloc_distances_get_by_depth'
../../../opal/.libs/libopen-pal.so: error: undefined reference to 'hwloc_distances_release'
../../../opal/.libs/libopen-pal.so: error: undefined reference to 'hwloc_topology_set_io_types_filter'
../../../opal/.libs/libopen-pal.so: error: undefined reference to 'hwloc_shmem_topology_adopt'
collect2: error: ld returned 1 exit status
make[2]: *** [opal_wrapper] Error 1

I am using the hwloc module built using EasyBuild (easyconfig file) after adjusting the dependency versions. I get the same errors if I use the internal hwloc (i.e. setting --with-hwloc=internal).

Is there something not correct with the current hwloc that I am using? Why the internal hwloc raises the same errors? What solution do you propose to proceed with the installation?

With kind regards,
Ehsan

openmpi.zip
openmpi.log.txt.zip

[jsquyres]

I don't see an obvious reason why this would happen -- I am able to build Open MPI v4.0.1 properly with the internal hwloc and with an external hwloc 2.0.2 configured the way you have in your easyconfig file (admittedly minus the libnuma configure CLI option).

Your attached openmpi build log file is truncated -- it stops partway through configure. Can you send all the information listed here?

[lyu]

Hi @moravveji, could you try compile OpenMPI with --without-tm? I believe I have experienced a similar issue here.

Basically I am trying to install OpenMPI 4.0.1 under my home directory on our cluster, with hwloc 2.0.2 installed to the same directory. Here is my full installation script:

export INSTALLPATH=$HOME/ompi

wget https://download.open-mpi.org/release/hwloc/v2.0/hwloc-2.0.2.tar.bz2
tar xf hwloc-2.0.2.tar.bz2
cd hwloc-2.0.2
./configure --prefix=$INSTALLPATH
make -j 8 && make install
cd ..

wget https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-4.0.1.tar.bz2
tar xf openmpi-4.0.1.tar.bz2
cd openmpi-4.0.1
./configure --prefix=$INSTALLPATH --disable-oshmem --disable-dlopen --disable-io-romio --without-slurm --without-x --without-ucx --with-hwloc=$INSTALLPATH --with-libevent=$HOME/.local --with-tm=/cm/shared/apps/torque/6.1.1
make -j 8 && make install
cd ..

The whole process finishes without errors. However, all the binaries compiled using this OpenMPI install are linked against both the correct hwloc install, and /cm/shared/apps/hwloc/1.11.8/lib/libhwloc.so.5, and they all crash immediately when I try to run them. Typical error message:

[login2:51884] *** Process received signal ***
[login2:51884] Signal: Segmentation fault (11)
[login2:51884] Signal code: Address not mapped (1)
[login2:51884] Failing at address: 0x28
[login2:51884] [ 0] /lib64/libpthread.so.0(+0xf6d0)[0x2aaaab60a6d0]
[login2:51884] [ 1] /my_home/ompi/lib/libopen-pal.so.40(opal_hwloc_base_get_nbobjs_by_type+0x69)[0x2aaaac0ac8e9]
[login2:51884] [ 2] /my_home/ompi/lib/libopen-pal.so.40(opal_hwloc_base_get_topo_signature+0x2b)[0x2aaaac0aee6b]
[login2:51884] [ 3] /my_home/ompi/lib/libopen-rte.so.40(+0x7fdf5)[0x2aaaaab4fdf5]
[login2:51884] [ 4] /my_home/ompi/lib/libopen-rte.so.40(orte_init+0x29a)[0x2aaaaaaf9c1a]
[login2:51884] [ 5] /my_home/ompi/lib/libopen-rte.so.40(orte_submit_init+0x8e3)[0x2aaaaabc0ed3]
[login2:51884] [ 6] mpirun[0x4012cf]
[login2:51884] [ 7] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2aaaad07a445]
[login2:51884] [ 8] mpirun[0x40116e]
[login2:51884] *** End of error message ***

I am 100% sure that I didn't load this hwloc module provided by our cluster, and all my environment variables are clean regarding this. The only explanation I can find is the following lines in our torque module (note the hwloc-related parts):

torque/6.1.1/lib/libdrmaa.la
20:dependency_libs=' /cm/shared/apps/torque/6.1.1/lib/libtorque.la -L/cm/shared/apps/hwloc/current/lib -lz -ldl -lpthread -lrt /cm/shared/apps/hwloc/1.11.8/lib/libhwloc.la -lnuma -ludev -lxml2'

torque/6.1.1/lib/libtorque.la
20:dependency_libs=' -lz -ldl -lpthread -lrt -L/cm/shared/apps/hwloc/current/lib /cm/shared/apps/hwloc/1.11.8/lib/libhwloc.la -lnuma -ludev -lxml2'

Compiling with --without-tm solved the issue for me(or removing the --disable-dlopen flag for my case).

If I use external pmix 3.1.2 and tell it to use external hwloc, it is able to pick the right hwloc install, but OpenMPI is still confused about which to link to.

[ggouaillardet]

@moravveji config.log was truncated.

note --with-verbs=/lib64 might have some unexpected side effects, so you might want to give it a try without this option.

[moravveji]

Thanks @ggouaillardet, @lyu and @jsquyres for your excellent input. So, I tried again with the following

• I removed the --with-verbs=/lib64 option
• I set --without-tm option
• I build hwloc from source, and pass the path to its installation folder using --with-hwloc=<path/to/hwloc>
• I do not get the error messages above; so, this is already a good news (?)
• However, I get now a different error: Some of the options are apparently not properly parsed to the compiler (?)
• I have attached a zip folder with the build script, and the stderr/stdout redicrected logs, and config.log and opal_config.h

  CC       libevent2022_module.lo
  CCLD     libmca_event_libevent2022.la
make[3]: Leaving directory `/easybuild/work/OpenMPI/from-source/openmpi-4.0.1/opal/mca
/event/libevent2022'
make[2]: Leaving directory `/easybuild/work/OpenMPI/from-source/openmpi-4.0.1/opal/mca
/event/libevent2022'
Making all in mca/hwloc/hwloc201
make[2]: Entering directory `/easybuild/work/OpenMPI/from-source/openmpi-4.0.1/opal/mc
a/hwloc/hwloc201'
Making all in hwloc
make[3]: Entering directory `/easybuild/work/OpenMPI/from-source/openmpi-4.0.1/opal/mca/hwloc/hwloc201/hwloc'
Making all in include
make[4]: Entering directory `/easybuild/work/OpenMPI/from-source/openmpi-4.0.1/opal/mca/hwloc/hwloc201/hwloc/include'
make[4]: Nothing to be done for `all'.
make[4]: Leaving directory `/easybuild/work/OpenMPI/from-source/openmpi-4.0.1/opal/mca/hwloc/hwloc201/hwloc/include'
Making all in hwloc
make[4]: Entering directory `/easybuild/work/OpenMPI/from-source/openmpi-4.0.1/opal/mca/hwloc/hwloc201/hwloc/hwloc'
  CC       topology.lo
gcc: error: unrecognized command line option ‘--without-tm’
gcc: error: unrecognized command line option ‘--without-slurm’
gcc: error: unrecognized command line option ‘--without-x’
gcc: error: unrecognized command line option ‘--with-hwloc=/hwloc/2.0.2-GCCcore-6.4.0’
gcc: error: unrecognized command line option ‘--with-cuda=/software/CUDA/10.1.105’
gcc: error: unrecognized command line option ‘--with-ucx=/software/UCX/1.4.0-GCCcore-6.4.0/lib/hwloc"’

Thanks a lot.
E.
ompi-output.zip

[yosefe]

@moravveji i think it may be because all options are passed to ./configure as single string and not as separate arguments:

+ ./configure '--prefix=/OpenMPI/4.0.1-GCC-6.4.0-CUDA-10.1.105 --enable-shared --enable-static --enable-mpi-thread-multiple --without-tm --without-slurm --without-x   --enable-mpirun-prefix-by-default --with-hwloc=/hwloc/2.0.2-GCCcore-6.4.0 --with-cuda=/software/CUDA/10.1.105 --with-ucx=/software/UCX/1.4.0-GCCcore-6.4.0'

because of this line in the openmpi.sh script:

 ./configure "${options}"

PS it's probably related to https://github.com/openucx/ucx/issues/3494

[jsquyres]

I think @yosefe is correct -- your use of quotes on that ./configure line may be the problem.

Note, however, that from the log you provided, you didn't end up using the installed hwloc because:

--- MCA component hwloc:external (m4 configuration macro, priority 90)
checking for MCA component hwloc:external compile mode... static
checking --with-hwloc-libdir value... simple ok (unspecified value)
checking looking for external hwloc in... ()
checking hwloc.h usability... yes
checking hwloc.h presence... yes
checking for hwloc.h... yes
looking for library without search path
checking for library containing hwloc_topology_init... -lhwloc
checking if libhwloc requires libnl v1 or v3...
checking if external hwloc version is 1.5 or greater... yes
checking if external hwloc version is 2.0 or greater... no
configure: WARNING: external hwloc version is less than internal version 2.0
configure: WARNING: using internal hwloc
checking if MCA component hwloc:external can compile... no

Specifically, configure compiled against your external hwloc and concluded that its version was less than v2.0. Hence, it just defaulted back to the internal hwloc.

[moravveji]

Big thanks to @yosefe and @jsquyres for the hints.
I stepped down from OpenMPI v 4x to v 3.1.4, and managed to compile it with UCX/1.5.1, CUDA/10.1.105 and hwloc/1.11.11. I have attached the log file. This time, I guess the external hwloc is correctly linked against.
I will keep testing this version with cuda-aware examples, and come back to you guys if I have a question.
log-v3-cuda.zip

[jsquyres]

If you have some time, I'd like to understand if there are actually issues that need to be resolved in the v4.0.x series.

For example:

1. Did removing the quotes from the configure line fix the compilation issue in Open MPI v4.0.x? (regardless of which hwloc it was using)
2. The logs you sent indicated that it was not actually finding hwloc 2.0.2 in /hwloc/2.0.2-GCCcore-6.4.0. Do you know for sure that hwloc 2.0.2 was installed in that tree (and no other hwloc)?

[hppritcha]

closing for now. user can reopen if there's more info on the problem.

[amirhossein-em]

I want to install openmpi v4.0.1 on a Debian system and use these commands for configuration and installation:
./configure --prefix=/usr/local
make all install

but at the middle of the installation I counter to this error:

make[2]: Entering directory '/home/openmpi-4.0.1/opal/tools/wrappers'
CCLD opal_wrapper
/usr/bin/ld: opal_wrapper.o: in function find_options_index': opal_wrapper.c:(.text+0x20e): undefined reference toregexecA'
/usr/bin/ld: opal_wrapper.c:(.text+0x21a): undefined reference to regfreeA' /usr/bin/ld: opal_wrapper.c:(.text+0x23f): undefined reference toregcompA'
/usr/bin/ld: opal_wrapper.c:(.text+0x261): undefined reference to `regfreeA'
collect2: error: ld returned 1 exit status
make[2]: * [Makefile:1872: opal_wrapper] Error 1
make[2]: Leaving directory '/home/openmpi-4.0.1/opal/tools/wrappers'
make[1]: [Makefile:2375: all-recursive] Error 1
make[1]: Leaving directory '/home/openmpi-4.0.1/opal'
make: ** [Makefile:1893: all-recursive] Error 1

Do you have any suggestion for this problem?

[jsquyres]

Please do not ask new questions on closed issues; please open a new issue.

Check out https://www.open-mpi.org/community/help/