Ompi: MPI_BOTTOM FORTRAN test fails when using xlf compiler

[AboorvaDevarajan]

Background information

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

ompi master

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

git clone and built using gfortran (default)

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

-bash-4.2$ git submodule status
 c4fe45f982b6cdf818f9d42d2bdf3a92019f92e4 3rd-party/openpmix (v1.1.3-2613-gc4fe45f)
 8cc88dd7f7627c2238b9b7bac2e68fe5a6092f7e 3rd-party/prrte (dev-30749-g8cc88dd)

Please describe the system on which you are running

• Operating system/version: RH8.2
• Computer hardware: ppc64le
• Network type: IB

---

Details of the problem

Here is a simpler reproducer:
https://gist.github.com/AboorvaDevarajan/f7ef6322a4ede4b74cb136251925ee42

Steps to recreate the issue:

export MPI_ROOT=/abd/os/ompi-install/
export UCX_ROOT=/abd/os/ucx-install/
export PATH=$MPI_ROOT/bin:$UCX_ROOT/bin:$PATH
export OMPI_FC=xlf
mpif90 -g bottom.F -o bottom
mpirun -np 2 ./bottom   #segfaults 

Here is the backtrace:

gdb ./bottom core.127622 
(gdb) bt
#0  0x0000200000f7b794 in __memcpy_power7 () from /lib64/libc.so.6
#1  0x0000200001119594 in opal_cuda_memcpy () from /nfs_smpi_ci/abd/os/ompi1-install/lib/libopen-pal.so.0
#2  0x0000200001109d60 in opal_convertor_pack () from /nfs_smpi_ci/abd/os/ompi1-install/lib/libopen-pal.so.0
#3  0x0000200002c54b00 in mca_btl_smcuda_sendi () from /nfs_smpi_ci/abd/os/ompi1-install/lib/openmpi/mca_btl_smcuda.so
#4  0x00002000033bed44 in mca_pml_ob1_send_inline.isra.4 () from /nfs_smpi_ci/abd/os/ompi1-install/lib/openmpi/mca_pml_ob1.so
#5  0x00002000033c0a14 in mca_pml_ob1_send () from /nfs_smpi_ci/abd/os/ompi1-install/lib/openmpi/mca_pml_ob1.so
#6  0x0000200000205a30 in PMPI_Send () from /nfs_smpi_ci/abd/os/ompi1-install//lib/libmpi.so.0
#7  0x00002000000f40fc in pmpi_send__ () from /nfs_smpi_ci/abd/os/ompi1-install//lib/libmpi_mpifh.so.0
#8  0x0000000010000fe0 in bottom02 () at bottom.F:29

This passes if I compile the application with gfortran, but If I compile the application with a different compiler, (in this case xlf)
for the constants like (MPI_FORTRAN_BOTTOM) the weak symbols generated doesn't match the mangling. Hence the test fails since it gets a undefined addr here,
https://github.com/open-mpi/ompi/blob/master/ompi/mpi/fortran/base/constants.h#L94

$ nm bottom | grep bottom
0000000010000e20 T bottom02
00000000100201d0 B mpi_fortran_bottom
$ nm $MPI_ROOT/lib/libmpi.so | grep fortran_bottom
00000000001a28c4 B mpi_fortran_bottom_

Looks like the mangling symbols (_, __, caps, ..) are generated for functions calls but not for the constants.

Is there a proper way to handle this? Thanks.

[jsquyres]

Good catch. @gpaulsen @jjhursey -- since this is POWER/xlf, I think that this is in your realm...

[markalle]

Internally in SMPI we have a couple commits in ompi/mpi/fortran/base/gen-mpi-mangling.pl and I'm not positive why we didn't upstream them. That's the file that already generates mpif-c-constants-decl.h and with our changes it constructs the lines there as

extern int mpi_fortran_bottom;
extern int MPI_FORTRAN_BOTTOM;
extern int mpi_fortran_bottom_;
extern int mpi_fortran_bottom__;
#define OMPI_IS_FORTRAN_BOTTOM(addr) \
        (addr == (void*) &mpi_fortran_bottom || \
         addr == (void*) &MPI_FORTRAN_BOTTOM || \
         addr == (void*) &mpi_fortran_bottom_ || \
         addr == (void*) &mpi_fortran_bottom__)

My understanding is the OMPI philosophy is more that you can build OMPI with gfortran, and if that's not compatible with xlf, you can build another copy of OMPI with xlf. OMPI doesn't seem to try to be a single-build solution that works across a wide range of environments. But I certainly wouldn't object to us upstreaming this enhancement

[ggouaillardet]

I am afraid I have to say I have to take some blame here

This was likely changed in open-mpi/ompi@9c77c6b66d48871d1ac0da843ff8c4e9dd39c75b and that was wrong for this kind of scenario.

IIRC, there is a GCC __attribute__ so different variables point to the same object
(kind of weak symbol for data instead of subroutines) and that could be used here
(the original idea was to replace 4 tests with one).
Of course, this trick might not work with other compilers.

That being said, if I understand correctly your scenario, you built Open MPI with gfortran, then build your app (that include 'mpif-h) with that Open MPI, xlf compiler and it failed at runtime.
Is this what you did?
I do understand the original intent was to support that use case.
Right now, I am not sure if the right action is to fix that, or simply not fix it (at least for compilers that do not support the __attribute__ trick) and "force" users to move to use mpi or use mpi_f08.

[AboorvaDevarajan]

@ggouaillardet yes, I built Open MPI with gfortran and then built the app (which includes mpif-h) using xlf which failed during runtime.

[jsquyres]

@ggouaillardet yes, I built Open MPI with gfortran and then built the app (which includes mpif-h) using xlf which failed during runtime.

FWIW, as noted above, this is currently expected to fail.

@ggouaillardet Keep in mind that the symbol matching is only one of the problems of supporting multiple Fortran compilers with a single build. There's also the value of .true. and .false., and the fact that file formats for Fortran modules are not standardized between compilers.

I'm not sure that trying to make a single Open MPI install support multiple Fortran compilers -- in this way, at least -- is useful. I thought that Spectrum had some mojo to change linker paths and/or Fortran compiler include paths to point to different trees (depending on which Fortran compiler was selected)...?