Ompi: openmpi 1.10 has stderr output by default, breaking autopkg tests

[amckinstry]

Upstream bug in Debian:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=814451

Seen at least for lammps, esys-particle, liggghts:

adt-run [15:36:24]: test dipole: -----------------------]
adt-run [15:36:24]: test dipole:  - - - - - - - - - - results - - - - - - - - - -
dipole               FAIL stderr: [adt:03802] 1 more process has sent help 
message help-mpi-btl-base.txt / btl:no-nics
adt-run [15:36:25]: test dipole:  - - - - - - - - - - stderr - - - - - - - - - -
[adt:03802] 1 more process has sent help message help-mpi-btl-base.txt / btl:no-nics
[adt:03802] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / 
error messages
adt-run [15:36:25]: test ellipse: preparing testbed

EDIT: Added verbatim markdown

[rhc54]

Not sure I completely understand the issue here - if we encounter an error, then we will emit a user-friendly error message. In this case, it looks like the error is that we have no way of communicating, which means the package is indeed "broken".

[jsquyres]

@amckinstry Yes, to be clear, it looks like Open MPI is complaining that it cannot communicate between peer processes. This may be a function of how Open MPI was built, or it may be a function of the kind of environment in which the test is running.

[ggouaillardet]

@amckinstry from reading the source code, this message can only be issued by the openib btl.
i suspect there is an infiniband interface that cannot be used (for example, there is no opensm running)

as a workaround, you can disable the openib btl
you can
mpirun --mca btl ^openib ...
or you can
export OMPI_MCA_btl=^openib
and then invoke mpirun as usual

[ggouaillardet]

as i wrote in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=814451#25

this is a warning message, not an error message (and i guess debian considers as an error if something is written to stderr)

that can be reproduced by loading mlx4_core kernel module on a system with no infiniband hardware. that can also happen if the __linux__ macro is not defined.
from an openmpi point of view, we could also disqualify the openib btl if /sys/class/infiniband is empty (see ompi_common_verbs_check_basics())

@jsquyres @miked-mellanox @hjelmn @hppritcha @rhc54 any thoughts ?

[amckinstry]

Thanks, correct, "autopkgtest" is a Debian infrastructure for testing packages: it runs the test suites for packages (eg. aces, nwchem, etc. ), which then presume openmpi is present / configured. If it spots stderr output, it considers the test failed.

Current potential solutions:
(1) Set "export OMPI_MCA_btl=^openib", don't presume openib is present/configured. This is a functionality test, not performance.
(2) Check / ensure that if openib is available, opensm is set up (I don't think it is)
(3) Add a patch (Debian-specific) to downgrade warnings from stderr to stdout based on an environment variable.

Comments on (3)?

Debian provides openmpi on kFreeBSD and Hurd (!) as well as Linux kernels, but I haven't seen autopkgtest logs on these systems and don't know if its failing on them. Will investigate.

[ggouaillardet]

about 2) this is just a guess, and that would be the case only if infiniband hardware is installed on the node.
can you update the ticket on Debian bugs so I fully understand why the warning is issued ?
I will double check about 3), but I guess we can already do that without any patch (e.g. environment variable only)

[ggouaillardet]

btw, if my initial guess is correct (e.g. no infiniband hardware installed, but infiniband kernel modules are loaded), the easiest "fix" is to unload the infiniband drivers.

[jsquyres]

@amckinstry In looking through the symbol versioning issue/PR, I noticed you also have this old issue open. Is it still an issue, or should we close it?

[amckinstry]

Closing this as no longer an issue.