Googletest: How to test MPI applications?

[floli]

Hello,

we want to do unit und integration testing of our scientific application that uses MPI. MPI uses different processes within one communication domain (MPI_COMM_WORLD) to implement parallelism. A MPI application is usally started like mpirun -n 4 ./application starting 4 instances in one comunication domain. Each process is assigned a rank = {0, 1, 2, ..., n-1}

For testing it using googletest I see two challenges:

• Some tests only run sequentially (some other use MPI to communicate between instances of the same test case), therefore they should be skipped when the rank != 0. For that we need some means of runtime skipping, like the boost.test prondition decorator (http://www.boost.org/doc/libs/1_61_0/libs/test/doc/html/boost_test/tests_organization/enabling.html). There is issue #490 which is the same, as far as I can tell.
• Started with the command above each process is connected to the same stdout, so the mesages mingle. For that we need some kind of message aggregation technique, so we can have a unified output from just one process at the end.

I'm not fixed to these requirements, what is your approach for testing MPI applications?

Thanks!

[AntoineD]

Hi floli,

I'm currently writing MPI tests as well, but unfortunately I do not know how to help you with your concerns.
So far I created an gtest environment to init and finalize the MPI context:

#include "gtest/gtest.h"

#include "mpi.h"

using namespace ::testing;

// This class deals with the MPI context,
// it shall be done globally for the test program.
// see https://github.com/google/googletest/blob/master/googletest/docs/AdvancedGuide.md#global-set-up-and-tear-down
class TestEnvironment : public Environment {
protected:

  virtual void SetUp() {
    char** argv;
    int argc = 0;
    int mpiError = MPI_Init(&argc, &argv);
    ASSERT_FALSE(mpiError);
  }

  virtual void TearDown() {
    int mpiError = MPI_Finalize();
    ASSERT_FALSE(mpiError);
  }

  virtual ~TestEnvironment() {};
};

Environment* const foo_env = AddGlobalTestEnvironment(new TestEnvironment);

I'm also trying to figure out how to handle death test within MPI.

[norci]

hi @floli

For this question:
Started with the command above each process is connected to the same stdout, so the mesages mingle. For that we need some kind of message aggregation technique, so we can have a unified output from just one process at the end.

I think this can be solved by Extending Google Test by Handling Test Events
That we can write the logs into files, instead of "printf". Each process should has it's own log file, such as rank_0.log, rank_1.log .

[kuxn]

To get the output from just one process, my current solution is to add below right before RUN_ALL_TESTS() line in main(), such that there will be only one listener for rank 0.

::testing::TestEventListeners& listeners =
    ::testing::UnitTest::GetInstance()->listeners();
if (world.rank() != 0) {
    delete listeners.Release(listeners.default_result_printer());
}

[floli]

Simple redirection of outputs of different ranks to different files can also be done using mpirun --output-filename=outfile ...

[goxberry]

I wrote a simple header to provide this functionality: https://github.com/LLNL/gtest-mpi-listener

I think Trilinos also contains an MPI-ready unit testing framework for C++, but Trilinos is a very heavy dependency, whereas Google Test plus an additional header is not.

If there's interest, I'd be happy to contribute the header upstream with tests I wrote to check functionality.

[gennadiycivil]

Thank you very much for this report. The best way to approach this would be to create a proper PR and submit it for consideration.

[qzxuhui]

I am using gtest for MPI program unit test. My platform is ubuntu 18.04.
I find the output of executable file have color when using simple command

./a.out

But when using mpirun command there is no color

mpirun -np 1 ./a.out

It will be really helpful if output have color, because we can get the result of unit test at a glace.
It there any way to deal with this problem?

The simple source file shown bellow:

#include <mpi.h>
#include "gmock/gmock.h"

int main(int argc, char *argv[])
{
    int result = 0;

    ::testing::InitGoogleTest(&argc, argv);
    MPI_Init(&argc, &argv);
    result = RUN_ALL_TESTS();
    MPI_Finalize();

    return result;
}

[gennadiycivil]

I re-opened this issue to see if someone from community would chime in. As there is no activity, I am closing this again

[jthies]

testing MPI programs with standard gtest is quite dangerous. There is no guarantee that all failures are detected, and some processes may prematurely leave a test due to a local failure, so the program runs into a deadlock if an MPI collective is encountered afterwards. Also, the output will probably be jumbled, especially if you write to files.

We therefore developed an MPI-aware googletest variant with the classical oogletest "look-and-feel" but avoiding these difficulties. If there is interest from the community, we could make it available on github.

[jthies]

should also mention that it is not possible to make all of gtest MPI-safe: death tests for instance are not feasible because a dying MPI process generally kills the job. That's one of the reasons why we haven't created a PR for our adaptations.

[planetA]

@jthies For all MPI runtimes I know, there is a possibility to subvert the behaviour, when dying MPI process kills the job. For OpenMPI, for example, there is a flag "--continuous" for mpirun, that will let all the processes run, despite others dying.

[jthies]

sure, but maintaining dedicated code for all MPI implementations known is not really what we want to do, right?

[planetA]

@jthies True, but this is just a single parameter to pass, that even can be provided by the user.