Ompi: MPI I/O Crashes and Corruption in HDF5

[ax3l]

Background information

What version of Open MPI are you using?

3.1.3

Update: affects 2.X, 3.X and 4.0.0

Describe how Open MPI was installed

From source via spack or as modules build from source on various different HPC systems.

Please describe the system on which you are running

• Operating system/version: Debian 9.6 and derivates such as Ubuntu
• Computer hardware: Laptops and HPC
• Network type: local and remote (Ethernet and IB)

Details of the problem

We are currently reporting two parallel HDF5 issues that either crash on write or corrupt data. The issues only occur with OpenMPI, not with MPICH 3.3 which we used for comparison.

We (@ax3l, @psychocoderHPC) are not the upstream authors of HDF5, but wanted to inform and connect you, so you are aware of these issues as they might be rooted in the OpenMPI I/O layer somewhere.

Link to HDF5 parallel I/O write issues with reproducers:

• [x] crash with some special-size struct types: https://forum.hdfgroup.org/t/cannot-write-more-than-512-mb-in-1d/5118
  -> #6286 #6287 (a specific OpenMPI I/O issue)
• [x] data corruption: https://forum.hdfgroup.org/t/hdf5bug-h5fd-mpio-collective-chunking/5279
  -> #6295 #6326 #6172 (a general OpenMPI Datatype issue)

[edgargabriel]

Looking into the issues.

[edgargabriel]

The first issue (cannot write more than 512MB) is an easy fix. First, the test passes with all collective I/O components except for 'individual', which is however the selected one for 1-process scenarios.

The culprit is a floating point division of two very close, large values.

no _of_cyles = ceil(536870916/536870912)

should be 2, but is 1 if you use floats (which is what the individual component used). If you use double precision floating point values, the result is as expected 2 (which is what all other collective I/O components use). A fix is coming soon.

If you want to run the code before the next release is available, simply exclude the individual fcoll component for this scenario, e.g.

mpirun --mca fcoll ^individual -np 1 ./....

[ax3l]

Thank you for so quickly looking into this!
So the first issue is located at:

• https://github.com/open-mpi/ompi/blob/v4.0.0/ompi/mca/common/ompio/common_ompio_file_write.c#L112
• https://github.com/open-mpi/ompi/blob/v4.0.0/ompi/mca/common/ompio/common_ompio_file_read.c#L134

Jippey, well spotted! Do you think we can get this in 4.0.1?

[edgargabriel]

I will file the pr's this afternoon (after my class), but the decision on whether it still makes it into the 4.0.1 release is out of my hand unfortunately.

[ax3l]

No problem, probably needs backporting to the other stable release lines as well.

Do you have an idea about the second issue?

[edgargabriel]

not yet, I will look at it later today.

[ax3l]

Thanks a lot, much appreciated!

[ax3l]

@edgargabriel just out of curiosity, did you have a chance to look into the data corruption issue as well? Do you already have an idea in which methods it might reside? Thanks already for your help!

[edgargabriel]

I started to look at it yesterday. I think I have narrowed it down to where the problem might be to two possible locations, but I wasn't able yet to precisely pinpoint the issue. It is clear however that it is a generic data type processing problem, either in the datatype engine or in ompio, since all ompio components produce exactly the same bug (the issue therefore not bound to just one component).

[ax3l]

Thanks! Do you like to share the locations in source that you look at? Maybe we can see something from review as well to assist you.

[ggouaillardet]

@ax3l can you please confirm this issue is specific to the io/ompio (e.g. "native" Open MPI MPI-IO) component ?

A workaround is to use the romio component (ported from MPICH) with

mpirun --mca io ^ompio ...

My tests show romio component is fine (at least in the master branch).

[ax3l]

Yes, both reported issues (the crash and the data corruption) are gone when switching the I/O backend to ROMIO. (Tested with OpenMPI 3.1.3).

Update: found and tested it already :)

I can also disable this via

export OMPI_MCA_io=^ompio

right?

Since when is ROMIO available as well (in terms of OpenMPI version ranges)?
I might want to use this as a temporary work-around for all systems we are running on until new releases are shipped (also, we still need to fix the second issue).

From the FAQ, it looks like OMPIO became the default in 2.X and ROMIO might be always available (not mentioned)?

[ax3l]

For the second issue that causes data corruption, I reduced the example to one rank for easier debugging:

#include <mpi.h>
#include <hdf5.h>

#include <stdlib.h>
#include <string.h>
#include <stdio.h>

#define X 1872llu
#define Y 1872llu

int write_HDF5(
    MPI_Comm const comm, MPI_Info const info,
    float* data, size_t len, int rank)
{
    // property list
    hid_t plist_id = H5Pcreate(H5P_FILE_ACCESS);

    // MPI-I/O driver
    H5Pset_fapl_mpio(plist_id, comm, info);

    // file create
    char file_name[100];
    sprintf(file_name, "%zu", len);
    strcat(file_name, ".h5");
    hid_t file_id = H5Fcreate(file_name, H5F_ACC_TRUNC,
                              H5P_DEFAULT, plist_id);

    // dataspace
    hsize_t dims[3] = {Y, X};
    hsize_t globalDims[3] = {Y, X};
    hsize_t max_dims[2] = {Y, X};
    hsize_t offset[2] = {0, 0};

    hid_t srcSize = H5Screate_simple(2, dims, NULL);
    hid_t filespace = H5Screate_simple(2,
        globalDims,
        max_dims);

    printf("%i: %llu,%llu %llu,%llu \n", rank,offset[0],offset[1],globalDims[0],globalDims[1]);

    // chunking
    hsize_t chunk[2] = {128, 128};
    hid_t datasetCreationProperty = H5Pcreate(H5P_DATASET_CREATE);
    H5Pset_chunk(datasetCreationProperty, 2, chunk);

    // dataset
    hid_t dset_id = H5Dcreate(file_id, "dataset1", H5T_NATIVE_FLOAT,
                              filespace, H5P_DEFAULT,
                              datasetCreationProperty, H5P_DEFAULT);

    // write
    hid_t dset_plist_id = H5Pcreate(H5P_DATASET_XFER);

#ifdef FIX
    H5Pset_dxpl_mpio(dset_plist_id, H5FD_MPIO_INDEPENDENT); // default
#else
    H5Pset_dxpl_mpio(dset_plist_id, H5FD_MPIO_COLLECTIVE);
#endif

    hid_t dd = H5Dget_space(dset_id);
    H5Sselect_hyperslab(dd, H5S_SELECT_SET, offset,
                        NULL, dims, NULL);

    herr_t status;
    status = H5Dwrite(dset_id, H5T_NATIVE_FLOAT,
                      srcSize, dd, dset_plist_id, data);

    // close all
    status = H5Pclose(plist_id);
    status = H5Pclose(dset_plist_id);
    status = H5Dclose(dset_id);
    status = H5Fclose(file_id);

    return 0;
}

int main(int argc, char* argv[])
{

    MPI_Comm comm = MPI_COMM_WORLD;
    MPI_Info info = MPI_INFO_NULL;

    MPI_Init(&argc, &argv);

    int rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    size_t lengths[1] = {X*Y};
    for( size_t i = 0; i < 1; ++i )
    {
        size_t len = lengths[i];
        printf("Writing for len=%zu ...\n", len);
        float* data = (float*)malloc(len * sizeof(float));
        for( size_t y = 0; y < Y; ++y)
            for( size_t x = 0; x < X; ++x)
                data[y * Y + x] = 100.f + y%1024;

        write_HDF5(comm, info, data, len, rank);
        free(data);
        printf("Finished write for len=%zu ...\n", len);
    }

    MPI_Finalize();

    return 0;
}

If either ompio is disabled or -DFIX is passed the issue does not occur:

mpicc -g hdf5-5279.c -lhdf5

# ISSUE:
./a.out
# check 3504384.h5

# not occuring:
mpirun --mca io ^ompio -n 1 ./a.out
# check 3504384.h5

# not occuring:
mpicc -DFIX -g hdf5-5279.c -lhdf5
./a.out
# check 3504384.h5

[ax3l]

I did some testing on the latest master branch of OpenMPI (d6cdbdfd399e63ec11d5d7d58db652073f53c211 which does not yet include #6172) together with HDF5 develop (7e792d7).

Both reported issues, crash and the data corruption issue, are gone now. Thanks to everyone involved fixing this! :+1: :rocket: :sparkles:

Since this affects all OpenMPI 2.X, 3.X and 4.0.0 releases, do you plan to prepare backports to these release lines? The current HDF5 announcement can be found here.

[edgargabriel]

I created pr's for the 4.0 and 3.1 branches, so the next releases of those series should have the fixes. 3.0 is borderline whether we should do that. The 2.1 and 2.0 series will not receive these updates however anymore.

[ax3l]

Ok, thanks. Just for completeness, we need to backport four (4) PRs so far, as listed in the PR description.

• crash with some special-size struct types -> #6286 #6287
• data corruption -> #6295 #6326 (#6172)

[jsquyres]

As a v2.x RM, I would not object to porting these fixes back to the v2.1.x branch. We will not make a new release, but the fixes can be there (if you care) and in the nightly snapshot tarballs.

If you want these in the next v3.0.x release, they need to be PR'ed ASAP.

[ax3l]

Thanks, sounds good!

• [ ] v2.0: affected, not yet fixed (EOL?)
• [ ] v2.x: affected, not yet fixed (EOL?)
• [x] I applied @edgargabriel's backport of the crash fixes (#6286 #6287) in v3.1.x (#6344) to v3.0.x (#6428)
• [x] crash fixes (#6286 #6287) did already go in v4.0.x (#6335)

@ggouaillardet @bosilca do you plan to backport the opal data type fixes (OMPI data corruption) from #6295 #6326 (#6172)? Maybe also to v3.0.x, v3.1.x and v4.0.x?

• [ ] v2.0: affected, not yet fixed (EOL?)
• [ ] v2.x: affected, not yet fixed (EOL?)
• [x] v3.0.x: #6438 (includes #6295 #6326)
• [x] v3.1.x: #6437 (includes #6295 #6326)
• [x] v4.0.x: #6347 (includes #6295 #6326)

[hppritcha]

@edgargabriel where's the PR for v4.0.x for this issue?

[edgargabriel]

I thought this was done already, but it looks like it wasn't. This is however a datatype issue, @bosilca can you maybe file a pr for this for the 4.0.x branch as well?

[gpaulsen]

@hppritcha, confirmed that (#6286 #6287) are already ported to v4.0.x branch via #6347.
We don't believe we need #6172 on v4.0.x as that's just a performance improvement.

[ax3l]

So #6347 backported both #6295 and #6326 to v4.0.x.
But shouldn't we port these two commits as well to v3.0.x and v3.1.x? Here is a little curated list from the things I saw.

[ggouaillardet]

I confirm #6347 is a backport of __both__ #6295 and #6326

I must admit I got lost in the various discussions. IIRC, there was two distinct issues. Are there both present in both v3.0.x and v3.1.x ?

[ggouaillardet]

I tried the test code above and the one at https://forum.hdfgroup.org/t/cannot-write-more-than-512-mb-in-1d/5118 with the latest v3.0.x and v3.1.x branches and could not find any issues.

[ax3l]

@ggouaillardet thanks for taking a look.

For the data corruption / opal issue, you have to test the second example, which is https://forum.hdfgroup.org/t/hdf5bug-h5fd-mpio-collective-chunking/5279 , on v3.0.x and v3.1.x.
Do you mean that code with "the one above"?

According to my tests, initially on 3.1.3, those do need a backport of #6295 + #6326 as well.

[ax3l]

Just found it: both are backported to these branches as well:
https://github.com/open-mpi/ompi/commits/v3.0.x/opal/datatype/opal_convertor_raw.c
https://github.com/open-mpi/ompi/commits/v3.1.x/opal/datatype/opal_convertor_raw.c

Found the backport PRs: #6438 and #6437

[ax3l]

the v3.0.x, v3.1.x and v4.0.x lines have now all backports for both the IO crash and the opal data corruption in place :+1:

v2.0.x and v2.x are also affected but seam to have reached end of life with regards to releases.

I will leave this issue open until the releases have landed, so we can document which versions fix it.

[hppritcha]

closing as fixed on all releases that we're still keeping alive.

[ax3l]

For future readers, the reported issues are fixed in:

• [ ] v2.0: affected, not fixed (EOL)
• [ ] v2.x: affected, not fixed (EOL)
• [x] [v3.0.4](https://raw.githubusercontent.com/open-mpi/ompi/v3.0.x/NEWS)
• [x] [v3.1.4](https://raw.githubusercontent.com/open-mpi/ompi/v3.1.x/NEWS)
• [x] [v4.0.1](https://raw.githubusercontent.com/open-mpi/ompi/v4.0.x/NEWS)