Adios2: SST EndStep SegFault

[AaronV77]

Hey Everyone,

I ran into an issue/bug/lack of understanding for ADIOS2. In the first example that I have provided the system crashes on the end_step, but in the second example, it does not. The difference between the two examples is that the first one open and closes ADIOS2 completely within the for loop on the reading side. Overall, the examples are just supposed to send and receive two messages and print here-1 & here-2 (twice per example). I double-checked the parameters to make sure that I have everything correct, I do need the adios2_data_sync, and the reading mode is being used.

How I am running the two examples is just with one proc apiece. These examples are just consolidated problems that I have ran into and have remnants of how I am applying them to our application.

How I expect the examples to operate is to print here-1 and here-2 twice either way. I should not have to open and close ADISO2 for each message that I want to receive.

Here is a makefile!

all:
    gcc -c av_adios2.c
    mpicc reader.c -o r av_adios2.o -ladios2 
    mpicc writer.c -o w av_adios2.o -ladios2

clean:
    rm r w av_adios2.o

Example 1:
reader-1.c

#include <adios2_c.h>
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>

int main(int argc, char **argv)
{
    size_t start[2] = {0,0};
    size_t shape[2];
    size_t count[2];

    int rank, size;
    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    if (size == 1) {

        int * master_transfer_array = NULL;
        adios2_step_status status;
        adios2_adios * adios = adios2_init(MPI_COMM_SELF, adios2_debug_mode_on);
        adios2_io * io = adios2_declare_io(adios, "sst_reader");
        adios2_set_engine(io, "sst");
        adios2_engine * reader_engine = adios2_open(io, "sst_example", adios2_mode_read);

        for (int i = 0; i < 2; i++) {
            adios2_begin_step(reader_engine, adios2_step_mode_read, -1.f, &status);
            adios2_variable * adios2_var = adios2_inquire_variable(io, "my_array");
            if (adios2_var) {
                    adios2_variable_shape(shape, adios2_var);
                    master_transfer_array = calloc((shape[0] * shape[1]), sizeof(int));
                    adios2_set_selection(adios2_var, 2, start, shape);
                    adios2_get(reader_engine, adios2_var, master_transfer_array, adios2_mode_sync);
                    free(master_transfer_array);

            } else {
                    printf("\tERROR: Could not find the variable you were looking for.\n");
            }
            printf("Here-1..\n");
            adios2_end_step(reader_engine);
            printf("Here-2..\n");
        }

        adios2_close(reader_engine);
        adios2_finalize(adios);
    }

    MPI_Finalize();

writer-1.c

#include <adios2_c.h>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv)
{
    size_t start[2] = {0,0};
    size_t shape[2] = {1,8};
    size_t count[2] = {1,8};

    int rank, size;
    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    if (size == 1) {
        size_t nelements = count[0] * count[1];
        int * my_array = calloc(nelements, sizeof(int));

        for (int j = 0; j < nelements; ++j)
            my_array[j] = j;

        for (int j = 0; j < 2; ++j) {

            adios2_step_status status;
            adios2_adios * adios = adios2_init(MPI_COMM_SELF, adios2_debug_mode_on);
            adios2_io *io = adios2_declare_io(adios, "sst_writer");
            adios2_set_engine(io, "sst");
            adios2_engine * writer_engine = adios2_open(io, "sst_example", adios2_mode_write);

            adios2_variable *var_array = adios2_define_variable(io, "my_array", adios2_type_int32_t, 2, shape, start, count, adios2_constant_dims_true);
            adios2_begin_step(writer_engine, adios2_step_mode_append, 0.0, &status);
            adios2_put(writer_engine, var_array, my_array, adios2_mode_sync);
            adios2_end_step(writer_engine);

            adios2_close(writer_engine);
            adios2_finalize(adios);

        }
        free(my_array);
    }

    MPI_Finalize();
    return 0;
}

Example-1:
reader-2.c

#include <adios2_c.h>
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>

int main(int argc, char **argv)
{
    size_t start[2] = {0,0};
    size_t shape[2];
    size_t count[2];

    int rank, size;
    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    if (size == 1) {

        int * master_transfer_array = NULL;

        for (int i = 0; i < 2; i++) {

            adios2_step_status status;
            adios2_adios * adios = adios2_init(MPI_COMM_SELF, adios2_debug_mode_on);
            adios2_io * io = adios2_declare_io(adios, "sst_reader");
            adios2_set_engine(io, "sst");
            adios2_engine * reader_engine = adios2_open(io, "sst_example", adios2_mode_read);
            adios2_begin_step(reader_engine, adios2_step_mode_read, -1.f, &status);
            adios2_variable * adios2_var = adios2_inquire_variable(io, "my_array");
            if (adios2_var) {
                    adios2_variable_shape(shape, adios2_var);
                    master_transfer_array = calloc((shape[0] * shape[1]), sizeof(int));
                    adios2_set_selection(adios2_var, 2, start, shape);
                    adios2_get(reader_engine, adios2_var, master_transfer_array, adios2_mode_sync);
                    free(master_transfer_array);

            } else {
                    printf("\tERROR: Could not find the variable you were looking for.\n");
            }
            printf("Here-1..\n");
            adios2_end_step(reader_engine);
            adios2_close(reader_engine);
            adios2_finalize(adios);
            printf("Here-2..\n");
        }
    }

    MPI_Finalize();
    return 0;
}

writer-2.c

#include <adios2_c.h>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv)
{
    size_t start[2] = {0,0};
    size_t shape[2] = {1,8};
    size_t count[2] = {1,8};

    int rank, size;
    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    if (size == 1) {
        size_t nelements = count[0] * count[1];
        int * my_array = calloc(nelements, sizeof(int));

        for (int j = 0; j < nelements; ++j)
            my_array[j] = j;

        for (int j = 0; j < 2; ++j) {

            adios2_step_status status;
            adios2_adios * adios = adios2_init(MPI_COMM_SELF, adios2_debug_mode_on);
            adios2_io *io = adios2_declare_io(adios, "sst_writer");
            adios2_set_engine(io, "sst");
            adios2_engine * writer_engine = adios2_open(io, "sst_example", adios2_mode_write);

            adios2_variable *var_array = adios2_define_variable(io, "my_array", adios2_type_int32_t, 2, shape, start, count, adios2_constant_dims_true);
            adios2_begin_step(writer_engine, adios2_step_mode_append, 0.0, &status);
            adios2_put(writer_engine, var_array, my_array, adios2_mode_sync);
            adios2_end_step(writer_engine);

            adios2_close(writer_engine);
            adios2_finalize(adios);

        }
        free(my_array);
    }

    MPI_Finalize();
    return 0;
}

[jychoi-hpc]

Basic idea is to call adios open/close only once and call begin_step/end_step per every iteration (or step). Also, call adios2_define_variable before adios2_open.

Here is my modification:
writer.c:

#include <adios2_c.h>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv)
{
    size_t start[2] = {0,0};
    size_t shape[2] = {1,8};
    size_t count[2] = {1,8};

    int rank, size;
    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    if (size == 1) {
        size_t nelements = count[0] * count[1];
        int * my_array = calloc(nelements, sizeof(int));

        for (int j = 0; j < nelements; ++j)
            my_array[j] = j;

        adios2_step_status status;
        adios2_adios * adios = adios2_init(MPI_COMM_SELF, adios2_debug_mode_on);
        adios2_io *io = adios2_declare_io(adios, "sst_writer");
        adios2_set_engine(io, "SST");
        adios2_variable *var_array = adios2_define_variable(io, "my_array", adios2_type_int32_t, 2, shape, start, count, adios2_constant_dims_true);

        adios2_engine * writer_engine = adios2_open(io, "sst_example.bp", adios2_mode_write);

        for (int j = 0; j < 2; ++j) {
            adios2_begin_step(writer_engine, adios2_step_mode_append, 0.0, &status);
            adios2_put(writer_engine, var_array, my_array, adios2_mode_sync);
            adios2_end_step(writer_engine);
        }

        adios2_close(writer_engine);
        adios2_finalize(adios);
        free(my_array);
    }

    MPI_Finalize();
    return 0;
}

reader.c

#include <adios2_c.h>
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>

int main(int argc, char **argv)
{
    size_t start[2] = {0,0};
    size_t shape[2];
    size_t count[2];

    int rank, size;
    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    if (size == 1) {

        int * master_transfer_array = NULL;
        adios2_step_status status;
        adios2_adios * adios = adios2_init(MPI_COMM_SELF, adios2_debug_mode_on);
        adios2_io * io = adios2_declare_io(adios, "sst_reader");
        adios2_set_engine(io, "SST");
        adios2_engine * reader_engine = adios2_open(io, "sst_example.bp", adios2_mode_read);

        for (int i = 0; i < 2; i++) {
            adios2_begin_step(reader_engine, adios2_step_mode_read, -1.f, &status);
            adios2_variable * adios2_var = adios2_inquire_variable(io, "my_array");
            if (adios2_var) {
                    adios2_variable_shape(shape, adios2_var);
                    master_transfer_array = calloc((shape[0] * shape[1]), sizeof(int));
                    adios2_set_selection(adios2_var, 2, start, shape);
                    adios2_get(reader_engine, adios2_var, master_transfer_array, adios2_mode_sync);
                    free(master_transfer_array);

            } else {
                    printf("\tERROR: Could not find the variable you were looking for.\n");
            }
            printf("Here-1..\n");
            adios2_end_step(reader_engine);
            printf("Here-2..\n");
        }

        adios2_close(reader_engine);
        adios2_finalize(adios);
    }

    MPI_Finalize();
}

[AaronV77]

@jychoi-hpc You are correct in that thinking and the code that you have pasted is similar. So you have to keep both sides open simultaneously or just one message at a time. You can't reopen and close either side of the communication channel without getting segfaults. Gotcha, that should probably be in the documentation somewhere, because I've been using the SST engine as a means to transport a single message.

[eisenhauer]

Hey Aaron! Yeah, I'm not sure it occurred to us (or at least me) to use SST in quite that way. In theory something like this is possible, with some tweaks. For example, the open/close is something more like creating a named communication channel and there's really nothing that persists once it's closed. So, things probably go as expected for the first iteration of your writer/reader for loops, but for the second there's a race condition. Maybe the writer is a little faster and it manages to delete and then rewrite the contact information in "sst_example.bp.SST" before the 2nd iteration of the reader reads it. If so, things probably work as you might expect. But if the reader is a little faster than the writer and it manages to find the contact information associated with a previous iteration just as the writer is in the middle of tearing down that connection, things likely go awry. What should happen in that circumstance (reading outdated contact info) is a nice exception on the reader side saying that there doesn't seem to be an active writer. That does happen when the writer is fully closed, but I'm guessing that here we're not fully closed and somehow a reader join request isn't properly accepted or rejected...

WRT takeaways: I think what you've done would actually have worked if you had a different stream name for each iteration of the for loop, (but given the setup/teardown costs for the streams, it wouldn't be the most efficient thing!). Also, I should snag this code and see if I can find/repair that race condition on reader join/close. I doubt that I can turn it into a proper regression test because of the intrinsic race between the reader and writer, but I can maybe make it more likely to die in some way other than a seg fault...

[eisenhauer]

Ah, I was wrong. With reader-1.c, the problem was that the 2nd iteration BeginStep failed on EndOfStream, as expected because there was only one timestep (message) there. The subsequent inquire_variable didn't return anything, but then end_step got called, and SST's end_step wasn't properly protected against being called without having had a successful begin_step happen first. It segfaulted using a deleted BP3Deserializer instance (that would have been re-initialized in begin_step, had it been successful). I have a pending PR that adds protection against end_step without successful begin_step. Running reader_1 doesn't get a segfault with this fix.

[AaronV77]

@eisenhauer Thank you for that summation of what is happening in the backend. I think I am using the SST in ways that it was not intended. One way that you might find to be terrible is that I use one instance of the SST engine as a writer and programmatically create several adios2 variables for writing with different names. If what @jychoi-hpc has said to be true, that you should declare your variables before the engine call then I'm really doing something wacky. I have to do it this way because of data being transmitted differently each time and so that I don't have to open and close ADIOS2 every time. Also please don't go changing anything that would ruin this workflow haha its taken months to get this far!

[eisenhauer]

What you are doing with variables is OK. SST is pretty forgiving. You can declare your variables dynamically, before or after the engine creation, even in the middle of a long stream. You can write one set of variables in one timestep and a different combination of variables in the next timestep. We'll only move what you write...

We'll never intentionally break any working system! But giving better error messages than a segfault is pretty desirable...