Openpmd-api: Streaming Read: Segfault on Close w/o Explicit Flush

[eschnett]

I am trying to read data from a file. I create a Series and the iteration via readIters = series->readIterations() and iter = *readIters.begin(). I schedule the reads via loadChunk, passing pointers to AMReX-allocated data structures.

I then tried to perform the reads via iter.close(). This lead to a segfault.

I can avoid the segfault by calling series->flush() before closing the iteration.

[ax3l]

Thanks for the report!

CCing @franzpoeschel: do you think the API contract for Iteration::close(flush) should support this? I can see how that could be expected...

Erik, so far (pre 0.13.0), we usually explicitly flush() to fill buffers from loadChunk. If we want the above logic to work, we probably need to flush inside the Iteration::close() immediately upon enter (when the parameter flush=True is set) and then we do the existing logic (which flushes again).

[franzpoeschel]

That workflow is supposed to be possible, if it fails that is a bug.
This test here uses Iteration::close() for loading data. I also tried to replace the test, making it more similar to your example:

    Series readSeries( file, Access::READ_ONLY );

    {
        std::vector< int > buffer( 1000, -1 );
        auto iter = *readSeries.readIterations().begin();
        iter.meshes[ "E" ][ "x" ].loadChunk(
            std::shared_ptr< int >{ buffer.data(), []( auto const * ) {} },
            Offset{ 0 },
            Extent{ 1000 } );
        std::cout << "BEFORE: [" << buffer[ 0 ] << ", ..., " << buffer[ 999 ]
                  << "]" << std::endl;
        iter.close();
        std::cout << "AFTER: [" << buffer[ 0 ] << ", ..., " << buffer[ 999 ]
                  << "]" << std::endl;
    }

And also that works as expected.

Can you share a failing code sample and the version of openPMD that you are using?

It should be noted that using Series::readIterations() is intended to support streaming workflows and puts some restrictions on read/write patterns. One of those is that you cannot use Series::readIterations() more than once (if the stream is over, it's not coming back). But if that were the error, I don't see how using Series::flush() would be fixing it, so I assume you're seeing something else.

[ax3l]

@eschnett one thing you could do is modify the linked test and open that as a demonstrator PR, so we can debug the logic that you use together.

[franzpoeschel]

I experienced a similar issue today and I think I have found a solution for that one. Will open a PR, maybe that could also fix your problem.

[ax3l]

Solution proposed in #1030

@eschnett feel free to close this issue in case that PR fixes it.