Adios2: HeatTransfer SST test failing in CI

[JasonRuonanWang]

@eisenhauer

I have a PR (#2231 ) which only changes a single line in the doc but I keep getting this error and couldn't merge. Any idea?

1361 - HeatTransfer.SST.BP.Mx1 (Failed)

[eisenhauer]

Actually, if you look, most every PR is failing with this currently. @chuckatkins diagnosed it last night, and it's mostly a slow-CI problem, exacerbated by the "try-5-times" feature in CI. WRT the latter, I think it's going to take Chucks intervention as I don't think there's anything I could change in SST that would fix things. But you might also try retriggering CI sometime when the public CI system is not so busy in the hopes that those tests don't suffer the initial timeout that is apparently triggering the failure. (That's what I plan to do with my PRs anyway...)

[chuckatkins]

@eisenhauer Could the heat transfer binaries optionally use a barrier to synchronize themselves when starting such that the writer side will be guaranteed to start first and thus in turn write the correct connection information in the file? Something like (pseudo code here):

// writer
adios = ADIOS2(comm)
io = adios.DeclareIO(...)
engine = io.Open(...)
if(use_sync_barrier)
    // Wait for reader to arrive
    MPI_Barrier(comm)

// reader
adios = ADIOS2(comm)
io = adios.DeclareIO(...)
if(use_sync_barrier)
    // Wait for writer to arrive
    MPI_Barrier(comm)
engine = io.Open(...)

You wouldn't want to do that in general with an application but maybe fore specific case of using the binaries on a overloded CI system? That would allow the retry to work.

[eisenhauer]

Offhand, I think that's not unreasonable, but I don't know the heatTransfer code at all. I do know the reader and writer don't run on the same MPIcomm because of the split, but MPI_COMM_WORLD would work for the barriers. Certainly as a quick fix, this probably the way to go. But I also have a diagnostic run that I'd like to look at. (My lurking worry about the retry feature is that it can effectively cover up bugs (rare deadlocks, etc.). Maybe it's all just slow CI, but feel like I should look deeper too.)

[eisenhauer]

OK, I took a quick look at the output of my diagnostic run (SstVerbose turned on for the HeatTransfer SST tests), and at first glance, I think there might be an SST problem, likely due to recent changes in Preload (which is why it's showing up a lot in Mx1, which is a preload situation for SST). Not certain yet, but there are anomolies that might not be caused by slow CI... I'll report as soon as I know more.

[chuckatkins]

My lurking worry about the retry feature is that it can effectively cover up bugs (rare deadlocks, etc.).

This was always a calculated risk when employing the retry functionality. At the time we decided that enabling SST in CI and having it generally stable with retries at the expense of possibly hiding intermittent issues was more valuable than disabling SST from CI entirely and having it not tested at all.

[eisenhauer]

OK, I have a theory now. The HeatTransfer Mx1 situation is relatively unique in that it uses the Lock* functions to tell ADIOS that the geometry is fixed, but it's also a situation where we default to doing speculative preloads. It's probably the multiple preload mode transitions that are causing trouble. Working on a fix now that I have an idea about where the problem might lie.

[eisenhauer]

@chuckatkins PR #2235 may fix this issue, but I'm also seeing an unrelated failure on Travis centos8:

error: rpmdb: damaged header #173 retrieved -- skipping.

error: rpmdbNextIterator: skipping h# 173 blob size(4836): BAD, 8 + 16 * il(70) + dl(3708)

Extra Packages for Enterprise Linux Modular 8 - 316 kB/s | 117 kB 00:00

Extra Packages for Enterprise Linux 8 - x86_64 5.1 MB/s | 6.6 MB 00:01

Package curl-7.61.1-11.el8.x86_64 is already installed.

No match for argument: libzstd-devel

Error: Unable to find a match: libzstd-devel

The command '/bin/sh -c sed 's|enabled=0|enabled=1|' -i /etc/yum.repos.d/CentOS-PowerTools.repo && yum upgrade -y && yum install -y epel-release && yum install -y sudo curl vim git gdb gcc gcc-c++ gcc-gfortran make pkg-config openmpi-devel blosc-devel bzip2-devel zeromq-devel libzstd-devel fftw-devel hdf5-openmpi-devel python3-devel libfabric-devel python3-mpi4py-openmpi python3-numpy' returned a non-zero code: 1

The command "${TRAVIS_BUILD_DIR}/scripts/travis/run.sh" exited with 1.

[eisenhauer]

https://travis-ci.com/github/ornladios/ADIOS2/jobs/333268426

[eisenhauer]

My lurking worry about the retry feature is that it can effectively cover up bugs (rare deadlocks, etc.).

This was always a calculated risk when employing the retry functionality. At the time we decided that enabling SST in CI and having it generally stable with retries at the expense of possibly hiding intermittent issues was more valuable than disabling SST from CI entirely and having it not tested at all.

It would be nice to find some middle ground. A way to report a count of retries, or some other external indication that there might be a problem... I can obviously turn off retries for some specific PRs (and then hopefully remember to turn them back off before merging). Or maybe I just need to do a weekly PR that only turns off retries and delete it without merging...

[chuckatkins]

https://travis-ci.com/github/ornladios/ADIOS2/jobs/333268426

Working on it... #2237

[chuckatkins]

https://travis-ci.com/github/ornladios/ADIOS2/jobs/333268426

Working on it... #2237

It looks like there's an issue with the upstream centos8 base image right now. Just comment it out of the build matrix in your branch (rather than wait for a seprate pr to move through) and I'll turn it back on when upstream resolved the issue:

diff --git a/.travis.yml b/.travis.yml
index a20150e93..f1da2beca 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -7,7 +7,7 @@ env:
     - BUILD_MATRIX_ENTRY=docker-ubuntu1804
     - BUILD_MATRIX_ENTRY=docker-ubuntu1910
     - BUILD_MATRIX_ENTRY=docker-centos7
-    - BUILD_MATRIX_ENTRY=docker-centos8
+    # - BUILD_MATRIX_ENTRY=docker-centos8
 script:
   - git reset --hard ${TRAVIS_PULL_REQUEST_SHA}
   - ${TRAVIS_BUILD_DIR}/scripts/travis/run.sh

[eisenhauer]

Done and pushed.