E3sm: Reproducibility on Chrysalis with OpenMPI-HPCX
Mellanox-optimized OpenMPI installation (openmpi/4.0.4-hpcx-cy5n3ft) is not producing BFB results: either run-to-run or in restart runs. To reproduce:
./cime/scripts/create_test ERS_Ln9.ne4_ne4.FC5AV1C-L --mpilib openmpi
Similar environment with Intel-MPI (intel-mpi/2019.9.304-tkzvizk) is BFB.
amametjanov
All 57 comments
any speculation as to how MPI could introduce such an issue? @worleyph do you think it could be related to our repro_sum?
mt5555
on 6 Jan 2021
I’ll take a look, but reprosum is primarily used for diagnostic output. Can’t remember - does it ever feed back into the simulation?
Sent from my iPhone
On Jan 5, 2021, at 20:08, Mark Taylor notifications@github.com wrote:

any speculation as to how MPI could introduce such an issue? @worleyph do you think it could be related to our repro_sum?—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
worleyph
on 6 Jan 2021
Just compared 2x1, 64x1, 96x1 PE layouts for
--res ne4_ne4 --compset FC5AV1C-L --machine chrysalis --compiler intel --project e3sm --mpilib openmpi
and these were deterministic between identical runs and BFB between the different layouts.
I'll look at ERS_Ln9.ne4_ne4.FC5AV1C-L more closely, to see what I need to do to reproduce the issue.
worleyph
on 6 Jan 2021
So, the ERS_Ln9.ne4_ne4.FC5AV1C-L test is nondeterministic, but only for the 5 step restart part of the test (starting within the first computed timestep, step 7). The initial 10 step run is deterministic (in one experiment of two invocations on the test). So this is all about the restart.
worleyph
on 6 Jan 2021
And, the test run with a 64x1 PE layout (instead of the default 96x1) passes. 64x1 fits in one node, so this is an internode issue (96x1 involves 2 nodes).
worleyph
on 6 Jan 2021
And ERS_Ln9.ne4_ne4.FC5AV1C-L with 64x1 fails when the 64 processes are spread over 2 nodes, so this is not a 32/64 imblanced node issue, just an issue of using more than one node.
worleyph
on 6 Jan 2021
Tried a 2 process (2x1) run of ERS_Ln9.ne4_ne4.FC5AV1C-L , with one process per node. This completed the initial 10 step run with no problems (and was BFB with the intial runs with all of the other ERS_Ln9.ne4_ne4.FC5AV1C-L tests). The restart run died with
1: WARNING: water balance error nstep= 6 local indexc= 3263
1: errh2o= -35.4333050140203
1: elm model is stopping - error is greater than 1e-4 (mm)
...
1: ERROR in BalanceCheckMod.F90 at line 425
...
1: ERROR: Unknown error submitted to shr_abort_abort.
1: Image PC Routine Line Source
1: libpnetcdf.so.3.0 0000155550C3C0AA tracebackqq_ Unknown Unknown
1: e3sm.exe 0000000002558650 shr_abort_mod_mp_ 114 shr_abort_mod.F90
1: e3sm.exe 00000000018B6BB5 abortutils_mp_end 77 abortutils.F90
1: e3sm.exe 0000000001C58474 balancecheckmod_m 425 BalanceCheckMod.F90
1: e3sm.exe 00000000018C5387 elm_driver_mp_elm 1248 elm_driver.F90
1: e3sm.exe 00000000018ABA7C lnd_comp_mct_mp_l 501 lnd_comp_mct.F90
1: e3sm.exe 00000000004438C4 component_mod_mp_ 737 component_mod.F90
1: e3sm.exe 00000000004250B5 cime_comp_mod_mp_ 2700 cime_comp_mod.F90
1: e3sm.exe 0000000000443500 MAIN__ 153 cime_driver.F90
soon after initialization completed. So, this 2 process restart job might be the easiest to debug? I'll start poking at it, but restarts are not something I am particularly familiar with. Concerned that the MPI bug will be in SCORPIO as well, which will further decrease my effectiveness :-).
worleyph
on 6 Jan 2021
I can't reproduce this problem with
--res ne4_ne4 --compset FC5AV1C-L --machine chrysalis --compiler intel --project e3sm --mpilib openmpi
2x1 PE layout, 1 process per node, initial 1 day run followed by a 1 day continuation run. The 1 day continuation run is deterministic in this experiment.
worleyph
on 6 Jan 2021
and the 1 day continuation run is BFB with the second day of a 2 day initial run.
worleyph
on 6 Jan 2021
CORRECTION: I mistakenly ran the 2 process job on one node instead of two. After trying again, using two nodes, I replicated the failure (in the same location!):
1: WARNING: water balance error nstep= 13 local indexc= 3263
1: errh2o= -35.0171556992675
1: elm model is stopping - error is greater than 1e-4 (mm)
Again, this was with
--res ne4_ne4 --compset FC5AV1C-L --machine chrysalis --compiler intel --project e3sm --mpilib openmpi
2x1 PE layout, 1 process per node, initial 1 day run followed by a 1 day continuation run. The continuation run dies with a runtime error.
worleyph
on 6 Jan 2021
I'll be reporting ongoing progress, hoping that someone who knows these parts of the code better than I will recognize what is going on before I do.
Still working with 2 process job that dies in the continuation run when run on two nodes but works fine when running on one node. The initial run is 1 day (as is the continuation run). Diff of atm.log for the successful and failed continuation runs, the first difference of import is:
initial state:
nstep= 156 time= 1.08333333333333 [day]
...
< qv( 2)= 0.0000000000000000E+00 0.4177449843997003E-03 0.2450758467193930E+01
---
> qv( 2)= 0.0000000000000000E+00 0.4177449843997003E-03 0.2407036438884810E+01
...
< qv( 22)= 0.9999974918391449E-36 0.3007158994441720E-08 0.7566430794811075E-05
---
> qv( 22)= 0.0000000000000000E+00 0.2972512731797596E-08 0.5450160178413672E-05
...
INITIALIZE_RADBUFFER: ntoplw = 1 pressure: 12.3825413055617
Creating new decomp: 16!28!!866!!d6!i2!
Creating new decomp: 16!72!28!!866!72!!d6!i2!
Creating new decomp: 16!28!!866!!d4!i2!
Creating new decomp: 16!73!28!!866!73!!d6!i2!
Creating new decomp: 16!288!28!!866!288!!d6!i2!
(GETFIL): attempting to find local file
where < is for the successful run and > is for the failed run. So the only differences in the initial EAM state by these statistics is in
2 CLDLIQ Grid box averaged cloud liquid amount wet
22 mom_a1 mom_a1 dry
I next printed out elem(ie)%state%Q(:,:,:,q) for q==2 and q==22 (modified subroutine prim_printstate in homme/src/theta-l/share/prim_state_mod.F90):
if (hybrid%masterthread) then
if (q==2) then
do ie=nets,nete
write(iulog,*) "Q=2:", ie
write(iulog,*) elem(ie)%state%Q(:,:,:,q)
call flush(iulog)
enddo
endif
if (q==22) then
do ie=nets,nete
write(iulog,*) "Q=22:", ie
write(iulog,*) elem(ie)%state%Q(:,:,:,q)
call flush(iulog)
enddo
endif
endif
The last 33 values of elem(ie)%state%Q(:,:,:,2) for each ie (=1,...,48) contain differences. Three values are printed on each line, so only one needs to be different for 'diff' to report a difference. The first 1122 values are identical (for each ie), I think. I did not go through all of the diffs carefully. For example, for ie=1, the diffs for the last 11 lines are
< 5.318690352714700E-014 4.730277714133582E-009 5.630219305417321E-010
< 2.383538782089622E-011 6.844756453121597E-013 8.476976636239778E-015
< 1.311045798507218E-011 0.000000000000000E+000 2.095335747429567E-013
< 9.497593157504737E-009 3.591172624004889E-016 2.110118609829044E-017
< 3.634207076647172E-010 6.593391102543217E-007 0.000000000000000E+000
< 6.933297347319903E-010 1.607105394129783E-011 5.648857448497226E-007
< 2.470273126627313E-010 1.157404920075237E-014 3.877021305172068E-009
< 0.000000000000000E+000 2.876562594937834E-009 0.000000000000000E+000
< 1.005784012656561E-011 9.632663975191111E-007 0.000000000000000E+000
< 5.716434513794619E-010 1.058656604092517E-013 5.042640180764963E-005
< 0.000000000000000E+000 7.776005462405105E-008 5.147584871650012E-010
---
> 5.318690352714700E-014 7.480048624040216E-019 2.647674752138004E-021
> 8.264392855276045E-022 5.545178529182347E-019 2.514095294422432E-019
> 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
> 8.798880063816426E-019 0.000000000000000E+000 0.000000000000000E+000
> 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
> 3.995713749704286E-018 0.000000000000000E+000 0.000000000000000E+000
> 9.479046743301757E-018 0.000000000000000E+000 0.000000000000000E+000
> 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
> 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
> 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
> 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
(Again, < is for the successful run and > is for the failed run.)
In contrast, for q=22 only the first 51 values for each ie are the same: all of the rest of the 1104 values are different (correct order of magnitude - just different), or, at least, 1/3 of these are (three per line, so diff triggers if even one is different), but, based on a quick scan, all of the values differ. Again, I did not go through all of the diffs carefully. This summary is based on the first few and last ie values.
What is the relationship between
2 CLDLIQ Grid box averaged cloud liquid amount wet
22 mom_a1 mom_a1 dry
if any? Does corruption in one make sense that there would be corruption in the other?
worleyph
on 7 Jan 2021
if any? Does corruption in one make sense that there would be corruption in the other?
Very speculative, but I _think_ data corruption in an aerosol tracer would affect CLDLIQ through a cloud condensation nuclei (CCN) process in the microphysics. I'm not sure whether corruption in CLDLIQ would immediately affect mom_a1, but I suppose it would. I think I'm more surprised that a diff in CLDLIQ is not immediately associated with diffs in all the other microphysics tracers, the first ~10 tracers in the q list.
One thing you might do is try to rule out large portions of code, as in other investigations of this sort. First easy thing: switch from theta-l to preqx. If that solves the problem, then the issue is in the dycore; if it doesn't, then we've ruled out the dycore.
ambrad
on 7 Jan 2021
@ambrad , please remind me how to switch back to preqx (great idea, thanks).
worleyph
on 8 Jan 2021
In this case, ./xmlchange CAM_TARGET=preqx
ambrad
on 8 Jan 2021
Thanks.
worleyph
on 8 Jan 2021
@ambrad , the case dies in an identical way when using preqx instead of theta-l .
worleyph
on 8 Jan 2021
Great. That rules out a lot. Two more ideas:
- Try a DEBUG=true run.
- You could also consider getting the scream repo and running the test SMS_D.ne4pg2_ne4pg2.F2010-SCREAM-HR to get a very different set of physics and no aerosols. If it reproduces the error you have isolated, that implies no single physics module is the issue; if it succeeds, that implies infrastructure things like the repro-sum code and the Allreduce it relies on are not the issue but rather something very specific in the physics packages.
ambrad
on 8 Jan 2021
I think it passes with BFBFLAG=FALSE
ndkeen
on 8 Jan 2021
Not in my reproducer. (Just tried it.)
worleyph
on 8 Jan 2021
Ah so that must be with BFBFLAG=FALSE _and_ some other compiler flags I'm testing.
Never mind.
It must be that I'm doing something wrong regarding the testing apparatus for ERS.
I can repeat my "pass" when I simply do a clean rebuild in the directory.
create_test ERS_D_Ln9.ne4_ne4.FC5AV1C-L --mpilib=openmpi
test fails compare
cd to case
case.setup --c;case.setup; case.build --clean-all; case.build; case.submit
test passes
And if I change the XML file (cime/src/drivers/mct/cime_config/config_component_e3sm.xml) to force BFBFLAG=FALSE and try the test again from scratch, it still fails.
ndkeen
on 8 Jan 2021
Do you mean "BFBFLAG=FALSE and some other compiler flags I'm testing." ? (yes! i corrected above comment)
worleyph
on 8 Jan 2021
Forgot to check the obvious - comparing the restart files from the initial 1 day run. They do differ:
SUMMARY of cprnc:
A total number of 268 fields were compared
of which 2 had non-zero differences
and 0 had differences in fill patterns
A total number of 16 fields could not be analyzed
A total number of 0 fields on file 1 were not found on file2.
diff_test: the two files seem to be DIFFERENT
The two fields with nonzero differences were
avg abs field values: 2.447068509160067E-03 rms diff: 2.6E-03 avg rel diff(npos): 1.9E-02
2.402015078077394E-03 avg decimal digits(ndif): 0.0 worst: 0.0
RMS dpCLDLIQ 2.6342E-03 NORMALIZED 1.0865E+00
and
avg abs field values: 8.708784524526662E-09 rms diff: 3.2E-08 avg rel diff(npos): 7.5E-01
6.197779134317006E-09 avg decimal digits(ndif): 0.2 worst: 0.0
RMS dpmom_a1 3.2245E-08 NORMALIZED 4.3262E+00
Other restart files seem fine (including elm). So, moving investigation to the end of the initial run.
worleyph
on 8 Jan 2021
The dpCLDLIQ values going into
call PIO_Write_Darray(File,Qdesc_dp(q),iodesc3d,var3d,ierr)
in write_restart_dynamics in restart_dynamics.F90 are correct, but what is being written to the restart file is not (when the two processes are on different nodes). Seems strange that the issue shows up for only these two fields (dpCLDLIQ and dpmom_a1), but the problem seems to occur within SCORPIO.
Changing env_build.xml to
<entry id="PIO_VERSION" value="1">
eliminates the failure (results identical to the PIO2 build but with both processes on the same node). So, something about MPI commands used by SCORPIO (that does not occur with SCORPIO classic)?
@jayeshkrishna , any SCORPIO test suite that we can use on Chrysalis with OpenMPI?
worleyph
on 8 Jan 2021
(results identical to the PIO2 build but with both processes on the same node)
results using the PIO1 build with each process on a different node are identical to the results using the PIO2 build with both processes on the same node. This is for the continuation run (where the PIO2 build has a runtime failure when running on two nodes, due to bad restart values).
worleyph
on 8 Jan 2021
I did try
ERS_D_Ln9.ne4_ne4.FC5AV1C-L --mpilib=openmpi
ERS_D_Ln9.ne4_ne4.FC5AV1C-L --mpilib=openmpi --compiler=gnu
ERS_Ln9.ne4_ne4.FC5AV1C-L --mpilib=openmpi --compiler=gnu
which all failed compare. Note that the DEBUG=TRUE tests will create 96 files of the form pio_log_0.txt. They are all empty.
ndkeen
on 8 Jan 2021
Since the MPI issue is causing erroneous writes inside of SCORPIO (and not SCORPIO classic), I'm punting, and will leave it to the SCORPIO experts. Hopefully the simple 2 process reproducer, and information on which fields are corrupted, will aid in the effort.
worleyph
on 8 Jan 2021
Note that I did try netcdf (instead of pnetcdf) and also 64bit_data (instead of 64bit_offset), and the failure was unchanged.
worleyph
on 8 Jan 2021
@jayeshkrishna Just making sure this is on your radar. Please prioritize as it would be good to have the vendor MPI working on Chrysalis.
sarats
on 8 Jan 2021
Thanks @worleyph for narrowing it down to this extent.
sarats
on 8 Jan 2021
The issue could be due to Mellanox optimizations. Manually built plain/vanilla openmpi-2.1.5 is BFB in e3sm_integration tests: branch azamat/chrysalis/add-plain-ompi-2_1_5. Trying plain openmpi-4.0.4 to confirm. If confirmed, will need a minimal reproducer to pass upstream.
amametjanov
on 9 Jan 2021
...And plain openmpi-4.0.4 is non-BFB in some restart tests. I'll ask for a openmpi/2.1.5-hpcx install.
12 fails:
$ ./cs.status.20210109_010421_btwjnz | grep "Overall: FAIL"
ERP_Ld3.ne30_oECv3_ICG.A_WCYCL1850S.chrysalis_intel.allactive-pioroot1 (Overall: FAIL) details:
ERP_Ln9.ne4_ne4.FC5AV1C-L.chrysalis_intel (Overall: FAIL) details:
ERP_Ln9.ne4_ne4.F-EAMv1-AQP1.chrysalis_intel (Overall: FAIL) details:
ERS_Ld5.ne4_ne4.FC5AV1C-L.chrysalis_intel.eam-gust_param (Overall: FAIL) details:
ERS_Ld5.ne4_ne4.FC5AV1C-L.chrysalis_intel.eam-rrtmgp (Overall: FAIL) details:
ERS_Ld5.T62_oQU120.CMPASO-NYF.chrysalis_intel (Overall: FAIL) details:
ERS_Ln5.ne4pg2_ne4pg2.FC5AV1C-L.chrysalis_intel.eam-thetahy_sl_pg2 (Overall: FAIL) details:
ERS_Ln9.ne4_ne4.FC5AV1C-L.chrysalis_intel (Overall: FAIL) details:
ERS.ne11_oQU240.A_WCYCL1850.chrysalis_intel (Overall: FAIL) details:
ERS.ne4_ne4.FC5AV1C-L.chrysalis_intel.eam-thetahy_ftype2 (Overall: FAIL) details:
ERS.ne4_ne4.FC5AV1C-L.chrysalis_intel.eam-thetanh_ftype2 (Overall: FAIL) details:
PET_Ln9_PS.ne30_oECv3_ICG.A_WCYCL1850S.chrysalis_intel.allactive-mach-pet (Overall: FAIL) details:
@amametjanov , If easy, try PIO1
<entry id="PIO_VERSION" value="1">
with openmpi-4.0.4 for these same restart tests. Would like to verify that this makes a real difference.
worleyph
on 9 Jan 2021
I ran a 1 month experiment, (2 processes, one experiment on two nodes and one on one node). The history files
FC5AV1C-L_ne4_ne4_chrys_intel_openmpi_2b.eam.h0.0001-01.nc
FC5AV1C-L_ne4_ne4_chrys_intel_openmpi_2.eam.h0.0001-01.nc
were identical, and the restart files (still) differed in the usual two fields. I assume that these two fields are not in the history files. Might be worth adding then, just to check whether the corruption is specific to the restart or not. Unfortunately, I forget how to specify which fields go into history. Can someone point me at an example (know that there are many of these on Confluence)? Thanks.
worleyph
on 9 Jan 2021
@amametjanov Reverting to OpenMPI 2.1.5 version (pretty old compared to current 4.x) seems like a drastic workaround as there may be bugfixes and performance regressions. Is it HPCX version or OpenMPI? Could you clarify why you picked that particular version?
Anyway, it appears that the reproducibility issue is limited to Scorpio and hopefully to _some_ MPI operation there.
sarats
on 9 Jan 2021
openmpi-4.0.4 had build issues with pnetcdf-1.11.0 (both e3sm_integration runs are with netcdf: branch azamat/chrysalis/add-plain-ompi-4_0_4).
With PIO_VERSION=1:
azamat@chrlogin1:20210109-int-ompi-4.0.4-pio1$ ./cs.status.20210109_101847_u3ei98 | grep "Overall: " | wc -l
88
azamat@chrlogin1:20210109-int-ompi-4.0.4-pio1$ ./cs.status.20210109_101847_u3ei98 | grep "Overall: PASS" | wc -l
84
azamat@chrlogin1:20210109-int-ompi-4.0.4-pio1$ ./cs.status.20210109_101847_u3ei98 | grep "Overall: FAIL" | wc -l
4
azamat@chrlogin1:20210109-int-ompi-4.0.4-pio1$ ./cs.status.20210109_101847_u3ei98 | grep "Overall: FAIL"
ERP_Ld3.ne30_oECv3_ICG.A_WCYCL1850S.chrysalis_intel.allactive-pioroot1 (Overall: FAIL) details:
SMS_B.ne4_ne4.F-EAMv1-AQP1.chrysalis_intel.eam-hommexx (Overall: FAIL) details:
SMS.ne30_oECv3.BGCEXP_BCRC_CNPECACNT_1850.chrysalis_intel.elm-bgcexp (Overall: FAIL) details:
SMS.ne30_oECv3.BGCEXP_BCRC_CNPRDCTC_1850.chrysalis_intel.elm-bgcexp (Overall: FAIL) details:
2 SMS fails are due to lack of pnetcdf in the build. ERP fail is due to a seg-fault in MPAS framework mpas_hash_search function.
pnetcdf-1.11.0 works very well on multiple machines.
I don't know of a specific reason why openmpi-2.1.5 (or 2.1.x series) would underperform other versions in E3SM runs: https://www.open-mpi.org/software/ompi/versions/timeline.php . It works well on other machines: mappy, grizzly, jlse, tulip.
HPCX provides optimizations for MPI collective operations, on top of plain OpenMPI. If plain OpenMPI works (openmpi-2.1.5 above) and OpenMPI+HPCX doesn't, the issue is within HPCX. Here, the issue was with openmpi-4.0.4 + HPCX and we narrowed down to openmpi-4.0.4.
None of 39 E3SM machines run the 4-series of OpenMPI and only 2 run openmpi-3.0.1: most run with 2.x.y and some with 1-series.
It's not clear why 4.0.4 was picked, but I'll check if there are any version floor constraints from HPCX (sys-admin should be back from vacation on Monday).
amametjanov
on 9 Jan 2021
@jayeshkrishna , I tried changing pio_numiotasks by changing PIO_STRIDE and/or PIO_NUMTASKS, but was unsuccessful. Are PIO_STRIDE and PIO_NUMTASKS ignored now? I tried with both SCORPIO and SCORPIO_classic . I looked in cpl.log to check the values.
worleyph
on 9 Jan 2021
@worleyph add fincl1 = 'varname1','varname2' to user_nl_eam
rljacob
on 9 Jan 2021
@rljacob , thanks. Adding these fields to history did not result in a difference between the history fields for the two experiments. So, this is more confirmation that the corruption creeps in as part of the restart write. Restart writes out many more fields, so perhaps the issue is a function of the number of fields output?
worleyph
on 10 Jan 2021
The stride and numtasks should not be ignored, I will take a look at it.
Meanwhile I tried the standalone Scorpio tests on Chrysalis with openmpi 4.0.4 but could not reproduce/find any issues (the tests passed)
jayeshkrishna
on 10 Jan 2021
One possibility is that the buffers, or part of the buffers, written out aren't initialized (in that case the Fortran - Scorpio classic - and C - Scorpio - versions of the code could have different behaviors, resulting in non-BFB results between Scorpio classic and Scorpio)
jayeshkrishna
on 10 Jan 2021
@jayeshkrishna , thanks.
worleyph
on 10 Jan 2021
Think that @ndkeen already tried this, but verified that building with DEBUG==TRUE (in env_build.xml) does not alter failure for the 2 process reproducer. Also set PIO debug level to 6, saw lots of output, but diffs with the working 1 node run does not indicate any issues (so far - still scanning the diff output).
worleyph
on 10 Jan 2021
I modified write_restart_dynamics to not call PIO_Write_Darray for q==2 (CLDLIQ) and q==22 (mom_a1). The error in the restart file then showed up in dpCLDICE (q==3 for CLDICE) and dpso4_a2 (q==24 for so4_a2), so still two fields, and same "position" once removing q==2 and q==22 in the restart write. So very reproducible.
If I am reading the cprnc output correctly, when including CLDLIQ and mom_a1, the number of values that are different are:
dpCLDLIQ: 2025
dpmom_a1: 105947
while when not including CLDLIQ and mom_a1 in the restart file, the number of differences in CLDICE and so4_a2 are
dpCLDICE: 1401
dpso4_a2: 105935
so, not identical counts, but similar magnitudes.
I tried looking at SCORPIO source, and convinced myself again that I likely can't contribute to diagnosing anything within SCORPIO.
worleyph
on 10 Jan 2021
Also tried changing rearranger option to use 'coll' (MPI_Alltoallw) instead of p2p, and this changed nothing (restart files were identical for the "failing" case). Also added timers to verify that 'coll' and 'p2p' change what SCORPIO is doing (and they do).
worleyph
on 10 Jan 2021
I ran the usual 2 process/2 node reproducer (one process on each node) but using default (Intel) MPI and compared this with OpenMPI using 2 processes both on the same node (which I have been assuming is "correct"), and the restart files are not the same:
A total number of 268 fields were compared
of which 5 had non-zero differences
I then compared this Intel MPI run with the OpenMPI build using pio1, and saw the same differences
A total number of 268 fields were compared
of which 5 had non-zero differences
The differences are "different" though:
1341 4608 ( 366, 8) ( 84, 9) ( 366, 8) ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
4608 ******** ************************ 0 0
4608 72 0 0 0
4608 ( 2, 1) ( 1, 2)
RMS numlev ********** NORMALIZED 0
1597 4608 ( 2, 7) ( 214, 5) ( 2, 7) ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
4608 ******** ************************ 0 0
4608 8 0 0 0
4608 ( 1, 12) ( 1, 2)
RMS hwrt_prec 417 NORMALIZED 0
648 4608 ( 22, 2) ( 20, 2) ( 20, 2) ( 5, 2)
4608 2.590356091471633+289 -1.366460216890483+294 1.4+294 -1.366460216890483+294 1.4E-01 -7.103705588869866E+74
4608 1.000000000000000E+36 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
4608 ( 1, 1) ( 1, 2)
avg abs field values: 5.930929341658394+290 rms diff:Infinity avg rel diff(npos): 1.4E-01
9.440104188368014E+34 avg decimal digits(ndif): 0.0 worst: 0.0
RMS fillvalue Infinity NORMALIZED Infinity
1199 4608 ( 40, 2) ( 1, 1) ( 40, 2) ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
4608 ******** -1**************** 0 0
4608 0 -1 0 0
4608 ( 1, 2) ( 1, 1)
RMS meridional_complement 0 NORMALIZED 0
620 4608 ( 2, 2) ( 1, 1) ( 2, 2) ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
4608 ******** -1**************** 0 0
4608 0 -1 0 0
4608 ( 1, 2) ( 1, 1)
RMS zonal_complement 0 NORMALIZED 0
Comparing instead with the OpenMPI 2 processes on 2 nodes, the differences are in 7 fields, the above 5 as well as the usual 2 corrupted fields (CLDLIQ, mom_a1).
So ... the E3SM "field" variables are the same using Intel MPI and 2-processes-on-one-node OpenMPI , but the above "auxilliary" variables are different? Have no idea what this means, but perhaps it will mean something to @jayeshkrishna .
worleyph
on 12 Jan 2021
I now have a Scorpio reproducer for this issue. I will update soon with more information.
jayeshkrishna
on 13 Jan 2021
Yay!!!!
worleyph
on 13 Jan 2021
Quick update: I have been able to reliably reproduce the issue with the Scorpio reproducer. I am currently trying to make sure that MPI works correctly for point to point and alltoall communication patterns (An MPI reproducer, if the issue is indeed in the MPI library, would be really useful. If these sanity checks with MPI looks ok, I will be looking deeper in Scorpio to debug the issue further. Currently point to point communication between two processes looks fine. I am looking at alltoall communication patterns now.)
jayeshkrishna
on 14 Jan 2021
Update:
- All my MPI communication tests seem to be working correctly (I tried alltoall I/O patterns, similar to E3SM, with non-blocking point to point routines - no issues seen so far)
- The Scorpio reproducer is still able to reproduce the issue (The job needs to run on at least 2 nodes to reproduce the issue, i.e., oversubscribing and running the job on a single node does not reproduce the issue)
- Looking inside Scorpio, the data passed to the lower level I/O libraries looks good, but the data read back is corrupted (and this occurs only when a lot of data is written out)
- Also tried the latest version of PnetCDF (1.12.2), but the issue persists.
I am going to try to see if I can reproduce the issues with an MPI I/O reproducer.
jayeshkrishna
on 19 Jan 2021
According to https://www.open-mpi.org/software/ompi/major-changes.php :
The v4.1.x series introduced the following major feature enhancements compared to the v4.0.x series:
The "OMPIO" module is now the default module for the MPI-IO APIs on all filesystems -- including Lustre.
Prior to v4.1.0, OMPI was the default for all filesystems except Lustre (and the "ROMIO" module was the default for Lustre).
HPCX-2.7.0 is based on OpenMPI-4.0.4rc3: on Chrysalis module load openmpi/4.0.4-hpcx-cy5n3ft.
Latest HPCX-2.8.0 is based on OpenMPI-4.1.1a1: module load openmpi/4.1.1a1-hpcx-2.8.0-u4oxaa4.
amametjanov
on 19 Jan 2021
Thanks @amametjanov , I tried the latest OpenMPI stack (OpenMPI-4.1.1a1) but unfortunately the issue still persists.
jayeshkrishna
on 19 Jan 2021
This issue was finally narrowed down to bugs in the OpenMPI library. Some of the transport protocols supported by OpenMPI on Chrysalis has data corruption issues and hence the workaround is to use (explicitly specify) the transport protocols that work.
- We have an MPI-only reproducer that shows the issue. The issue reliably shows up on runs, on Chrysalis, with more than a single node.
- The issue does not show up with sm, tcp and ud UCX transport protocols supported by OpenMPI. We have tried the MPI-only reproducer with up to 4 nodes (4x64 MPI processes) and the failing E3SM test case above to verify that these transport protocols do not cause data corruption.
- The issue also does not show up with OpenMPI installations (OpenMPI 4.0.4+) on anlworkstation (a single node workstation)
- @amametjanov also verified that the performance is not affected by explicitly specifying the UCX transport protocols
jayeshkrishna
on 2 Feb 2021
Just documenting for reference:
sm: shared memory transport (_so shouldn't affect across-node which makes sense_)
tcp: TCP over SOCK_STREAM sockets _(plain TCP works but not accelerated)_
ud: unreliable datagram
@jayeshkrishna What transports exhibit the data corruption?
Is it dc (Mellanox scalable offloaded dynamic connection transport) and/or rc (reliable connection)?
sarats
on 3 Feb 2021
Both dc and rc transport protocols show data corruption.
jayeshkrishna
on 3 Feb 2021
Ok, so infiniband specific transports are not working.
I presume dc is the vendor recommended transport. I wonder if ud_verbs would show similar corruption, anyway it's for the upstream OpenMPI/vendor to dig into.
Ref: https://openucx.readthedocs.io/en/master/faq.html#which-transports-does-ucx-use
On machines with RDMA devices, RC transport will be used for small scale, and DC transport (available with Connect-IB devices and above) will be used for large scale. If DC is not available, UD will be used for large scale.
sarats
on 3 Feb 2021
I have created an issue with the systems folks and created an OpenMPI issue (https://github.com/open-mpi/ompi/issues/8442).
jayeshkrishna
on 3 Feb 2021
Update: OpenMPI 4.1.1 has the required fix and ERS_Ln9.ne4_ne4.FC5AV1C-L now passes on Chrysalis (without the UCX transport workaround above).
This change will soon be integrated to the configuration files.
jayeshkrishna
on 27 Apr 2021
Related issues
kaizhangpnl
·
12Comments
ndkeen
·
8Comments
jasonltorchinsky
·
12Comments
minxu74
·
5Comments
brhillman
·
13Comments
Most helpful comment
I have created an issue with the systems folks and created an OpenMPI issue (https://github.com/open-mpi/ompi/issues/8442).