E3sm: maint-1.0 crashes with A_WCYCL "L" configuration on Compy

[mt5555]

On compy, with the maint-1.0 branch, this case

./create_newcase -case /qfs/people/wolf966/E3SM-cases/20190717.test3.compy -compset A_WCYCL1850S_CMIP6 -mach compy -res ne30_oECv3_ICG -compiler intel --project e3sm -pecount L

will crash - with errors indicating memory corruption in HOMME's dof_mod.F90:

* Error in `/compyfs/tayl519/e3sm_scratch/w1/bld/e3sm.exe': corrupted size vs. prev_size: 0x0000000038dc59d0 *

With the following workarounds, (and changing I/O type to netcdf because the current maint-1.0 will try and fail to read a few NETCDF4 files), the code will run and get to the point where it reads the restart files and then stops (because I dont have the correct restart files)

Any one of these will let the code run:

1. Change the two arrays used in mpi_allreduce from heap arrays to stack arrays. E3SM master happens to have done this (for completely unrelated reasons) and hence E3SM master avoids this issue.

2. Break the mpi_allreduce (which is of length 5400) into smaller arrays. doing in mpi_allreduce on 1:5000 followed by 5001:5400 works. Also doing 5400 mpi_allreduces, each of lenght 1 works.

3. use the MPI_IN_PLACE feature of mpi_allreduce.

4. Running the "M" configuration instead of "L" (as reported by @jonbob )

So the main thing that doesn't work is calling mpi_allreduce() when the atmosphere is using 2700 tasks, and with large arrays (> 5000) that happens to have been allocated instead of
placed on the stack.

Behavior is very strange, but since there are so many configurations that work, it does suggest the bug might not be in the E3SM code but could be in Intel MPI

[rljacob]

Intel MPI is the default on maint-1.0 and master for compy.

[mt5555]

thanks, edited the issue mvapich->intelmpi

[mt5555]

a few more updates:

Verified that the problem is the same on E3SM master.

Examining the arrays before and after the call to mpi_allreduce, it looks like the call to mpi_allreduce (without using the MPI_IN_PLACE option) is producing the correct data in the output array, but is corrupting the input array.

if the input array is a stack variable, then this is harmless, since we dont use this data after the mpi_allreduce.

if the input array is a heap variable (i.e. allocatable), then it appears the input array data is not corrupted, but instead the metadata that describes the array (such as its length) is corrupted, and this leads to the deallocate errors.

CURRENT CONCLUSION: master is working, but its call the mpi_allreduce() is corrupting the data in the input array. this appears to be harmless as long as this is a stack array, but it suggests a more serious bug lurking somewhere. I think it would be safer to adopt the same solution as maint-1.0, and switch to an MPI_IN_PLACE allreduce

[mt5555]

another update: using master, and MPI_IN_PLACE, with allocated arrays, avoids the deallocate error messages, but the data in the array is corrupt, and this bad data triggers an abort() latest on during the initialization. So MPI_IN_PLACE is not 100% safe either.

[worleyph]

Hi @mt5555 , I tried

 ./create_newcase -case0190717.test3.compy -compset A_WCYCL1850S_CMIP6 -mach compy -res ne30_oECv3_ICG -compiler intel --project e3sm -pecount L

in master, to start trying to reproduce these issues. This dies with

0: pio_support::pio_die:: myrank= -1 : ERROR: ionf_mod.F90: 235 :
0: Feature is not yet supported.
0: NC_FORMAT_NETCDF4 is not yet supported

How did you get this case to run?

Thanks.

[worleyph]

@mt5555 - ignore previous question. Found the answer in your original description of the problem.

[worleyph]

I reproduced the error (saved a copy of the array gOffset before the call to MPI_Allreduce, then compared with the array after the call - the file was changed). I then tried with the same layout for all components except ATM, where I reduced the number of processes from 2700 to 1350. In this case, the input file was not corrupted. Strange to be sensitive in this way.

[mt5555]

Agreed - it's strange, and I dont think there is enough evidence to rule out a bug in e3sm code.

recently @amametjanov posted (maybe in slack?) some other mysterious behavior and he was starting to suspect intel19.

glad your taking a look at this - good luck :-)

[worleyph]

same issue with 5400 ATM processes, so 5400 bad, 2700 bad, 1350 good.

[worleyph]

@mt5555 , I replaced

 call MPI_Allreduce(gOffset,numElem2P,nelem,MPIinteger_t,MPI_SUM,par%comm,ierr)

with

 call MPI_Reduce(gOffset,numElem2P,nelem,MPIinteger_t,MPI_SUM,0,par%comm,ierr)
 call MPI_Bcast(numElem2P,nelem,MPIinteger_t,0,par%comm,ierr)

and this also seemed to eliminate the corruption in gOffset. Please give this a try in your experiments. It seems even more likely that this is a real bug in MPI_Allreduce, and not a side effect of something in E3SM.

[ambrad]

@worleyph, if this is a real bug in the MPI_Allreduce impl, does it follow that a standalone reproducer can be written to send to the implementors? Or are there complexities due to the E3SM runtime environment or perhaps state that complicate making a simple standalone reproducer?

[worleyph]

I'll try to generate a reproducer later today. Code in question is very simple, so should not take much time. And thanks for pointing out the obvious next step (or obvious previous step :-) ).

[mt5555]

A reproducer is a good idea - but I'll note that in the e3sm code, if you change the variable to be allocated instead of stack, then the input array data is not corrupted (but some other weirdness happens) causing crashes later. so something bad is happening, but it's not always detectable.

[worleyph]

I have a reproducer. Want to see how simple I can make it but ... who should I give this to?

[mt5555]

Does this confirm it's a bug in Intel MPI? In which case Bibi (Compy POC) will know who to report it to.

[worleyph]

Seems to be a compiler bug rather than an MPI bug. My reproducer calls a subroutine that then calls MPI_Allreduce. However if I move the call to MPI_Allreduce and associated logic up out of the subroutine, the error disappears.

[worleyph]

or maybe an MPI bug that corrupts memory as a side effect. In any case, not an E3SM bug, unless my reproducer mimics an E3SM error that I am not recognizing. Someone else should look at this code.

[worleyph]

@bibiraju , I have a reproducer for you to pass onto the system administrators and then perhaps the Intel contact. You should verify that you see the same behavior as well, and also look at the code to make sure that you don't see anything obviously wrong.

[bibiraju]

@worleyph please let me know how to test it and I can report to the system admins.

[worleyph]

@bibiraju , please grab ~worl312/FOR_BIBI.tar.gz . In this directory (FOR_BIBI/) is a Notes.txt file with instructions. OLD_EXPTS/ has results from my runs, so you can compare with this. The corruption appears to not be tied to specific processes, but it happens every time for at least one process when using 2700 or 5400 processes.

[amametjanov]

intel18 appears to be better,

module load cmake/3.11.4 intel/18.0.0 intelmpi/2018 netcdf/4.6.3 mkl/2018

but it shows signs of message replay within the cluster:
with 2700 tasks

1549: RES:   1549   5400      1   2700
1549: n0071.local:CMA:2e749:1afaa700: 4274452 us(4274452 us!!!): dapl_cma_active: CM ROUTE ERROR: -> DST 172.17.100.73 retry (15)..
1549: RES:   1549   5400      1   2700
1549: RES:   1549   5400      1   2700
1549: n0071.local:CMA:2ec84:99b57700: 4266530 us(4266530 us!!!): dapl_cma_active: CM ROUTE ERROR: -> DST 172.17.100.72 retry (15)..
1549: RES:   1549   5400      1   2700

with 5400 tasks

5319:  STATS:        5400  1275069467
5319:  DIFF:         197           0           1
5319:  STATS:        5400  1275069467
5319:  DIFF:         198           0           1
5319: RES:   5319   5400      1   5400

[bibiraju]

I have verified the behavior and reported the issue to Compy Support team. Balwinder also verified the issue.

[worleyph]

I found documentation on the algorithms supported for MPI_Allreduce in the Intel MPI library, in

 https://software.intel.com/en-us/mpi-developer-reference-windows-i-mpi-adjust-family-environment-variables

I couldn't find a command to describe what the default is, but I tested each of the major algorithms and discovered:

 SUCCESS?    ALG #     ALGORITHM NAME
 Worked           1    Recursive doubling
 Failed           2    Rabenseifner's
 Worked           3    Reduce + Bcast
 Worked           4    Topology aware Reduce + Bcast
 Worked           5    Binomial gather + scatter
 Worked           6    Topology aware binominal gather + scatter
 Worked           7    Shumilin's ring
 Probably Worked  8    Ring
 Worked           9    Knomial
 Worked          10    Topology aware SHM-based flat
 Worked          11    Topology aware SHM-based Knomial
 Worked          12    Topology aware SHM-based Knary

The only algorithm that demonstrated the error (for 2700 processes) is Rabenseifner's. The Ring algorithm was very slow, and the test program ended without output from all processes showing up in the output file. I might need to add a barrier to the test code to get Ring to report all of the results, but I did not see any "DIFF" error messages, and most processes reported.

Most "sophisticated" Allreduce algorithms are hybrids - the algorithm used is a function of the vector length and the number of processes, and it may even switch algorithms internally for intranode optimizations or as the vector length changes (e.g. recursive doubling). I am guessing that the default algorithm is choosing Rabenseifner's for 2700 and higher process counts, and using something else for smaller process counts. For the Intel18 results that @amametjanov reported, probably the "switch" was set differently, but Rabenseifner's is still broken.

In short, we can fix this now be setting the environment variable

 I_MPI_ADJUST_ALLREDUCE

to one of the working algorithms and overriding the default.

[bibiraju]

Here's the response from Compy support team.

It appears that setting an environment variable to change the algorithm of the allreduce is a workaround. That being said, there is also a 2019u4 update that we can try before opening a case with Intel. I'm going to put Kurt as the contact on the item. Intel almost never supplies us with a list of the bugs that are fixed between releases.
Tim

[worleyph]

So, there is a workaround. Should we drop it in? This can be a compy-specific environment variable. I don't think that we have any performance-sensitive allreduces in the code, but Recursive doubling is probably a reasonable choice - just don't use Ring, or Rabenseifner's :-).

[sarats]

As we have a workaround, this is FYI for future. In a conversation with @vamsi-sripathi, he suggested setting I_MPI_DEBUG=5 and running the reproducer to understand what's happening under the hood.
Ref: https://software.intel.com/en-us/mpi-developer-reference-linux-other-environment-variables

As the bug appears/disappears based on whether the array is allocatable/stack, he suggested looking into alignment as well ... forcing alignment to 64 bytes for the the array.

[rljacob]

Should we also do this on master?

[worleyph]

I believe that @mt5555 observed the same behavior using master, so I assume so. Just got a note that the latest version of the Intel MPI library is available for me to try. I'll give that a whirl as well.

[worleyph]

The new intel library (intelmpi/2019u4) and associated version of the compiler (intel/19.0.4) changed nothing, so looks like we will have to go with manually setting the MPI_Allreduce algorithm for the moment. @sarats , thanks for contacting Vamsi, but I_MPI_DEBUG=5 provided no information: ...

   21: MPI startup(): Imported environment partly inaccesible. Map=0 Info=0
 ...
 1660: MPI startup(): Imported environment partly inaccesible. Map=0 Info=0
    0: [0] MPI startup(): libfabric version: 1.8.0a1-impi
    0:
    0: [0] MPI startup(): libfabric provider: psm2
    0:
    0: [0] MPI startup(): Rank    Pid      Node name    Pin cpu
    0: [0] MPI startup(): 0       202165   n0257.local  0
 ...
    0: [0] MPI startup(): 2699    177483   n0366.local  19
    0: [0] MPI startup(): I_MPI_ROOT=/share/apps/intel/2019u3/compilers_and_libraries_2019.3.199/linux/mpi
    0: [0] MPI startup(): I_MPI_DEBUG=5

and then the normal output.

[worleyph]

@rljacob , I am preparing a PR to set the environment variable in env_mach_specific.xml for Compy when using the Intel MPI library. This will be for master only. If all goes well, I'll let someone else do something similar for maint-1.0 .