E3sm: diagnosing ATM "sum of areas on globe does not equal 4*pi" (Discussion)

[worleyph]

For the case

 -res ne256pg2_ne256pg2 -compset F-EAMv1-AQP1

(actually created with ne30pg2, then modified to get ne256pg2, as per instructions from @mt5555 ) on Cori-KNL using a 32768x4 PE layout with 32 processes per node, out of 13 attempted runs on Aug. 8, 10 of them failed with

  "sum of areas on globe does not equal 4*pi"

which results from

       call get_horiz_grid_d(ngcols, area_d_out=area_d, wght_d_out=wght_d)
 ...
     if ( abs(sum(area_d) - 4.0_r8*pi) > 1.e-10_r8 ) then
        write(iulog,*) ' ERROR: sum of areas on globe does not equal 4*pi'
        write(iulog,*) ' sum of areas = ', sum(area_d), sum(area_d)-4.0_r8*pi
        call endrun('phys_grid')
     end if

I have seen this (intermittently) in the past, as have others. Given the frequency of the occurrence yesterday, I tried to chase it down.

I) I replaced the call to sum(area_d) with a call to shr_reprosum_calc (with ommid=MPI_COMM_SELF, so just using the "infinite precision" summation in serial mode on each process) to eliminate any problems with the "sum" intrinsic. The failures were not eliminated, and all processes generated the same sum. So, it looks like the problem is arising from the values in area_d (being returned by get_horiz_grid_d, calculated in dyn_grid.F90 in the SE dynamics).

Also, the sum differed from failed run to failed run, so the failure mode is not deterministic (by this metric).

II) I added logic for masterproc to write out all values of area_d. I captured 3 failures and 1 success after adding this logic. The end result is that exactly 104 values differed between the "correct" area_d values and the "incorrect" values. Which 104 values were affected differed with each failed run, though they are all in the same range. They were in blocks, but were not all contiguous.

 DIFF1: 938977-938992, 940001-940016, 941033-941040, 942057-942064
 DIFF2: 947073-947088, 948097-948112, 949129-949136, 950153 -950160
 DIFF3: 987169-987184, 988193-988208, 989225-989232, 990249-990256

out of a range of 1-1572864 . The "bad" values were not obviously bad, e.g. ('<': correct; '>': incorrect)

 990249,990256c990249,990256
 < AREA_D:           990249       0.7231727626E-05
 < AREA_D:           990250       0.7249076126E-05
 < AREA_D:           990251       0.7225339877E-05
 < AREA_D:           990252       0.7242703676E-05
 < AREA_D:           990253       0.7257767690E-05
 < AREA_D:           990254       0.7274943486E-05
 < AREA_D:           990255       0.7251295801E-05
 < AREA_D:           990256       0.7268486890E-05
 ---
 > AREA_D:           990249       0.6288712975E-05
 > AREA_D:           990250       0.6303799236E-05
 > AREA_D:           990251       0.6283158186E-05
 > AREA_D:           990252       0.6298257752E-05
 > AREA_D:           990253       0.6311367966E-05
 > AREA_D:           990254       0.6326304070E-05
 > AREA_D:           990255       0.6305740000E-05
 > AREA_D:           990256       0.6320689402E-05

For this one case, all of the correct values were of the form 0.7E-05 while all of the incorrect values were of the form 0.6E-05, with one exception:

 < AREA_D:           987171       0.7160102700E-05
 > AREA_D:           987171       0.1509975041E-05

Looking at where the values came from, in compute_global_area in dynamics/se/dyn_grid.F90, there is a sort step:

  ! sort according to global element ID
  do icol = 1,ncol_fv_gbl
    area_d( col_id_global(icol) ) = rbuf(icol)
  end do

so perhaps these are a contiguous block of values before the sort.

III) I removed the endrun for one job that generated the "bad sum" error message, and it later died with

 17227:  WARNING: T/theta initialization inconsistent!
 17227:            1 -5.687913929198806E+064  2.842170943040401E-014
 17227:  T,Tnew   163.546196661751        163.546196661751
 ...
 17227: SHR_REPROSUM_CALC: Input contains  0.10000E+02 NaNs and  0.80000E+01 INFs on process   17227
 17227:  ERROR: shr_reprosum_calc ERROR: NaNs or INFs in input
 17227: Image              PC                Routine            Line        Source
 ...

Note that this is the only process that reported this failure.

HYPOTHESIS:
I believe that this is a signature of a bad processor. The fact that all processes see the identical bad sum would seem to indicate that the error occurs before the MPI_Allgatherv in compute_global_area (because every process reorders and sums the values independently). The fact that the same number of values are bad for each failure (on Aug. 8) implies that it is the same bad processor(s) or node(s) each time. The fact that the specific bad values shift slightly from run to run probably means that a slightly diferent process id is assigned to the bad processor in each run.
The fact that the bad values are not really bad is the one thing to indicate that perhaps there is some code that is partially at fault? or at least sensitive to whatever problem the processor is suffering from?
The fact that letting the one job continue to run then failed for exactly one process may indicated that this was the bad processor (17227 was on node nid10240 for this run).

DISCUSSION:
Since this is not the first time that this has happened, and since it is repeatable in some sense by repeated job submissions for large jobs, it would be nice to try and do something about it. One option is to put some logic in compute_global_area to try to capture this failure earlier, and perhaps to provide a clearer link to a particular process (and processor?). Another option is to remove the nedrun and let it continue running, and thus, maybe indicate the offender by a different failure message.

QUESTION to HOMME/pg2 experts: Does 104 bad values mean anything based this mesh resolution and PE count?

[tangq]

I recently had this error too on cori-knl. But resubmission works fine. The two cases are exactly the same.

[oksanaguba]

iirc these calls to area computations are not in threaded regions.

[mt5555]

interesting results! I'm surprised you can see that at ne30pg2. Are you sure that's correct: ne30pg2 on 32768 MPI tasks?
With the ne30pg2 grid, only the first 5400 MPI tasks will be active in the dycore. With pg2 we have 21600 columns. So I think we can assume physics has one column per MPI task, and there are about 11K idle MPI tasks?

I cant think of anything related to 104 at this resolution / PE count. With 2100 columns on 1024 nodes, I'm not sure what EAM will do: pack them all on the first 675 nodes (32 columns per node), or spread them out (21-22 columns per node). Either way it's not 104.

[worleyph]

@mt5555 - Sorry - read the README.case and copied without thinking. This was an ne256pg2 mesh, with the case created using your instructions, starting from the ne30pg2 create_newcase.

[mt5555]

So 393216 elements, 4x as many columns, so 12 elements per MPI task. and (probably more relevant here) 48 columns per MPI task. I still cant see any way to come up with 104 of something though. would any of the phys_loadbalance options do something other than make sure every task had 48 columns?

[ambrad]

One quick comment. Mod this line:
https://github.com/E3SM-Project/E3SM/blob/43d69ebb67ab1c3e2ce25b21f2abe34302f903cd/components/homme/src/share/gllfvremap_mod.F90#L187
to initialize gfr%check = 1. This will produce diagnostic area output within the physgrid remap module, verifying the actual area sum is correct and it's something at the level of get_horiz_grid_d or lower that is the problem. The output will be in atm.log and will look something like:

 gfr> Running with dynamics and physics on separate grids (physgrid).
gfr> init nphys  2 check 1 ftype 2 boost_pg1 F
gfr> npi  2 nphys  2
 gfr> area fv raw   12.5663706143593       1.159135483911148E-014
 gfr> area fv adj  9.895059008997609E-016  0.000000000000000E+000
 gfr> area gll     9.895059008997609E-016

[worleyph]

@mt555, this is directly out of the dynamics, so has nothing to do with phys_loadbalance.

[worleyph]

@ambrad, thanks. I'll take a look at this. Note that while this current case is the one that I was seeing the issue with, I believe that I (and others) have also seen this error when using the GLL mesh. Can't confirm this at the moment though.

[worleyph]

Just thought to check: every failed job included node nid10240, and none of the successful jobs did. So, very strongly points to this being a system or hardware problem. Still strange that the "bad" vaules aren't "badder".

[mt5555]

I also think it happens with GLL physics grid and not just PG2. In both cases, each MPI task computes the area associated with each column, and then there is an allgatherv array to collect them into a global array on each MPI task? And you conjecture above that the allgatherv is ok - the data going into it is bad. But thinking of the two cases (GLL and PG2), the local area calculation is very different - with PG2 computing. a spherical area, but in the GLL case it's based on the GLL weights and then some communication for edge and corner nodes.

Also, before all of this, HOMME has done a global sum (without the allgatherv) to compute the area of the sphere because it corrects the GLL weight so that they sum to 4pi. I think that is probably just a straight repro_sum() call. (In the GLL case, this is the same data that ultimately feeds into area_d(), but this data is not related to area_d() in the PG2 case)

Hence I remain suspicious of the allgather/allgatherv routines in compute_global_area().

After the allgratherv, would it be easy for each MPI task to check the data in the global array and make sure it matches the data contributed by that MPI task?

[tangq]

The case I had this issue is a coupled simulation with the maint-1.0 branch, so it is not PG2.

[worleyph]

and you conjecture above that the allgatherv is ok

Well, the allgatherv is consistent - all processes appear to be receiving that same gathered vector. Trying to think of ways that allgatherv might fail, that significantly limits the options. As only a few elements are "corrupted", and there appears to be an association with a particular node (or process), any error in the allgatherv would have to happen at the input to the routine, so that the same error is then propagated everywhere?

So my assumption is that there is (or was) a bad node, and thus my 1024 node jobs were allocated the same bad nodes at a very high frequency. If this node is still bad, then continued high node counts experiments should have the same high likelihood of failure. I can look at this - and can in particular look at whether the local contribution to the vector is the same after the allgather.

[worleyph]

I'm trying a one node job, and requesting the suspected bad node.

[worleyph]

I ran a case

 -res ne4pg2_ne4pg2 -compset F-EAMv1-AQP1

with a 64x2 PE layout in one node. Worked fine for two tests (grabbing random nodes). However, when specifying that it run on nid10240,, it failed with

   ERROR: sum of areas on globe does not equal 4*pi
   sum of areas =    17.0509610622142        4.48459044785507

I had added some print statements inside of compute_global_area, and the error occurs at least immediately after the mpi_allgatherv (i.e., before the sort by global element ID). I'll see if I can trace it back further, but this is clearly a bad node that needs to be removed from the system.

[worleyph]

 353,360c353,360
 < AREA_D:              353       0.5163471222E+00
 < AREA_D:              354       0.5870778098E+00
 < AREA_D:              355       0.1251350083E+00
 < AREA_D:              356       0.5729215627E+00
 < AREA_D:              357       0.7394981990E+00
 < AREA_D:              358       0.7679336765E+00
 < AREA_D:              359       0.7101930742E+00
 < AREA_D:              360       0.7394981990E+00
 ---
 > AREA_D:              353       0.3013304920E-01
 > AREA_D:              354       0.3426075941E-01
 > AREA_D:              355       0.2921057714E-01
 > AREA_D:              356       0.3343462739E-01
 > AREA_D:              357       0.3675460349E-01
 > AREA_D:              358       0.3816790605E-01
 > AREA_D:              359       0.3529807764E-01
 > AREA_D:              360       0.3675460349E-01

out ot 384 values. "Bad" are '<' .

[worleyph]

Added more instrumentation. For the latest run, all of the "bad" values are from process 22, and they are bad before going into the MPI_Allgatherv (and they are unchanged by the MPI_Allgatherv). I see that some processes have more more columns than others? (I see 4 and 8). Could this be true for the ne256pg2 case as well? Explaining the 104 columns? Is the per dynamics process contribution to the physics columns not uniform?

[ambrad]

For clarity, I want to write what I think the process/thread decomp is in these runs, and then please correct what I get wrong.

1024 nodes
32 ranks/node
4 threads/rank
12 spectral elements/rank
48 physics columns/rank
12 physics columns/thread

So could it be that somehow pcols=8 and you're seeing chunk sizes of 8 and 4? If so, a separate issue is we'd like to see pcols=12.

Edit: Also, to answer @worleyph's question: Every process for this problem should be a dynamics process. The space-filling curve by its nature will give a perfectly balanced decomp if one is possible, and for ne256 on 32768 ranks, it is. Finally, each spectral element has 4 physics columns. So we definitely want to see 12 physics columns/thread as above, if I did the arithmetic correctly.

[worleyph]

@ambrad , the 4 vs. 8 was for the small ne4pg2 case I ran on one node (64x2 decomposition). This is the number of local columns contributed to the MPI_Allgatherv in compute_global_area . This has nothing to do with pcols.

[ambrad]

Oops, sorry. So for ne4pg2 with 1x64x2, we get

96 spectral elements
32 ranks with 2 spectral elements
32 ranks with 1 spectral element

I don't know how the infrastructure decomposes the physics columns -- IIRC, and of course you know far better than I, there are different options for physics columns decomp -- but it seems in this case we then get

32 ranks with 8 physics columns
32 ranks with 4

Edit: Also, likely this is also independent of physics columns decomp during time stepping. I think the phys_grid MPI_Allgatherv calls are based on spectral elements, so at this stage, the 8 vs 4 described above likely holds regardless of physics column decomp option. In contrast, in the ne256pg2 case on 1024 nodes, I believe at every stage the perfect load balancing will be reflected in identical numbers across processes.

[worleyph]

Thanks. Again, this has nothing to do with the physics decomposition. The physics hasn't even seen the grid when this all goes down. It is the dynamics sending the physics the grid that it will be decomposing.

[ambrad]

Thanks; I see that now. In the ne256pg2 case on 1024 nodes, I believe at every stage the perfect load balancing will be reflected in identical numbers across ranks, so 12 spectral elements and 48 physics columns per rank during the early initialization phase as well as during time stepping.

[oksanaguba]

i dont really follow this, but what about a different compiler? noel showed me this issue https://github.com/E3SM-Project/E3SM/issues/3628 which now i think is compiler-related.

[worleyph]

@oksanaguba , since the error is eliminated by not using node nid10240 (and always happens if I do use this node) I do not believe that this is a compiler issue.

[whannah1]

@worleyph This is great detective work!

Would it be possible to permanently build in some code for catching this and spitting out an error message that includes the offending node (among other potentially helpful info)? I'm thinking it would only need to be triggered when the area sum is invalid.

[mt5555]

going thru some of my log files:

I found three runs that crashed with NaNs in repro_sum, but they crashed in the dycore spherical area check (which sums the GLL weights using repro_sum), before the phys_grid check being looked at above.

Only one of the runs was using nid10240. However, repro_sum reports the MPI task with the NaNs, and in all three cases this MPI task was owned by nid05577

Edit: in addition, all of my successful runs were not using nid05577

So add nid05577 to the suspicious node list !

Since this is a different error (but also intermittent), here's the signature of the error I'm seeing, from e3sm.log:
@whannah1 : note that it gives the MPI task, and then at the top of the log file you can find the node with that MPI task.

    0:  getMetaSchedule: tmpP:            5           1           3          23
    0:        18205          34           1          56       18206         206
    0:            2         207       14564         229
19602: SHR_REPROSUM_CALC: Input contains  0.12000E+02 NaNs and  0.00000E+00 INFs on process   19602
19602:  ERROR: shr_reprosum_calc ERROR: NaNs or INFs in input
19602: Image              PC                Routine            Line        Source
19602: e3sm.exe           0000000004AF010A  Unknown               Unknown  Unknown
19602: e3sm.exe           00000000030EB1DE  shr_abort_mod_mp_         114  shr_abort_mod.F90
19602: e3sm.exe           000000000324844B  shr_reprosum_mod_         480  shr_reprosum_mod.F90
19602: e3sm.exe           0000000002228B8D  cube_mod_mp_set_a        2228  cube_mod.F90
19602: e3sm.exe           0000000001D1E56C  prim_driver_base_         506  prim_driver_base.F90
19602: e3sm.exe           0000000001D1EDF3  prim_driver_base_         108  prim_driver_base.F90
19602: e3sm.exe           00000000022095BD  dyn_comp_mp_dyn_i         148  dyn_comp.F90
19602: e3sm.exe           0000000001C699DB  restart_dynamics_         422  restart_dynamics.F90
19602: e3sm.exe           0000000000544688  cam_restart_mp_ca         396  cam_restart.F90
19602: e3sm.exe           00000000004FFE57  cam_comp_mp_cam_i         163  cam_comp.F90
19602: e3sm.exe           00000000004F19EC  atm_comp_mct_mp_a         313  atm_comp_mct.F90
19602: e3sm.exe           0000000000425DD8  component_mod_mp_         257  component_mod.F90
19602: e3sm.exe           0000000000413191  cime_comp_mod_mp_        1344  cime_comp_mod.F90
19602: e3sm.exe           0000000000422904  MAIN__                    122  cime_driver.F90

[worleyph]

@whannah1 , I'm not sure. I'll think about it a bit. In this case, delaying the endrun after the sum of areas check allowed it to die in a particular process (for other reasons), and I was able to identify the node that this process was assigned to postmortem.

[worleyph]

One issue that we still have is documenting that the node is bad, so that NERSC will do something about it. The current error is very persistent, so perhaps I can identify a line where things go obviously wrong. In the meantime, perhaps we can add commands to exclude the suspect nodes when we submit jobs. (We can request specific nodes, so I assume that we can avoid them as well).

[whannah1]

Seems like it wouldn't be too hard to make a small reproducer that does some simple sum across all processes of a node.
BTW, I just looked back at ne30 and ne120 runs that had this failure and they both included nid10240.

@worleyph how do you request a specific node?

[worleyph]

@whannah1

 ./case.submit -a "-w nid10240"

The '-a' flags allows you to append options to the underlaying batch submit command.

[worleyph]

Also, the sum is not the problem. The error arises in the calculation of the input to the sum.

[worleyph]

FYI, to exclude this node (to avoid the failure):

 ./case.submit -a "-x nid10240"

(for systems using SLURM, like Cori-KNL).

[ambrad]

FYI, if anybody is trying to use gfr%check = 1, @whannah1 alerted me to the fact that it doesn't show diagnostic output. Turns out I never enabled that for regular runs; it only runs in the unit test. In a future PR, I'll enable it for regular runs. Sorry about the confusion, and thanks @whannah1 for pointing out the output doesn't actually appear.

[mt5555]

@worleyph : can you try your test on nid05577?

As posted above, I looked at some old runs with NaNs during intialization that went away on resubmit, and following your approach they all indicate nid05577. Just this morning I had another bad run, also due to nid05577.

Different issue than described here - failing much earlier with NaNs instead of bad numbers. But fiirst detected in the area-of-the-sphere calculation.

[worleyph]

@mt5555 , no I haven't. I'll try right now.

[ndkeen]

I'm looking at older runs of mine to see if those 2 nodes were used and what the results were.
I found 2 of them that were in May of 2020.

s13-may26/f.ne256pg2.s13-may26.n0128t64x2.wr.24s.N144.pio2.pbs128.st064S2m/run/e3sm.log.31161038.200527-030450:5441:   ERROR: sum of areas on globe does not equal 4*pi
s14-jun4/f.ne256pg2.s14-jun4.n0128t64x1.nr.24s.nc00b.N144.pio2.DEBUG/run/e3sm.log.31558393.200609-162251: 721:   ERROR: sum of areas on globe does not equal 4*pi

Neither of these jobs used nid10240 or nid05577.

Also searching thru older jobs that used scream repo (from around April to June), I don't find any that used nid10240, but I see at least 21 that used nid05577. Now all but 1 of those jobs failed to complete (gzip e3sm.log). I did have a series of jobs that stopped for various reasons, but grepping shows there is not a sum of areas on globe error message. I can go thru these more closely, but were likely larger F cases. One of them did complete (a ne256 job).

Searching thru jobs using some version of master, I surprisingly still don't see any with nid10240, but found 3 jobs with nid05577. These 3 did not complete, but I only know it did not write sum of areas on globe error messages.

We just recently had a nightly test that failed with the sum of areas on globe error.

/global/cscratch1/sd/e3smtest/e3sm_scratch/cori-knl/SMS.ne120_ne120.FC5AV1C-H01A.cori-knl_intel.JMasterHi_res20200808_160343/run/e3sm.log.33283925.200808-182755

It doesn't look like it used nid10240 or nid05577. There was also a G case that failed for some reason. Might be nice to know what happens when other jobs (that are not calling this routine to check sum of areas) get one of these nodes.

When Pat you mentioned asin() in the nersc ticket, this reminded me of my issues with asin(1) -- which may be something different.

https://nersc.servicenowservices.com/nav_to.do?uri=%2Fincident.do%3Fsys_id%3D9b30c322dbd79300b259fb0e0f961923%26sysparm_stack%3Dincident_list.do%3Fsysparm_query%3Dactive%3Dtrue

@ndkeen

[worleyph]

@ndkeen , thanks for looking. This issue is clearly deteministic for these nodes now, but, without knowing the underlying reason, I have no idea if it makes sense that these nodes were not troublesome earlier. Note that nid05577 does not die with the area sum check failure. Rather than the area, the failure is in the calculation of the latitude (and cosine latitude) coordinates, and these aren't actually used until the chemistry initialization. As I mentioned, ASIN is producing NaNs for one process on this node. Wish that I had not given up nid10240 so soon, as I would like to know more precisely where the error is arising when using that node.

[worleyph]

FYI, I get "Info Message: Record not found" when trying to look at yoru asin(1) nersc ticket.

[ndkeen]

OK, the NERSC ticket is INC0123284
It may be that you are not included -- I will try to add you.

There is an older confluence page here:
https://acme-climate.atlassian.net/wiki/spaces/PERF/pages/668926145/Notes+regarding+tracking+down+some+floating-point+issues+on+cori-knl

[worleyph]

Getting a few more runs with nid05577, when the node becomes available. Just now tried running with a DEBUG=TRUE build. It still has problems in the same location. But the NaN is not produced now, because if fails immediately, with:

 18: forrtl: error (73): floating divide by zero
 18: Image              PC                Routine            Line        Source
 18: e3sm.exe           0000000007F29674  Unknown               Unknown  Unknown
 18: e3sm.exe           000000000786B7F0  Unknown               Unknown  Unknown
 18: e3sm.exe           0000000007FDE4C7  Unknown               Unknown  Unknown
 18: e3sm.exe           000000000427F62B  coordinate_system         393  coordinate_systems_mod.F90

at the line

 sphere%lat=ASIN(one/r)

in coordinate_systems_mod.F90 . I also tried with

 asinarg = one/r
 sphere%lat=ASIN(asinarg)

printing out the arguments in both cases, and they are fine. So the divide by zero is happening inside of ASIN?

[ndkeen]

This does sound like the issue I was having with asin(1.0)

[worleyph]

Yeah - but "persistent" with respect to compiler flags (only tried Intel compiler so far), and specific to certain nodes (only nid05577 so far; didn't seem to be a similar problem on nid10240, though I can't verify this now).

[worleyph]

So, I tried a build with gnu, and it died for different reasons (and unrelated to nid05577). Are gnu builds on Cori-KNL supposed to work?

[worleyph]

rebuilt with DEBUG set to true and the build with GNU worked this time. FYI.

[worleyph]

A build using GNU with DEBUG=true does not fail when run on nid05577. I just checked again, and a build using Intel with DEBUG=true still does fail when run on nid05577 (in the usual way), so the problem has not cleared up. ... Not sure what to do with this information, nor what to try next.

[ambrad]

Here, "the usual way" refers to the 4*pi problem, right?

Does the DEBUG=true build disable vector instructions? Perhaps disable them in the release build and see what happens.

Another thought: Although it seems reasonable based on the evidence so far that there is an FP h/w error somewhere, but general to both np4 and pg2 so perhaps not in the column area calculation itself, you could still check the MPI calls by replacing the column area value by 1.0 and seeing if you get the number of columns to machine precision as the sum. You could do this first, then run the area calculation. Then check both values in the log. If the first is right and the second wrong, then that strongly suggests the MPI part of the area check has nothing to do with the issue. (Or perhaps that has already been established.)

[worleyph]

@ambrad, Sorry - I muddled the narrative. Since nid10240 is now unavailable, I've been poking at suspect node number two (nid05577). For this node, the "usual" is generating NaNs from the call to ASIN(one/r) in unit_face_based_cube_to_unit_sphere (in coordinate_systems_mod.F90) for one process. Or, when building with DEBUG=true, an error of the form "divide by zero" in the same ASIN call. This is before the MPI calls, so has nothing to do with anything MPI as far as I can tell. I'll take a look at the DEBUG option with the Intel compiler. Perhaps there is even more basic compiler flags I can try. Thanks.

[ambrad]

OK, thanks for the clarification. Since you've isolated the problem, you could try things like !dir$ no vector and !dir$ optimize:0 in that routine instead of messing with compiler flags. That would make rebuilding and testing fast. (Not sure if mod'ing Depends.make files retriggers a total rebuild or not.)

Edit: Also, at this point and for this problem, maybe you have enough to suggest a single-file standalone reproducer would demonstrate the problem.

[worleyph]

Thanks. I'll check it out. Based on the Macro.cmake file, I've disabled pretty much everything:

   if("${DEBUG}" STREQUAL "TRUE")
     set(CFLAGS "${CFLAGS}  -O0 -g")
     set(CXXFLAGS "${CXXFLAGS}  -O0 -g")
     set(CXXFLAGS "${CXXFLAGS}  -O0 -g")
     set(FFLAGS "${FFLAGS}  -O0 -g -check uninit -check bounds -check pointers -fpe0 -check noarg_temp_created")
   endif()

I also deleted the Depends files, so nothing is sneaking back in from that.

Also, I have yet to find a simple standalone reproducer. The obvious did not work. I might try stripping out files from E3SM and see how far I get, though that does not excite me.

[ambrad]

So Intel DEBUG=true with the above still fails. Maybe include -no-vec in the flags just to be sure.

[ambrad]

Sorry to spam, but something else occurred to me. When you checked the code for failing values, was the asin argument by chance just over 1? Maybe that's triggering a NaN.

Edit: No, this probably can't happen. I think in the 1/r calc, r must come from r = SQRT( one + (cart%x)**2 + (cart%y)**2), so r >= 1 in FP.

[worleyph]

I included the details in my ticket to NERSC - forgot that I had not done the same here. I did check the argument, and it is "middle of the road" for the first failure (1/r = 0.678598344545847). Others are all similar. And the arguments are the same when run on other nodes, in which the failures do not occur.

[worleyph]

Latest "result": if pass a constant to ASIN, e.g. (ASIN(0.0D0)), then it does not abort at this call. However, if I set a variable, e.g. asinarg=0.0D0, and call ASIN(asinarg), it then does abort with "divide by zero" error (when build with DEBUG=true). Both of these calls were included in the same run, so I saw the success for the first call (with output), and then a failure for the second. If this wasn't restricted to this one node (nid05577) it would sure look like a code or compiler bug. asinarg was declared as real_kind, which is actually r8. The original failure was with ASIN(one/r) where one is a real_kind parameter and r is a real_kind variable.

[worleyph]

Even ASIN(1.0D0/1.47362575820790D0) works, but one=1.0D0, r=1.47362575820790D0, ASIN(one/r) and asinarg=one/r, ASUN(asinarg) does not.

[worleyph]

Latest result, for nid05577. Putting the code

 real(r8) :: asinarg

 asinarg = 0.0D0
 write(6,*) "AAA asinarg:", asinarg
 call flush(6)
 write(6,*) "AAA ASIN(asinarg):", ASIN(asinarg)
 call flush(6)

as the first executable lines in the model (in cime_driver.F90), process 18 died with

 18:  AAA asinarg:  0.000000000000000E+000
 18: forrtl: error (73): floating divide by zero

and the other 63 processes wrote out the correct values, e.g.

  0:  AAA asinarg:  0.000000000000000E+000
  0:  AAA ASIN(asinarg):  0.000000000000000E+000

Given this, you'd think that it would be easy to generate a reproducer. I'll look at the environment loaded before running as well? Any other suggestions? Maybe look at the assembly code? I may just report this officially soon.

[worleyph]

Back to looking at nid10240, thanks to @ndkeen , and Steven Leake, and others in NERSC support. (My current job for nid05577 is in the queue). nid10240 is now available upon demand via a reservation, so we do not have to wait for it any more.

a) repeated original failure: one node case using

  -res ne4pg2_ne4pg2 -compset F-EAMv1-AQP1

failing with

 ERROR: sum of areas on globe does not equal 4*pi

and all error in the sum is coming from the wrong summands calculated on process 22. Erroneous values change from run to run though.

b) also tried

 -res ne4_ne4 -compset F-EAMv1-AQP1

and this calculated the sum of areas correctly (different branch of logic calculating the area though), but still died soon after with

 22:  ERROR: shr_assert_in_domain: state%t has invalid value   -12.0043119066262
 22:   at location:            7           2
 22:  Expected value to be greater than   0.000000000000000E+000
 22:  ERROR: Invalid value produced in physics_state by package clubb_ice1.

so still process 22.

c) and tried

 --res f19_g16 --compset X

so no active anything, just the driver. This also failed, with

 29: (seq_frac_check) [lnd init] ERROR aborting
 29:  ERROR: Unknown error submitted to shr_abort_abort.
 30: (seq_frac_check) [lnd init] ERROR aborting
 30:  ERROR: Unknown error submitted to shr_abort_abort.
 22: (seq_frac_check) [atm init] ERROR aborting
 22:  ERROR: Unknown error submitted to shr_abort_abort.

(so still process 22, but processes 29 and 30 also). Looking for more details here.

In any case, multiple error modes. Have to guess which one to pursue that would be most illuminating.

[worleyph]

Latest result, for nid05577: X case did not fail with ASIN call as first executable lines in the model. ???

[worleyph]

Just left the following note in our NERSC help ticket:

 I have three different configurations of E3SM all that fail on nid10240, all in different ways, 
 all deterministically, and all with the errors showing up first on process 22 (out of 64). I took
 one of the new configurations (a simpler case than the other two, kind of) and tried to drill 
 down. I determined that by writing out the values that are erroneous right after the calculation 
 when the errors first occur, these errors disappear. This run then died later with
      22: forrtl: severe (154): array index out of bounds
 So this is an awful combination of HW and compiler. Based on this limited evidence, I'd 
 conjecture that perhaps there is something wrong in the memory or cache or other memory-related 
 HW on this node? Based on this, I do not think that I can generate a reproducer (other than what I 
 have, i.e. the whole E3SM model, though built with much of the code disabled - this is still a lot of 
 code, and that may be important if the issue is with memory). If you think that I can still do something, 
 please advise. Otherwise I'll move on to nid05577.

The three cases were:

a) F-EAMv1-AQP1, ne4pg2_ne4pg2: sum of areas is wrong, and error is bad values being contributed to the sum by process 2
b) F-EAMv1-AQP1, ne4_ne4: state%t has invalid value -12.0043119066262 (expected to be positive), also on process 22
c) X, f19_g16: some fractions_l values computed incorrectly on process 22 in seq_frac_init (where input are all correct); after adding write statements, erroneous calculation is eliminated, but then get index out of bounds in prep_ocn_mod.F90 on process 22.

The X case job was compiled with Intel and DEBUG=false. I also checked with DEBUG=true, and this experiment did not fail. However, the DEBUG=false case works fine on other nodes, so still a bug to report.

[worleyph]

Correction: The failing X case job on nid10240 was compiled with Intel and DEBUG=false. I just tried with DEBUG=true, and the failure disappeared (on nid10240). Correcting the previous comment.

[ambrad]

That's great. So maybe it is a vector instruction that is the problem.

[worleyph]

@ambrad, yes - that now makes sense (as much as anything). This is not true for the other bad node though - Intel/DEBUG still failed for nid05577.

[worleyph]

FYI - I created two tarballs, one with just CIME and an X case example to document the issues with nid10240, and one with E3SM (without any components except cam and homme) and an aquaplanet case example to document the issues with nid05577. I pointed NERSC at these in the existing NERSC incident ticket.

[worleyph]

Closing this, since the nodes in question have been removed and we are not seeing additional occurrences (to my knowledge).