E3sm: Build error on MPAS-LI with PGI compiler on Summit

[minxu74]

There is a build error when comping MPAS-LI on Summit, The error message is as follows:

PGF90-S-0000-Internal compiler error. items were added to sem.p_dealloc but not freed       0  (mpas_li_subglacial_hydro.f90: 1288)

The test name is SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.summit_pgi. I have tried PGI/18.7, 18.10 and 19.1 on Summit and got the same error.

[singhbalwinder]

@minxu74 : I confirmed that I get the same build error on Compy with PGI 18.10. It works fine with the Intel compiler.

[minxu74]

@singhbalwinder Cool! Thanks a lot for the confirmation.

[matthewhoffman]

@minxu74 or @singhbalwinder , are you able to report what line 1288 of mpas_li_subglacial_hydro.f90 is? The .f90 files are preprocessed versions of the .F files in the repo, so it's not clear exactly what line I should be looking at. I don't have access to Summit or Compy yet, so I'm not able to try to reproduce this error myself yet.

[minxu74]

@matthewhoffman The 1288 is the line of "end subroutine", it seems not very useful.

1274       waterPressure = max(0.0_RKIND, waterPressure)
1275       waterPressure = min(waterPressure, rhoi * gravity * thickness)
1276       ! set pressure correctly under floating ice and open ocean
1277       where ( (li_mask_is_floating_ice(cellMask)) .or. &
1278               ((.not. li_mask_is_ice(cellMask)) .and. (bedTopography < config_sea_level) ) )
1279          waterPressure = rhoo * gravity * (config_sea_level - bedTopography)
1280       end where
1281 
1282       waterPressureTendency = (waterPressure - waterPressureOld) / deltatSGH
1283 
1284       call calc_pressure_diag_vars(block, err_tmp)
1285       err = ior(err, err_tmp)
1286 
1287    !--------------------------------------------------------------------
1288    end subroutine calc_pressure

[matthewhoffman]

@minxu74 , thanks for posting that. That's what I was afraid of! :)

[matthewhoffman]

I can't find anything obviously wrong with that subroutine or the code that calls it. My understanding is that an internal compiler error is most likely a bug in the compiler, and sem.p_dealloc is most likely an internal data structure used by the compiler. I'm not sure how to proceed in trying to debug this without input from a PGI developer. @minxu74 , @singhbalwinder , @rljacob , do we have any PGI developer contacts (e.g. supporting roll-out on Summit) that we could ask for help with this?

[matthewhoffman]

@amametjanov , does this mean that we should not use where-constructs from now on? I'm guessing that is not the case, because I still see quite a few where-constructs in the code. How were these specific where-constructs identified as problematic? I just want to make sure we don't add new code that triggers this same problem.

[philipwjones]

@matthewhoffman - @amametjanov can comment on the specifics for these loops, but as a general practice, explicit loops are both safer and more performant. You might think the compiler would translate into identical code behind the scenes and that should be the case in many situations, but sometimes when attempting to optimize a where construct, the compiler might calculate both true/false branches or precompute array masks that lead to computations on garbage data and create problems. In the older POP model, we eventually had to replace all where's with explicit loops. Not something you need to do right away, but when opportunities present, you might think about the same.