E3sm: Out of bounds error with PGI 19.9 on Compy

[singhbalwinder]

With PGI 19.9, I get the following error:

Subscript out of range for array zvirv (/qfs/people/sing201/delete/2nd_e3sm_pgi_down/E3SM/components/cam/src/physics/cam/physics_types.F90: 423)
0:     subscript=1, lower bound=140722428184144, upper bound=140722428184171, dimension=1

zvirv id declared as follows:
real(r8) :: zvirv(state%psetcols,pver) ! Local zvir array pointer

As there is nothing which is obviously wrong here, I tried a few things to see if it works with these ad-hoc fixes. There is another instance where the same named variable is used in this file, I tried renaming this variable (PGI sometimes confuses scope for variables) and that didn't fix this issue. After that I hardwired a number instead of state%psetcols, that worked but then the model crashes in another subroutine with a very similar error.

To reproduce this, simply switch PGI to 19.9 in the machine file for Compy and run the following test:
cd cime/scripts
./create_test SMS_D.ne4_ne4.FC5AV1C-L.compy_pgi -t <id> -p <project>

[rljacob]

@minxu74 @mrnorman @whannah1 Do you see the error above for that case on Summit? This is preventing us from using the same compiler version on Compy.

@singhbalwinder is that the only test that fails with that compiler version?

[mrnorman]

I haven't seen this before. It seems to suggest one of three possibilities: (1) Another memory error aliasing onto this occurrence; (2) An error in the index checking engine (perhaps it doesn't manifest if we turn off index checking); or (3) An internal error in the PGI compiler that corrupted the lower bounds of the array descriptor. Only option (1) would be a coding error result, but I don't know how we test that.

Is E3SM regularly run through valgrind by chance?

[whannah1]

I think @brhillman has looked into this.

[mrnorman]

Looking at the code, it looks like psetcols is given a default initialization in the derived type (components/cam/src/physics/cam/physics_types.F90):

  59   type physics_state
  60      integer                                     :: &
  61           lchnk,                &! chunk index
  62           ngrdcol,              &! -- Grid        -- number of active columns (on the grid)
  63           psetcols=0,           &! --             -- max number of columns set - if subcols = pcols*psubcols, else = pcols

I wonder if we're uncovering a case where the PGI compiler did not actually initialize this value.

[minxu74]

Let me try it on Summit. I have seen this error before on Titan. I updated the compiler versions or reduced the optimization level of this file to solve the problem, but I cannot remember what I exactly did.

[minxu74]

I got the same error on Summit using PGI 19.9. So it is relevant to the compiler itself.

1: 1: 0: Subscript out of range for array zvirv (/autofs/nccs-svm1_home1/mxu/MyGit/test_pgi/components/cam/src/physics/cam/physics_types.F90: 423)
1: 1:     subscript=1, lower bound=140737327915344, upper bound=140737678851223, dimension=1

[minxu74]

Update:

• the test case was passed on Summit if the PGI compiler was changed from 19.9 to 19.10.
• there was a compiler error from a code in the land component if the PGI compiler version is 20.1
  So shall we plan to update the PGI version to 19.10 if there is no problem for GPU applications under 19.10 version?

[mrnorman]

Please go ahead, the PGI GPU port is garbage right now anyway. We're moving to GNU with C++ ASAP.

[whannah1]

@minxu74 I think there actually was a problem with the OpenACC version of the MMF code when we tried 19.10. However, we're planning to abandon that given how well the C++ version of the CRM code is running. So I'm ok if this breaks the MMF.

[singhbalwinder]

@rljacob : This is the only test I tried as it covers most of the atmosphere components.

Thanks @minxu74 for running the test to confirm that it is more like a compiler bug rather than an architecture or code specific error. We are currently using 19.10 on Compy but it has its own issues. If 20.1 is also not useable, we can try their 20.2 version which I think released this May. Hopefully it will work with all components.

[minxu74]

@singhbalwinder Thanks a lot. Unfortunately, the latest PGI version on Summit is 20.1. It seemed hard to get all tests pass for pgi compilers :). Since the Compy PGI tests are the mirror tests of Summit, we need to keep their PGI version same. I am open to which version we shall choose since all versions have their own issues. If we get very good results by using 20.2, we could send our request to OLCF to let them install the latest version on Summit.

[singhbalwinder]

Thats a good point. I will see if Compy's support can get hold of this new compiler version. If it works on Compy, we can probably ask OLCF to install this newest compiler version.

[rljacob]

If the ECP project is no longer using PGI for their production runs, then we don't need to test PGI on compy.

[minxu74]

@rljacob if we drop off the pgi test on Compy, does it mean that E3SM will not support PGI compiler any more?

[rljacob]

PGI has always been a troublesome compiler and so if no science campaign wants to use it, I don't see the need for testing it with the integration suite and baselines like we are trying to do on compy. We could test developer (without baselines) just for coverage reasons.

[philipwjones]

Well...we currently need it for the ocean OpenACC port so if anyone needs to use the GPU-enabled ocean for a sim, we would need it for that.

[rljacob]

Do you know if the ocean needs a particular version of PGI?

[philipwjones]

Not really. As far as I know, we only have one outstanding (threading) issue that was supposed to be fixed in the July (20.7) release, but we had already committed a workaround. I've been using the 19.9 version for the standalone MPAS work without problems. Haven't tried 19.10 but could do so if need be.

[rljacob]

I assume you are running MPAS standalone in that work? So a C-case in E3SM would be a near-equivalent config we could test with.

[sarats]

FYI, we will probably use PGI for any planned runs on Perlmutter next year. So, just something to keep in mind if we decide to stop tracking PGI now. We will run into all the accumulated bugs then.

[philipwjones]

That would be fine for this ongoing development, so it only would be a problem if we decide to do a coupled run with this enabled, as @sarats just noted. Also, I believe @amametjanov maintains an OpenACC version (or at least can translate his directives to OpenACC) within sea-ice work so maybe a G case would be useful to catch both?

[minxu74]

I forgot that the MPAS are using OpenACC with PGI for GPU acceleration too. I knew that some users are still using OpenACC with PGI on some part of CAM for GPU porting.

[rljacob]

What is the most recent version of PGI available on Summit?

[minxu74]

@rljacob It is 20.1

[rljacob]

@philipwjones can you verify your code works with 20.1 on Summit? If so, we'll switch to that on Compy. Perlmutter will be closer to that version then anything 19.x

[minxu74]

@rljacob there was a compiler error from a code in the land component if the PGI compiler version is 20.1. I have not looked into the error carefully. If we want to move to 20.1, I will try to fix it.

[singhbalwinder]

@minxu74 : If you have time to reproduce the error you got with 20.1, that will be helpful. Other option is to ask OLCF and Compy to move to 20.2 (like you mentioned before) and hope that it works for everybody.

Even if we report a bug to PGI, they will most likely advise us to try the newest compiler version. They may not fix a bug in an older version of the compiler.

[amametjanov]

The latest PGI should be hpcsdk/20.5: rename due to PGI being now part of NVidia.

> pgfortran --version

pgfortran (aka nvfortran) 20.5-0 LLVM 64-bit target on x86-64 Linux -tp haswell 
NVIDIA Compilers and Tools
Copyright (c) 2020, NVIDIA CORPORATION.  All rights reserved.

[philipwjones]

@rljacob looks like I can run with 20.1 - only tried one case but it finished fine

[minxu74]

@singhbalwinder The error message for PGI 20.1 is:

PGF90-S-0000-Internal compiler error. import: module bshr_infnan_mod (4083,base=4086) member symbol shr_infnan_nan$ac (offset=61): not found!  (/autofs/nccs-svm1_home1/mxu/MyGit/test_pgi/components/clm/src/external_models/sbetr/src/driver/alm/ALMbetrNLMod.F90: 28)
PGF90-S-0000-Internal compiler error. import: module bshr_infnan_mod (4083,base=4086) member symbol shr_infnan_snan$ac (offset=63): not found!  (/autofs/nccs-svm1_home1/mxu/MyGit/test_pgi/components/clm/src/external_models/sbetr/src/driver/alm/ALMbetrNLMod.F90: 28)
PGF90-S-0000-Internal compiler error. import: module bshr_infnan_mod (4083,base=4086) member symbol shr_infnan_qnan$ac (offset=65): not found!  (/autofs/nccs-svm1_home1/mxu/MyGit/test_pgi/components/clm/src/external_models/sbetr/src/driver/alm/ALMbetrNLMod.F90: 28)
PGF90-S-0000-Internal compiler error. import: module bshr_infnan_mod (4083,base=4086) member symbol shr_infnan_inf$ac (offset=67): not found!  (/autofs/nccs-svm1_home1/mxu/MyGit/test_pgi/components/clm/src/external_models/sbetr/src/driver/alm/ALMbetrNLMod.F90: 28)
PGF90-S-0000-Internal compiler error. import: module bshr_infnan_mod (4083,base=4086) member symbol shr_infnan_posinf$ac (offset=69): not found!  (/autofs/nccs-svm1_home1/mxu/MyGit/test_pgi/components/clm/src/external_models/sbetr/src/driver/alm/ALMbetrNLMod.F90: 28)

It is an ICE. The line 28 in ALMbetrNLMod is:

    use tracer_varcon  , only : advection_on, diffusion_on, reaction_on, ebullition_on, reaction_method