E3sm: Restart runs with EAM+PGI+Scorpio

[amametjanov]

It appears that Scorpio is writing NULL's to strings in a cam restart file. ERS tests with active ATM pass with scorpio_classic but fail with scorpio during restart run: e.g. for ERS_Ln9_P96x1.ne4pg2_ne4pg2.F-MMF1-TEST.compy_pgi.cam-crmout e3sm.log

80: FIO-F-219/formatted write/internal file/attempt to read/write past end of record.
80:  In source file /qfs/people/mame117/repos/E3SM-tests-comp/components/cam/src/control/cam_history.F90, at line number 2634

Prints of fincllonlat array show up as ^@.

[dqwu]

@amametjanov
Is this issue reproducible on other machines using other compilers?

[amametjanov]

This is specific to PGI compiler and the error occurs on Summit with pgi/19.9 too (Compy loads pgi/19.10): i.e. ERS_Ln9_P96x1.ne4pg2_ne4pg2.F-MMF1-TEST.summit_pgi.cam-crmout.

Side note: threaded ERP tests after the PGI compiler workaround in MPAS-Dev/MPAS-Model#449 run into this issue as well. After that PR makes its way into E3SM,

• PET_Ln9_PS.ne30_oECv3_ICG.A_WCYCL1850S.compy_pgi.allactive-mach-pet
• SMS_P12x2.ne4_oQU240.A_WCYCL1850.compy_pgi.allactive-mach_mods

will PASS (running into Illegal instruction run-time error now).

[dqwu]

@amametjanov @jayeshkrishna
This looks like an issue with PGI compiler. The following simple test program has different run results with gfortran on anlworkstation and pgfortran on Summit. Due to the same issue, subroutine replace_c_null in scorpio code fails to convert NULLs to spaces in a Fortran string.

module foo_mod1
contains

subroutine modify_string(istr)
  character(len=*), intent(inout) :: istr

  ! With GNU, length is 256
  ! With PGI, length is 0
  print *, 'len(istr) = ', len(istr)

  ! Modify 2nd character in the string
  istr(2:2) = 'Y'
end subroutine modify_string

subroutine set_text_internal(ival)
  character(len=*), intent(out) :: ival(*)

  ! Modify 3rd string in the array: array_of_strings(3, 1)
  call modify_string(ival(3))
end subroutine set_text_internal

subroutine get_var_2d_text(ival)
  character(len=*), intent(out) :: ival(:,:)

  ! Size is 4 * 8 = 32
  print *, 'Size of ival = ', size(ival)
  call set_text_internal(ival)
end subroutine get_var_2d_text

end module foo_mod1

program foo1
  use foo_mod1
  Implicit none

  ! 2D array of 32 strings, and each string has a length of 256
  character(len=256), dimension(4, 8) :: array_of_strings

  array_of_strings(3, 1)(2:2) = 'X'

  call get_var_2d_text(array_of_strings)

  ! With GNU, the updated value is 'Y'
  ! With PGI, the value is not updated (still 'X')
  print *, 'Current value of array_of_strings(3, 1)(2:2) = ', array_of_strings(3, 1)(2:2)
end program

[dqwu]

@amametjanov @jayeshkrishna
Test result on anlworkstation with gfortran

 Size of ival =           32
 len(istr) =          256
 Current value of array_of_strings(3, 1)(2:2) = Y

Test result on Summit with pgfortran

 Size of ival =            32
 len(istr) =             0
 Current value of array_of_strings(3, 1)(2:2) = X

[amametjanov]

@minxu74 can OLCF folks take a look at this reproducer? All available PGI versions 19.9, 19.10, 20.1 produce same answer.

[jayeshkrishna]

@minxu74 @amametjanov @dqwu : I am working on this issue too, please wait before contacting the OLCF folks regarding the issue (I still need to verify that its not a Scorpio issue).

[minxu74]

@jayeshkrishna @amametjanov @dqwu Thanks. Once we figure out that the issue is not from Scorpio, I will let OLCF know.

[jayeshkrishna]

Update: A fix for this issue is in https://github.com/E3SM-Project/scorpio/tree/jayeshkrishna/fix_text_put_pgi , this branch is being tested and will be merged to Scorpio (and E3SM) master soon.