E3sm: optimization and parameter constant debug flags

[rgknox]

This is a performance question to all.

In FATES, we often will set a parameter constant debug flag at the top of each module. Something like:

logical, parameter :: debug = .true.

I was curious if anyone knew if the compilers are smart enough to auto remove code encapsulated within if clauses that use these parameter constants? For instance:

do i = 1, many_i_steps
    ....
    if(debug)then
        ! various model checks go here
    end
end

be compiled as:

do i = 1, many_i_steps
    ....
end

I was kind-of assuming they were, but I have not found a definitive answer on this on the internets. I suppose this is also a compiler by compiler question.

[philipwjones]

Sometimes compilers are smart enough at dead code elimination, but I don't know that I would rely on that. I suspect a pre-processor directive (#ifdef DEBUG) is the safer bet.

[rgknox]

Thanks at @philipwjones. Currently we have to avoid pre-processor directives in our code to stay compatible with CESM (unless they changed that requirement), but that does sound like a safe way to do it.

[ndkeen]

I would prefer preprocessor directives myself (and recommend), however, I would think the compiler should be able to eliminate that code as the variable has the parameter keyword. You could easily construct a simple test to verify by inserting the test within a very busy loop.

[rgknox]

I tried @ndkeen 's test, with the gfortran compiler. It uses the system_clock to measure wall-time (which I'm sure has errors, but hopefully that is circumvented by having a suitably long loop). It compares two loops, each of which do some simple math, and only one of which calls the parameter constant in an if clause.

The following is a little program called "TestIfConstants.f90":

program TestIfConstants


  integer :: i       ! Loop index

  integer, parameter :: r8 = selected_real_kind(12)

  real(r8) :: a

  integer, parameter :: nsteps = 1000000000
  integer :: c1,c2,c3,cr,cm

  logical, parameter :: debug = .false.


  call system_clock(count_rate=cr)
  call system_clock(count_max=cm)

  ! Perform a loop without any call to an if

  call system_clock(c1)
  a = 0._r8
  do i = 1, nsteps
     ! Do some simple math so the loop is ensured to be activated
     a = a + 1._r8/ real(i,r8)
  end do
  call system_clock(c2)

  write(*,*) 'a1: ',a

  a = 0._r8
  do i = 1, nsteps
     ! Do some simple math so the loop is ensured to be activated
     a = a + 1._r8/ real(i,r8)

     if(debug)then
        print*,"this will never be printed",a
     end if

  end do

  call system_clock(c3)

  write(*,*) 'a2: ',a
  write(*,*) 'dtime w/o if: ',(c2-c1)/real(cr,r8)
  write(*,*) 'dtime w   if: ',(c3-c2)/real(cr,r8)



end program TestIfConstants

I ran with -O3 and -O0, and you can see the wall-times are the same. Actually, the loop with the if clause was always faster, maybe because the variables were already moved to the CPU?

rgknox@>:~/SyncLRC/TestIfConstants$ gfortran -O3 TestIfConstants.f90
rgknox@>:~/SyncLRC/TestIfConstants$ ./a.out 
 a1:    21.300481502348550     
 a2:    21.300481502348550     
 dtime w/o if:    4.0970000000000004     
 dtime w   if:    4.0060000000000002     

rgknox@>:~/SyncLRC/TestIfConstants$ gfortran -O0 TestIfConstants.f90
rgknox@>:~/SyncLRC/TestIfConstants$ ./a.out 
 a1:    21.300481502348550     
 a2:    21.300481502348550     
 dtime w/o if:    4.0999999999999996     
 dtime w   if:    4.0069999999999997     

I am a little suspicious that the optimization is not improving things (it should allow faster math operations...), but in both situations, the extra if call to a constant "false" does not seem to have any impact.

Not that I also ran this through the DDD debugger. I set a break-point to just before the "if" call in question, and took a step. The debugger did not step to the logical clause and just went right bye. However, I just don't know enough about how debuggers work, and if they have their own layer of intelligence that would cause them to ignore an evaluation (but I would guess not). Either way, I thought I would mention this.

TLDR: These tests seem to indicate that using constant logicals like "debug" in our code, WILL be auto-removed by the compiler, and will not suffer performance penalties during run-time by evaluating the if clause ... at least with gfortran.

Does anyone see any flaws, or different ways to approach this?

[ambrad]

With ifort, you can run ifort -fsource-asm -S [other compiler flags] foo.F90 to get foo.s with inline source annotation of assembly. Then you can directly verify the dead code was compiled out.

[billsacks]

@rgknox This is good to know - thanks.

In case it's helpful, we do allow limited use of ifdefs in CTSM, specifically for the debug / non-debug case. git grep NDEBUG in latest CTSM master to see a few. This works if you want your debug logic to be tied to whether you're compiling in debug mode... it seems like that may not be the intent here.

FYI if you do this: I prefer #ifndef NDEBUG over #ifdef DEBUG for a few reasons:

• NDEBUG is what shr_assert.h checks, I think for consistency with C

• We've had problems in the past where DEBUG wasn't being defined, yet
  NDEBUG was

• If the build process breaks such that the given token isn't defined,
  I'd rather have logic that causes the error checks to happen in
  production mode rather than logic that causes the error checks to NOT
  happen in debug mode.

[ekluzek]

My feeling is that if you have a parameter in the file -- the compiler optimizer better do the removal -- because it's just so dead simple. If the compiler is crappy enough that it won't do the removal -- is it a compiler you'd even want to use? Now, if DEBUG is on and optimization is lowered -- maybe it won't, but I think that's OK, you already know it's slower because DEBUG is on. I know you can't trust compiler optimizers -- but for something this dead simple I think we can. And really we need to rely on optimizers, to help us make code that is more readable.

There are two concerns we have with CPP tokens and #ifdefs. One is readability, it's much easier to read and work with a code without them. The second is to save multiple compiles. CPP token's by nature require multiple compiles to compile the code for all the different CPP options. This means you have to save the different ones for each configuration, or have to recompile. This makes testing slower and more difficult. You still have to make sure both branches are exercised during run -- but you don't have to during compile. the initial compile will ensure that all branches will at least compile.

That said as @billsacks points out above, there are still cases where it makes sense to use CPP tokens and #ifdefs. So it's not that CPP tokens are completely forbidden, but we do want them to be limited and only used when it's the best solution.