E3sm: running with pgiacc option on Titan fails when using more than one OpenMP thread

[worleyph]

(This error was reported in the middle of an old github issue that I just now closed. Creating a new issue with this as the main topic and providing more information.)

On Titan when building with pgiacc, I am seeing two different runtime errors.

a) If running with only MPI and on all 16 cores, the jobs dies with

 call to cuMemAllocHost returned error 205: Map failed
 Failing in Thread:1

This is primarily an annoyance since we would not run in this fashion when using the GPU.

b) The serious error is that if run with more than 1 OpenMP thread, the job dies at runtime with the error

 malloc: cuMemHostAlloc returns error code 201
 0: ALLOCATE: 256 bytes requested; not enough memory

However, if run with only one thread and not on all 16 cores, the job finishes with no problems.

For example (--compset A_WCYCL1850S --res ne30_oECv3_ICG ):

 1) ATM 2700x1 on all 16 cores (-N 16 -S 8 -d 1): "Map failed"

 2) ATM 1350x2 on all 16 cores (-N 8 -S 4 -d 2): "not enough memory"
 3) ATM 1350x1 on 8 cores (-N 8 -S 4 -d 1): worked

 4) ATM 675x4 on all 16 cores (-N 4 -S 2 -d 4): "not enough memory"
 5) ATM 675x2 on 8 cores (-N 4 -S 2 -d 2): "not enough memory"
 6) ATM 675x2 on 4 cores (-N 2 -S 1 -d 2): "not enough memory"
 7) ATM 675x1 on 8 cores (-N 8 -S 4 -d 1): works
 8) ATM 675x1 on 4 cores (-N 4 -S 2 -d 1): works

The aprun layouts when there are idle cores (-d 1 for MPI-only and -d 2 for 2 threads) are simply what .case.run generates - I had no control. This should be irrelevant to this issue as the problem is with the GPU and not the node memory.

Note 1: the pgiacc option used to work for 2 or more OpenMP threads (and 4 OpenMP threads was suggested as the minimum that should be used for good GPU performance). So this may be easy to find and fix. The error has been there awhile so bisection might have to start aways back.

Note 2: There is an earlier error message:

 free: cuMemFreeHost returns error code 1

but this occurs in all cases, both working and nonworking.

Note 3: Running on Titan (with any compiler option) requires updates to the modules. This update is currently on next, but not yet on master.

[worleyph]

Also, BUILD_THREADED is TRUE so the compiler is building thread-aware even when there is only one OpenMP thread. For the successfull (MPI-only) runs, the timers indicate that the GPU is being used (as the OpenACC code has some distinct timer names).

[minxu74]

@worleyph As I know, PGIACC on Titan is only workable for E3SM with the MPI-only and maximum MPI task per numa node less than 4 (-S 4), otherwise, there will be a memory issue. @mrnorman may have some ideas about it. I will merge the PR of the module updates to master now. Thanks.

[worleyph]

@minxu74 , this did not used to be true, and so represents a bug. I do not believe that this is really a memory bug as using more than one thread even with only 2 MPI task per node results in the same "memory" error. I am assuming that some OpenMP logic modification broke pgiacc support, but you or @mrnorman will need to verify.

[worleyph]

@minxu74 , note that this comment is not on your PR but is a separate github issue. Please do go ahead and merge the PR with the module updates.

[minxu74]

@worleyph I agree with you that there is a bug. We need to take care of it. @mrnorman Please let me know if anything I can do to find the bug. Thanks.

[mrnorman]

I'm looking at this now.

I'm moving to pgi/17.10, and I'm adding the acc_set_device_num(0,acc_device_nvidia) routine right after horizontal OpenMP threads are launched in dyn_comp.F90. Hopefully this will resolve the issue.

[worleyph]

@mrnorman , sounds like you do not need any help, but, if you do, do not hesitate to ask. BTW - you don't see anything similar on Summit in your ECP work?

[worleyph]

@mrnorman , I went ahead and kept poking at this. I have narrowed it down (to the extent that write statements can be relied upon) to executing

    ! ================================
    ! Accumulate mean Vel_rho flux in vn0
    ! ================================
    elem(ie)%derived%vn0(:,:,:,k)=elem(ie)%derived%vn0(:,:,:,k)+eta_ave_w*vdp(:,:,:,k)

by thread 1 (when running with 2 threads) the first time it tries (ie=nets; k=1) in compute_and_apply_rhs in prim_advance_mod.F90.

Thread 1 never gets beyond this for any of the processes (675 in this case) and thread 1 on each process always gets to this point before the job dies.

Depending on how out of sync thread 0 and thread 1 are, thread 0 can get past this, and has no problems executing this statement all 72 times (for nlevs), or the rest of the loop, or commands after it. How far thread 0 gets varies from process to process (based on the output of write(6,*) ).

My question is what does this have to do with the GPU? There is no OpenACC in this routine?

[worleyph]

@mrnorman , it failed in the same way (only thread 0 can get beyond the indicated instruction) even when using 1 MPI process per node and not setting CRAY_CUDA_MPS. I also tried removing the CRAY_CPU_TARGET of istanbul, and this also had not effect. This is very reproducible - not that it seems to be helping any.

[worleyph]

@mrnorman , perhaps CRAY_CUDA_MPS.=1 is the default now? I'll try again with this set to 0. ... Made no difference.

[worleyph]

Correction: prim_advance_mod.F90 is built with

 prim_advance_mod.o: prim_advance_mod.F90
     $(FC) -c $(INCLDIR) $(INCS) $(FFLAGS) -DUSE_OPENACC=1 -acc -ta=tesla,pinned,cc35,cuda7.5,ptxinfo -Minfo=accel $(FREEFLAGS) $<

so I just don't know how the OpenACC works in the code.

[worleyph]

@mrnorman, compiling without

 prim_advance_mod.o: prim_advance_mod.F90
      $(FC) -c $(INCLDIR) $(INCS) $(FFLAGS) -DUSE_OPENACC=1 -acc -ta=tesla,pinned,cc35,cuda7.5,ptxinfo -Minfo=accel $(FREEFLAGS) $<

the error moved to a later location in the code (not identified with current debug writes but with identical error message). So does appear to be an OpenACC interaction with OpenMP.

[mrnorman]

@worleyph , you answer your first question, that line is using a data structure I flattened out in the OpenACC port into "derived_vn0" and then pointed to from elem(:)%derived%vn0. So, what could be happening is that perhaps the OpenMP is erroneously making that pointer threadprivate, and perhaps only the master thread ended up actually assigning the pointer? I don't fully buy that, but it's all I've got.

I do know that if I accidentally use the pointer form inside a kernel, sometimes it will give an invalid address error (GPU's version of segfault), but we're seeing a different error than that.

I think it probably has to do with PGI's pool allocator doing bad things. I'll try turning it off with "PGI_ACC_POOL_ALLOC=0". I'm going to try turning this off and see if it gets us past this error for now. I'm running that job now.

[mrnorman]

Nope, turning off the pool allocator didn't do anything. Since the error is cudaMallocHost (which is associated with pinned memory copies to and from the GPU), I'm now trying a new approach of taking off the ",pinned" attribute of the tesla flag. I'll have results in a couple hours.

This is just as well if it works, since I'm planning to migrate HOMME to use managed / unified memory this week because it's significantly more robust in giving the right answer.

[worleyph]

@mrnorman , I was still hoping that this was a code bug, with something in OpenMP logic changing that broke OpenACC. However, I just tried the PR that you merged in September 2016:

 Cuda moved to 7.5; OpenACC flags applied in Depends, not Macros #1072

After updating to the latest modules, it failed in exactly the same when using more than one thread:

 0: ALLOCATE: 256 bytes requested; not enough memory
 malloc: cuMemHostAlloc returns error code 201

so a compiler bug. (bummer)

[mrnorman]

Yep, I just confirmed it's a compiler bug in the pinned memory. Hopefully managed memory doesn't follow suit. I'll push a branch and reference this issue.

[worleyph]

@mrnorman , I'm trying your branch now. Note that I tried PR#1072 again, but with pig/15.7.0 . This failed in the same way, so the bug is in some common back end. FYI.