Climatemachine.jl: Call to `cuIpcCloseMemHandle` failed

[kpamnany]

Description

Long-running experiments such as OceanBoxGCM/homogeneous_box.jl and test/DGmethods/advection_diffusion/pseudo1D_advection_diffusion.jl are frequently failing due to a segmentation fault at the end of the run. The error looks like:

--------------------------------------------------------------------------
The call to cuIpcCloseMemHandle failed. This is a warning and the program
will continue to run.
  cuIpcCloseMemHandle return value:   1
  address: 0x2aab81fe6000
Check the cuda.h file for what the return value means. Perhaps a reboot
of the node will clear the problem.
--------------------------------------------------------------------------

signal (11): Segmentation fault
in expression starting at none:0
opal_free_list_destruct at /central/software/OpenMPI/4.0.1_cuda-10.0/lib/libopen-pal.so.40 (unknown line)
mca_rcache_rgpusm_finalize at /central/software/OpenMPI/4.0.1_cuda-10.0/lib/openmpi/mca_rcache_rgpusm.so (unknown line)
mca_rcache_base_module_destroy at /central/software/OpenMPI/4.0.1_cuda-10.0/lib/libopen-pal.so.40 (unknown line)
mca_btl_smcuda_del_procs at /central/software/OpenMPI/4.0.1_cuda-10.0/lib/openmpi/mca_btl_smcuda.so (unknown line)
mca_bml_r2_del_procs at /central/software/OpenMPI/4.0.1_cuda-10.0/lib/openmpi/mca_bml_r2.so (unknown line)
ompi_mpi_finalize at /central/software/OpenMPI/4.0.1_cuda-10.0//lib/libmpi.so (unknown line)
_Finalize at /central/scratchio/esm/slurmci/sources/d166aa0ad43803cc7cea9a4038e3f16808783885/.slurmdepot/gpu/packages/MPI/O0dMQ/src/environment.jl:206 [inlined]
refcount_dec at /central/scratchio/esm/slurmci/sources/d166aa0ad43803cc7cea9a4038e3f16808783885/.slurmdepot/gpu/packages/MPI/O0dMQ/src/environment.jl:27 [inlined]
free at /central/scratchio/esm/slurmci/sources/d166aa0ad43803cc7cea9a4038e3f16808783885/.slurmdepot/gpu/packages/MPI/O0dMQ/src/comm.jl:30
_jl_invoke at /central/groups/esm/common/julia-1.3/src/gf.c:2135 [inlined]
jl_apply_generic at /central/groups/esm/common/julia-1.3/src/gf.c:2305
jl_apply at /central/groups/esm/common/julia-1.3/src/julia.h:1631 [inlined]
run_finalizer at /central/groups/esm/common/julia-1.3/src/gc.c:231
jl_gc_run_finalizers_in_list at /central/groups/esm/common/julia-1.3/src/gc.c:317
run_finalizers at /central/groups/esm/common/julia-1.3/src/gc.c:345
jl_atexit_hook at /central/groups/esm/common/julia-1.3/src/init.c:257
unknown function (ip: 0x401344)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x4013d3)
Allocations: 1599881255 (Pool: 1599704151; Big: 177104); GC: 1077

Steps to reproduce

See:

https://gist.github.com/climabot/45fd52357cf0f13835ba33aaf2623a0c#file-out_9616081
https://gist.github.com/climabot/b8588a0fe9f965308af0a8b4070b9237#file-out_9636351
https://gist.github.com/climabot/7a4cc4614ba8ac085b7d1d033a6ead3f#file-out_9636108
https://gist.github.com/climabot/6171f6ade1b8a482c6b1a9e06cc4d6ae#file-out_9636864

etc.

For CLIMA developers

• [ ] The bug can be reproduced
• [ ] Labels have been added and developers assigned
• [ ] The bug can be fixed (if not, please explain why not in a comment below)
• [ ] There is a timeline to fix the bug
• [ ] The bug has been fixed (Please link the PR)

[kpamnany]

Cc: @vchuravy in case you have any ideas/suggestions ;)

[blallen]

@kpamnany, do we have command line args for experiments working? that's by far the easiest way to split this out into separate tests

[kpamnany]

958 will add this capability, but it's not ready yet. I'll hopefully merge it this week though.

[blallen]

Should I push a quick fix before then or can we wait until I can fix it properly with #958? Quick fix will just reduce the number of iterations for now, instead of splitting into different files.

[mwarusz]

Got the same error in the pseudo1D_advection_diffusion.jl test which calls CLIMA.init() only once

https://gist.github.com/climabot/6171f6ade1b8a482c6b1a9e06cc4d6ae#file-out_9636864

[kpamnany]

Updated the issue.

This might have something to do with https://github.com/open-mpi/ompi/issues/6648 and can maybe be solved by upgrading to OpenMPI 4.0.2. I'll follow up on this.

[charleskawczynski]

Also
https://gist.github.com/climabot/2aa8c4bb41631530f5da5509428d7e0b#file-out_9637567-L1114

[leios]

@lcw mentioned in #923 that the adding --mca btl_smcuda_use_cuda_ipc 0 to mpirun will disables using cuda-aware via PCI, which could solve the issue. I'll give that a go for #896 and see if that solves the issue temporarily.

EDIT: nevermind, this breaks tests locally.

[akshaysridhar]

Is this related - https://gist.github.com/akshaysridhar/f6695c112f001f18100f536779d2e4f3 ? (slurmci log from the #972 run)

[kpamnany]

Closing this as it looks like the upgrade to OpenMPI/4.0.3 has fixed the problem.