Climatemachine.jl: code breaking on multi GPUs

Created on 12 Jul 2019 · 12Comments · Source: CliMA/ClimateMachine.jl

The following code revision breaks within a few time steps on multi GPUs.
The code works correctly on 1 GPU only.

The error on multi GPU seems CUDA related but it is not completely clear to me.

Info: Starting... └ norm(Q) = 5.5625668847868290e+09 ┌ Info: Update │ simtime = 2.5000000000000001e-03 └ runtime = 00:00:37 ┌ Info: Update │ simtime = 2.4999999999999998e-02 └ runtime = 00:00:48 ┌ Info: Update │ simtime = 5.0000000000000010e-02 └ runtime = 00:00:58 ┌ Info: Update │ simtime = 7.5000000000000025e-02 └ runtime = 00:01:09 ┌ Info: Update │ simtime = 1.0000000000000005e-01 └ runtime = 00:01:20 ERROR: a exception was thrown during kernel execution. Run Julia on debug level 2 for device stack traces. ERROR: LoadError: CUDA error: unspecified launch failure (code #719, ERROR_LAUNCH_FAILED) Stacktrace: [1] macro expansion at /home/smarras/.julia/packages/CUDAdrv/WVU1H/src/base.jl:147 [inlined] [2] #copy!#10(::Nothing, ::Bool, ::typeof(CUDAdrv.Mem.copy!), ::Ptr{Float64}, ::CUDAdrv.Mem.DeviceBuffer, ::Int64) at /home/smarras/.julia/packages/CUDAdrv/WVU1H/src/memory.jl:344 [3] copy! at /home/smarras/.julia/packages/CUDAdrv/WVU1H/src/memory.jl:335 [inlined] [4] copyto!(::Array{Float64,3}, ::Int64, ::CuArray{Float64,3}, ::Int64, ::Int64) at /home/smarras/.julia/packages/CuArrays/ibLSJ/src/array.jl:194 [5] fillsendbuf!(::Array{Float64,3}, ::CuArray{Float64,3}, ::CuArray{Float64,3}, ::CuArray{Int64,1}, ::Tuple{Colon,Colon}) at /home/smarras/.julia/packages/GPUArrays/fAX0Q/src/abstractarray.jl:101 [6] #start_ghost_exchange!#14(::Bool, ::typeof(CLIMA.MPIStateArrays.start_ghost_exchange!), ::MPIStateArray{Tuple{125,6},Float64,CuArray,3,CuArray{Float64,3},2,CuArray{Int64,1},CuArray{Float64,3}}, ::Tuple{Colon,Colon}) at /central/home/smarras/smarras79/CLIMA/src/Arrays/MPIStateArrays.jl:256 [7] start_ghost_exchange! at /central/home/smarras/smarras79/CLIMA/src/Arrays/MPIStateArrays.jl:250 [inlined] (repeats 2 times) [8] #odefun!#13(::Bool, ::typeof(CLIMA.SpaceMethods.odefun!), ::DGBalanceLaw, ::MPIStateArray{Tuple{125,6},Float64,CuArray,3,CuArray{Float64,3},2,CuArray{Int64,1},CuArray{Float64,3}}, ::MPIStateArray{Tuple{125,6},Float64,CuArray,3,CuArray{Float64,3},2,CuArray{Int64,1},CuArray{Float64,3}}, ::Float64) at /central/home/smarras/smarras79/CLIMA/src/DGmethods/DGBalanceLawDiscretizations.jl:599 [9] (::getfield(CLIMA.SpaceMethods, Symbol("#kw##odefun!")))(::NamedTuple{(:increment,),Tuple{Bool}}, ::typeof(CLIMA.SpaceMethods.odefun!), ::DGBalanceLaw, ::MPIStateArray{Tuple{125,6},Float64,CuArray,3,CuArray{Float64,3},2,CuArray{Int64,1},CuArray{Float64,3}}, ::MPIStateArray{Tuple{125,6},Float64,CuArray,3,CuArray{Float64,3},2,CuArray{Int64,1},CuArray{Float64,3}}, ::Float64) at ./none:0 [10] (::getfield(CLIMA.LowStorageRungeKuttaMethod, Symbol("###12#13#15")){DGBalanceLaw})(::Bool, ::getfield(CLIMA.LowStorageRungeKuttaMethod, Symbol("##12#14")){getfield(CLIMA.LowStorageRungeKuttaMethod, Symbol("###12#13#15")){DGBalanceLaw}}, ::MPIStateArray{Tuple{125,6},Float64,CuArray,3,CuArray{Float64,3},2,CuArray{Int64,1},CuArray{Float64,3}}, ::Vararg{Any,N} where N) at /central/home/smarras/smarras79/CLIMA/src/ODESolvers/LowStorageRungeKuttaMethod.jl:68 [11] (::getfield(CLIMA.LowStorageRungeKuttaMethod, Symbol("#kw###12#14")))(::NamedTuple{(:increment,),Tuple{Bool}}, ::getfield(CLIMA.LowStorageRungeKuttaMethod, Symbol("##12#14")){getfield(CLIMA.LowStorageRungeKuttaMethod, Symbol("###12#13#15")){DGBalanceLaw}}, ::MPIStateArray{Tuple{125,6},Float64,CuArray,3,CuArray{Float64,3},2,CuArray{Int64,1},CuArray{Float64,3}}, ::Vararg{Any,N} where N) at ./none:0 [12] dostep!(::MPIStateArray{Tuple{125,6},Float64,CuArray,3,CuArray{Float64,3},2,CuArray{Int64,1},CuArray{Float64,3}}, ::LowStorageRungeKutta2N{Float64,Float64,MPIStateArray{Tuple{125,6},Float64,CuArray,3,CuArray{Float64,3},2,CuArray{Int64,1},CuArray{Float64,3}},5}, ::Int64, ::Bool) at /central/home/smarras/smarras79/CLIMA/src/ODESolvers/LowStorageRungeKuttaMethod.jl:91 [13] #solve!#9(::Int64, ::Bool, ::Int64, ::Tuple{CLIMA.GenericCallbacks.EveryXWallTimeSeconds,CLIMA.GenericCallbacks.EveryXSimulationSteps}, ::typeof(solve!), ::MPIStateArray{Tuple{125,6},Float64,CuArray,3,CuArray{Float64,3},2,CuArray{Int64,1},CuArray{Float64,3}}, ::LowStorageRungeKutta2N{Float64,Float64,MPIStateArray{Tuple{125,6},Float64,CuArray,3,CuArray{Float64,3},2,CuArray{Int64,1},CuArray{Float64,3}},5}) at /central/home/smarras/smarras79/CLIMA/src/ODESolvers/ODESolvers.jl:70 [14] (::getfield(CLIMA.ODESolvers, Symbol("#kw##solve!")))(::NamedTuple{(:timeend, :callbacks),Tuple{Int64,Tuple{CLIMA.GenericCallbacks.EveryXWallTimeSeconds,CLIMA.GenericCallbacks.EveryXSimulationSteps}}}, ::typeof(solve!), ::MPIStateArray{Tuple{125,6},Float64,CuArray,3,CuArray{Float64,3},2,CuArray{Int64,1},CuArray{Float64,3}}, ::LowStorageRungeKutta2N{Float64,Float64,MPIStateArray{Tuple{125,6},Float64,CuArray,3,CuArray{Float64,3},2,CuArray{Int64,1},CuArray{Float64,3}},5}) at ./none:0 [15] macro expansion at /home/smarras/.julia/packages/TimerOutputs/7zSea/src/TimerOutput.jl:216 [inlined] [16] run(::MPI.Comm, ::Int64, ::Tuple{Int64,Int64,Int64}, ::Int64, ::Int64, ::Type, ::Float64) at /central/home/smarras/smarras79/CLIMA/test/DGmethods/compressible_Navier_Stokes/dycoms3d-reference.jl:705 [17] top-level scope at /central/home/smarras/smarras79/CLIMA/test/DGmethods/compressible_Navier_Stokes/dycoms3d-reference.jl:760 [18] include at ./boot.jl:328 [inlined] [19] include_relative(::Module, ::String) at ./loading.jl:1094 [20] include(::Module, ::String) at ./Base.jl:31 [21] exec_options(::Base.JLOptions) at ./client.jl:295 [22] _start() at ./client.jl:464 in expression starting at /central/home/smarras/smarras79/CLIMA/test/DGmethods/compressible_Navier_Stokes/dycoms3d-reference.jl:714
The code I am running is the following:

https://github.com/smarras79/CLIMA/blob/sm79/dycoms3d-reference/test/DGmethods/compressible_Navier_Stokes/dycoms3d-reference.jl

I tried several things and so did @asraero for the past few days, but nothing solved it.

Source

smarras79

Most helpful comment

I would suggest trying to run with bounds checking turned on for the CPU on a single processor

julia --project=@. --check-bounds=yes test/DGmethods/compressible_Navier_Stokes/dycoms3d-reference.jl

jkozdon on 12 Jul 2019

👍2

All 12 comments

I would suggest trying to run with bounds checking turned on for the CPU on a single processor

julia --project=@. --check-bounds=yes test/DGmethods/compressible_Navier_Stokes/dycoms3d-reference.jl

jkozdon on 12 Jul 2019

👍2

If that's fine then try the same things with 2 mpiranks (but still on the CPU)

If there are no errors after this then start looking at multi-GPU

jkozdon on 12 Jul 2019

👍1

After adding TimerOutputs to the Project.toml

julia --project=@.
]add TimerOutputs

Running the code with bounds checking gives:

❯ julia --project=@. --check-bounds=yes test/DGmethods/compressible_Navier_Stokes/dycoms3d-reference.jl

[ Info:  ------------------------------------------------------
[ Info:    ______ _      _____ __  ________
[ Info:   |  ____| |    |_   _|  ...  |  __  |
[ Info:   | |    | |      | | |   .   | |  | |
[ Info:   | |    | |      | | | |   | | |__| |
[ Info:   | |____| |____ _| |_| |   | | |  | |
[ Info:   | _____|______|_____|_|   |_|_|  |_|
[ Info:
[ Info:  ------------------------------------------------------
[ Info:  Dycoms
[ Info:    Resolution:
[ Info:      (Δx, Δy, Δz)   = (3.50e+01, 3.50e+01, 5.00e+00)
[ Info:      (Nex, Ney, Nez) = (28, 28, 75)
[ Info:      DoF = 44100000
[ Info:      Minimum necessary memory to run this test: 2.9988 GBs
[ Info:      Time step dt: 2.50e-03
[ Info:      End time  t : 1.44e+04
[ Info:  ------------------------------------------------------
┌ Info: Starting...
└ norm(Q) =    5.5625795509266300e+09
ERROR: LoadError: BoundsError: attempt to access 3×3 MArray{Tuple{3,3},Float64,2,9} with indices SOneTo(3)×SOneTo(3) at index [1, 5]
Stacktrace:
 [1] throw_boundserror(::MArray{Tuple{3,3},Float64,2,9}, ::Tuple{Int64,Int64}) at ./abstractarray.jl:538
 [2] checkbounds at ./abstractarray.jl:503 [inlined]
 [3] getindex at /home/jekozdon/.julia/packages/StaticArrays/3KEjZ/src/indexing.jl:12 [inlined]
 [4] compute_stresses! at /home/jekozdon/codes/CLIMA_triage/test/DGmethods/compressible_Navier_Stokes/dycoms3d-reference.jl:305 [inlined]
 [5] stresses_penalty! at /home/jekozdon/codes/CLIMA_triage/test/DGmethods/compressible_Navier_Stokes/dycoms3d-reference.jl:445 [inlined]
 [6] faceviscterms!(::Val{3}, ::Val{4}, ::Val{6}, ::Val{(1, 2, 3, 4, 5, 6)}, ::Val{6}, ::Val{13}, ::Val{9}, ::typeof(stresses_penalty!), ::typeof(stresses_boundary_penalty!), ::typeof(gradient_vars!), ::Array{Float64,3}, ::Array{Float64,3}, ::Array{Float64,3}, ::Array{Float64,3}, ::Array{Float64,4}, ::Float64, ::Array{Int64,3}, ::Array{Int64,3}, ::Array{Int64,2}, ::UnitRange{Int64}) at /home/jekozdon/codes/CLIMA_triage/src/DGmethods/DGBalanceLawDiscretizations_kernels.jl:482
 [7] #launch#18(::Base.Iterators.Pairs{Symbol,Int64,Tuple{Symbol,Symbol},NamedTuple{(:threads, :blocks),Tuple{Int64,Int64}}}, ::typeof(GPUifyLoops.launch), ::GPUifyLoops.CPU, ::Function, ::Val{3}, ::Vararg{Any,N} where N) at /home/jekozdon/.julia/packages/GPUifyLoops/ZKqaZ/src/GPUifyLoops.jl:67
 [8] (::getfield(GPUifyLoops, Symbol("#kw##launch")))(::NamedTuple{(:threads, :blocks),Tuple{Int64,Int64}}, ::typeof(GPUifyLoops.launch), ::GPUifyLoops.CPU, ::Function, ::Val{3}, ::Vararg{Any,N} where N) at ./none:0
 [9] macro expansion at /home/jekozdon/.julia/packages/GPUifyLoops/ZKqaZ/src/GPUifyLoops.jl:54 [inlined]
 [10] #odefun!#13(::Bool, ::typeof(CLIMA.SpaceMethods.odefun!), ::DGBalanceLaw, ::MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}}, ::MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}}, ::Float64) at /home/jekozdon/codes/CLIMA_triage/src/DGmethods/DGBalanceLawDiscretizations.jl:612
 [11] (::getfield(CLIMA.SpaceMethods, Symbol("#kw##odefun!")))(::NamedTuple{(:increment,),Tuple{Bool}}, ::typeof(CLIMA.SpaceMethods.odefun!), ::DGBalanceLaw, ::MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}}, ::MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}}, ::Float64) at ./none:0
 [12] (::getfield(CLIMA.LowStorageRungeKuttaMethod, Symbol("###12#13#15")){DGBalanceLaw})(::Bool, ::getfield(CLIMA.LowStorageRungeKuttaMethod, Symbol("##12#14")){getfield(CLIMA.LowStorageRungeKuttaMethod, Symbol("###12#13#15")){DGBalanceLaw}}, ::MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}}, ::Vararg{Any,N} where N) at /home/jekozdon/codes/CLIMA_triage/src/ODESolvers/LowStorageRungeKuttaMethod.jl:68
 [13] (::getfield(CLIMA.LowStorageRungeKuttaMethod, Symbol("#kw###12#14")))(::NamedTuple{(:increment,),Tuple{Bool}}, ::getfield(CLIMA.LowStorageRungeKuttaMethod, Symbol("##12#14")){getfield(CLIMA.LowStorageRungeKuttaMethod, Symbol("###12#13#15")){DGBalanceLaw}}, ::MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}}, ::Vararg{Any,N} where N) at ./none:0
 [14] dostep!(::MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}}, ::LowStorageRungeKutta2N{Float64,Float64,MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}},5}, ::Int64, ::Bool) at /home/jekozdon/codes/CLIMA_triage/src/ODESolvers/LowStorageRungeKuttaMethod.jl:91
 [15] #solve!#9(::Int64, ::Bool, ::Int64, ::Tuple{CLIMA.GenericCallbacks.EveryXWallTimeSeconds,CLIMA.GenericCallbacks.EveryXSimulationSteps}, ::typeof(solve!), ::MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}}, ::LowStorageRungeKutta2N{Float64,Float64,MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}},5}) at /home/jekozdon/codes/CLIMA_triage/src/ODESolvers/ODESolvers.jl:70
 [16] (::getfield(CLIMA.ODESolvers, Symbol("#kw##solve!")))(::NamedTuple{(:timeend, :callbacks),Tuple{Int64,Tuple{CLIMA.GenericCallbacks.EveryXWallTimeSeconds,CLIMA.GenericCallbacks.EveryXSimulationSteps}}}, ::typeof(solve!), ::MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}}, ::LowStorageRungeKutta2N{Float64,Float64,MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}},5}) at ./none:0
 [17] macro expansion at /home/jekozdon/.julia/packages/TimerOutputs/7zSea/src/TimerOutput.jl:216 [inlined]
 [18] run(::MPI.Comm, ::Int64, ::Tuple{Int64,Int64,Int64}, ::Int64, ::Int64, ::Type, ::Float64) at /home/jekozdon/codes/CLIMA_triage/test/DGmethods/compressible_Navier_Stokes/dycoms3d-reference.jl:705
 [19] top-level scope at /home/jekozdon/codes/CLIMA_triage/test/DGmethods/compressible_Navier_Stokes/dycoms3d-reference.jl:760
 [20] include at ./boot.jl:328 [inlined]
 [21] include_relative(::Module, ::String) at ./loading.jl:1094
 [22] include(::Module, ::String) at ./Base.jl:31
 [23] exec_options(::Base.JLOptions) at ./client.jl:295
 [24] _start() at ./client.jl:464
in expression starting at /home/jekozdon/codes/CLIMA_triage/test/DGmethods/compressible_Navier_Stokes/dycoms3d-reference.jl:714

jkozdon on 12 Jul 2019

Another issue is that we do not sync the aux state by default ever. So modifying it in the preodefun will cause problems on multiple GPUs and CPUS (it's just luck I think that this didn't cause a problem on the CPU).

Two solutions are to sync aux after this or to sync Q before this. Not sure what the right approach is. In both cases the balance law solver will need to be modified if you want to do things efficiently (e.g., overlapping communication and computation).

jkozdon on 12 Jul 2019

Can we avoid @inbounds until we know that the code works properly and we are ready to optimize it?

smarras79 on 12 Jul 2019

After adding TimerOutputs to the Project.toml

julia --project=@.
]add TimerOutputs

Running the code with bounds checking gives:

❯ julia --project=@. --check-bounds=yes test/DGmethods/compressible_Navier_Stokes/dycoms3d-reference.jl

[ Info:  ------------------------------------------------------
[ Info:    ______ _      _____ __  ________
[ Info:   |  ____| |    |_   _|  ...  |  __  |
[ Info:   | |    | |      | | |   .   | |  | |
[ Info:   | |    | |      | | | |   | | |__| |
[ Info:   | |____| |____ _| |_| |   | | |  | |
[ Info:   | _____|______|_____|_|   |_|_|  |_|
[ Info:
[ Info:  ------------------------------------------------------
[ Info:  Dycoms
[ Info:    Resolution:
[ Info:      (Δx, Δy, Δz)   = (3.50e+01, 3.50e+01, 5.00e+00)
[ Info:      (Nex, Ney, Nez) = (28, 28, 75)
[ Info:      DoF = 44100000
[ Info:      Minimum necessary memory to run this test: 2.9988 GBs
[ Info:      Time step dt: 2.50e-03
[ Info:      End time  t : 1.44e+04
[ Info:  ------------------------------------------------------
┌ Info: Starting...
└ norm(Q) =    5.5625795509266300e+09
ERROR: LoadError: BoundsError: attempt to access 3×3 MArray{Tuple{3,3},Float64,2,9} with indices SOneTo(3)×SOneTo(3) at index [1, 5]
Stacktrace:
 [1] throw_boundserror(::MArray{Tuple{3,3},Float64,2,9}, ::Tuple{Int64,Int64}) at ./abstractarray.jl:538
 [2] checkbounds at ./abstractarray.jl:503 [inlined]
 [3] getindex at /home/jekozdon/.julia/packages/StaticArrays/3KEjZ/src/indexing.jl:12 [inlined]
 [4] compute_stresses! at /home/jekozdon/codes/CLIMA_triage/test/DGmethods/compressible_Navier_Stokes/dycoms3d-reference.jl:305 [inlined]
 [5] stresses_penalty! at /home/jekozdon/codes/CLIMA_triage/test/DGmethods/compressible_Navier_Stokes/dycoms3d-reference.jl:445 [inlined]
 [6] faceviscterms!(::Val{3}, ::Val{4}, ::Val{6}, ::Val{(1, 2, 3, 4, 5, 6)}, ::Val{6}, ::Val{13}, ::Val{9}, ::typeof(stresses_penalty!), ::typeof(stresses_boundary_penalty!), ::typeof(gradient_vars!), ::Array{Float64,3}, ::Array{Float64,3}, ::Array{Float64,3}, ::Array{Float64,3}, ::Array{Float64,4}, ::Float64, ::Array{Int64,3}, ::Array{Int64,3}, ::Array{Int64,2}, ::UnitRange{Int64}) at /home/jekozdon/codes/CLIMA_triage/src/DGmethods/DGBalanceLawDiscretizations_kernels.jl:482
 [7] #launch#18(::Base.Iterators.Pairs{Symbol,Int64,Tuple{Symbol,Symbol},NamedTuple{(:threads, :blocks),Tuple{Int64,Int64}}}, ::typeof(GPUifyLoops.launch), ::GPUifyLoops.CPU, ::Function, ::Val{3}, ::Vararg{Any,N} where N) at /home/jekozdon/.julia/packages/GPUifyLoops/ZKqaZ/src/GPUifyLoops.jl:67
 [8] (::getfield(GPUifyLoops, Symbol("#kw##launch")))(::NamedTuple{(:threads, :blocks),Tuple{Int64,Int64}}, ::typeof(GPUifyLoops.launch), ::GPUifyLoops.CPU, ::Function, ::Val{3}, ::Vararg{Any,N} where N) at ./none:0
 [9] macro expansion at /home/jekozdon/.julia/packages/GPUifyLoops/ZKqaZ/src/GPUifyLoops.jl:54 [inlined]
 [10] #odefun!#13(::Bool, ::typeof(CLIMA.SpaceMethods.odefun!), ::DGBalanceLaw, ::MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}}, ::MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}}, ::Float64) at /home/jekozdon/codes/CLIMA_triage/src/DGmethods/DGBalanceLawDiscretizations.jl:612
 [11] (::getfield(CLIMA.SpaceMethods, Symbol("#kw##odefun!")))(::NamedTuple{(:increment,),Tuple{Bool}}, ::typeof(CLIMA.SpaceMethods.odefun!), ::DGBalanceLaw, ::MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}}, ::MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}}, ::Float64) at ./none:0
 [12] (::getfield(CLIMA.LowStorageRungeKuttaMethod, Symbol("###12#13#15")){DGBalanceLaw})(::Bool, ::getfield(CLIMA.LowStorageRungeKuttaMethod, Symbol("##12#14")){getfield(CLIMA.LowStorageRungeKuttaMethod, Symbol("###12#13#15")){DGBalanceLaw}}, ::MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}}, ::Vararg{Any,N} where N) at /home/jekozdon/codes/CLIMA_triage/src/ODESolvers/LowStorageRungeKuttaMethod.jl:68
 [13] (::getfield(CLIMA.LowStorageRungeKuttaMethod, Symbol("#kw###12#14")))(::NamedTuple{(:increment,),Tuple{Bool}}, ::getfield(CLIMA.LowStorageRungeKuttaMethod, Symbol("##12#14")){getfield(CLIMA.LowStorageRungeKuttaMethod, Symbol("###12#13#15")){DGBalanceLaw}}, ::MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}}, ::Vararg{Any,N} where N) at ./none:0
 [14] dostep!(::MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}}, ::LowStorageRungeKutta2N{Float64,Float64,MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}},5}, ::Int64, ::Bool) at /home/jekozdon/codes/CLIMA_triage/src/ODESolvers/LowStorageRungeKuttaMethod.jl:91
 [15] #solve!#9(::Int64, ::Bool, ::Int64, ::Tuple{CLIMA.GenericCallbacks.EveryXWallTimeSeconds,CLIMA.GenericCallbacks.EveryXSimulationSteps}, ::typeof(solve!), ::MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}}, ::LowStorageRungeKutta2N{Float64,Float64,MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}},5}) at /home/jekozdon/codes/CLIMA_triage/src/ODESolvers/ODESolvers.jl:70
 [16] (::getfield(CLIMA.ODESolvers, Symbol("#kw##solve!")))(::NamedTuple{(:timeend, :callbacks),Tuple{Int64,Tuple{CLIMA.GenericCallbacks.EveryXWallTimeSeconds,CLIMA.GenericCallbacks.EveryXSimulationSteps}}}, ::typeof(solve!), ::MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}}, ::LowStorageRungeKutta2N{Float64,Float64,MPIStateArray{Tuple{125,6},Float64,Array,3,Array{Float64,3},2,Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true}},5}) at ./none:0
 [17] macro expansion at /home/jekozdon/.julia/packages/TimerOutputs/7zSea/src/TimerOutput.jl:216 [inlined]
 [18] run(::MPI.Comm, ::Int64, ::Tuple{Int64,Int64,Int64}, ::Int64, ::Int64, ::Type, ::Float64) at /home/jekozdon/codes/CLIMA_triage/test/DGmethods/compressible_Navier_Stokes/dycoms3d-reference.jl:705
 [19] top-level scope at /home/jekozdon/codes/CLIMA_triage/test/DGmethods/compressible_Navier_Stokes/dycoms3d-reference.jl:760
 [20] include at ./boot.jl:328 [inlined]
 [21] include_relative(::Module, ::String) at ./loading.jl:1094
 [22] include(::Module, ::String) at ./Base.jl:31
 [23] exec_options(::Base.JLOptions) at ./client.jl:295
 [24] _start() at ./client.jl:464
in expression starting at /home/jekozdon/codes/CLIMA_triage/test/DGmethods/compressible_Navier_Stokes/dycoms3d-reference.jl:714

The check-bounds in fact discovered a memory issue in an unallocated array.
Can we avoid @inbounds until we know that the code works properly and we are ready to optimize it? Without @inbounds we can detect memory errors on arrays earlier on in the development process.

smarras79 on 12 Jul 2019

You just need to use the julia flag --check-bounds=yes and this will ignore all the @inbounds in the code.

Your code will still have problems with multiple mpiranks due to the way preodefun is being used and the fact that auxstate is not synchronized after updating.

The poor man's approach to fixing this (to see if this helps it to run) is to add:

MPIStateArrays.start_ghost_exchange!(disc.auxstate)
MPIStateArrays.finish_ghost_exchange!(disc.auxstate)

after the dof_iteration!.

Long term this is not a solution, but its a work around that works now.

If you do this put a big FIXME or TODO there saying that we are not overlapping this communication with any computation.

jkozdon on 12 Jul 2019

You just need to use the julia flag --check-bounds=yes and this will ignore all the @inbounds in the code.

Your code will still have problems with multiple mpiranks due to the way preodefun is being used and the fact that auxstate is not synchronized after updating.

The poor man's approach to fixing this (to see if this helps it to run) is to add:
MPIStateArrays.start_ghost_exchange!(disc.auxstate)
MPIStateArrays.finish_ghost_exchange!(disc.auxstate)
after the dof_iteration!.

Long term this is not a solution, but its a work around that works now.

If you do this put a big FIXME or TODO there saying that we are not overlapping this communication with any computation.

should this also be added after dof_iteration! at output?

smarras79 on 13 Jul 2019

No after the preodefun!

jkozdon on 13 Jul 2019

Do you have a working benchmark written for multi-GPU to write mine from?
I imagine this will be an issue also with @trontrytel squall line and, possibly, more drivers hanging around the repo.

smarras79 on 13 Jul 2019

Do you have a working benchmark written for multi-GPU to write mine from?

Not that modifies the aux stat in the way your driver does.

jkozdon on 13 Jul 2019

I believe this is now obsolete.

smarras79 on 14 Oct 2019

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Call to `cuIpcCloseMemHandle` failed

kpamnany · 10Comments

implement implicit solvers

kmdeck · 8Comments

Possible issue with climaformat.jl after Merge 1172

smarras79 · 5Comments

Flatten directory structure

jkozdon · 9Comments

Simple interface to apply RHS of a balance law to state

bischtob · 4Comments