Climatemachine.jl: KernelAbstractions made stack-traces opaque

[charleskawczynski]

Description

KernelAbstractions has made the stack traces a bit more difficult to trace. For example, I'm debugging something on a branch right now and the stack trace is:

  MethodError: no method matching linearized_pressure(::DryModel, ::EarthParameterSet, ::NoOrientation, ::Vars{NamedTuple{(:ρ, :ρu, :ρe, :turbulence, :hyperdiffusion, :moisture, :radiation),Tuple{Float64,SArray{Tuple{3},Float64,1,3},Float64,NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}}}},MArray{Tuple{5},Float64,1,5},0}, ::Vars{NamedTuple{(:∫dz, :∫dnz, :coord, :orientation, :ref_state, :turbulence, :hyperdiffusion, :moisture, :radiation),Tuple{NamedTuple{(:radiation,),Tuple{NamedTuple{(),Tuple{}}}},NamedTuple{(:radiation,),Tuple{NamedTuple{(),Tuple{}}}},SArray{Tuple{3},Float64,1,3},NamedTuple{(),Tuple{}},NamedTuple{(:ρ, :ρe, :p, :T),NTuple{4,Float64}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(:θ_v, :air_T),Tuple{Float64,Float64}},NamedTuple{(),Tuple{}}}},MArray{Tuple{9},Float64,1,9},0})
  Closest candidates are:
    linearized_pressure(::DryModel, !Matched::CLIMA.Parameters.AbstractParameterSet, ::CLIMA.Atmos.Orientation, ::Vars, ::Vars) at /Users/charliekawczynski/Dropbox/Caltech/work/dev/ClimateMachine/CLIMA/src/Atmos/Model/linear.jl:43
    linearized_pressure(!Matched::CLIMA.Atmos.EquilMoist, !Matched::CLIMA.Parameters.AbstractParameterSet, ::CLIMA.Atmos.Orientation, ::Vars, ::Vars) at /Users/charliekawczynski/Dropbox/Caltech/work/dev/ClimateMachine/CLIMA/src/Atmos/Model/linear.jl:53
  Stacktrace:
   [1] macro expansion at /Users/charliekawczynski/.julia/packages/Cassette/7OymZ/src/context.jl:450 [inlined]
   [2] cpu_volumerhs! at /Users/charliekawczynski/Dropbox/Caltech/work/dev/ClimateMachine/CLIMA/src/DGmethods/DGmodel_kernels.jl:230 [inlined]
   [3] __thread_run(::Int64, ::Int64, ::Int64, ::KernelAbstractions.Kernel{KernelAbstractions.CPU,KernelAbstractions.NDIteration.StaticSize{(5, 5, 1)},KernelAbstractions.NDIteration.DynamicSize,typeof(CLIMA.DGmethods.cpu_volumerhs!)}, ::Tuple{Int64,Int64,Int64}, ::KernelAbstractions.NDIteration.NDRange{3,KernelAbstractions.NDIteration.DynamicSize,KernelAbstractions.NDIteration.StaticSize{(5, 5, 1)},CartesianIndices{3,Tuple{Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}},Nothing}, ::Tuple{RemainderModel{AtmosModel{Float64,EarthParameterSet,NoOrientation,IsentropicVortexReferenceState{Float64},ConstantViscosityWithDivergence{Float64},CLIMA.Atmos.NoHyperDiffusion,DryModel,NoPrecipitation,NoRadiation,Nothing,Tuple{},typeof(isentropicvortex_initialcondition!),Nothing},Tuple{AtmosAcousticLinearModel{AtmosModel{Float64,EarthParameterSet,NoOrientation,IsentropicVortexReferenceState{Float64},ConstantViscosityWithDivergence{Float64},CLIMA.Atmos.NoHyperDiffusion,DryModel,NoPrecipitation,NoRadiation,Nothing,Tuple{},typeof(isentropicvortex_initialcondition!),Nothing}}}},Val{2},Val{4},CLIMA.Mesh.Grids.EveryDirection,Array{Float64,3},Array{Float64,3},Array{Float64,3},Array{Float64,3},Array{Float64,3},Array{Float64,3},Float64,Array{Float64,1},Array{Float64,2},UnitRange{Int64},Bool}, ::KernelAbstractions.NDIteration.NoDynamicCheck) at /Users/charliekawczynski/.julia/packages/KernelAbstractions/cpaX2/src/backends/cpu.jl:148
   [4] __run(::KernelAbstractions.Kernel{KernelAbstractions.CPU,KernelAbstractions.NDIteration.StaticSize{(5, 5, 1)},KernelAbstractions.NDIteration.DynamicSize,typeof(CLIMA.DGmethods.cpu_volumerhs!)}, ::Tuple{Int64,Int64,Int64}, ::KernelAbstractions.NDIteration.NDRange{3,KernelAbstractions.NDIteration.DynamicSize,KernelAbstractions.NDIteration.StaticSize{(5, 5, 1)},CartesianIndices{3,Tuple{Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}},Nothing}, ::Tuple{RemainderModel{AtmosModel{Float64,EarthParameterSet,NoOrientation,IsentropicVortexReferenceState{Float64},ConstantViscosityWithDivergence{Float64},CLIMA.Atmos.NoHyperDiffusion,DryModel,NoPrecipitation,NoRadiation,Nothing,Tuple{},typeof(isentropicvortex_initialcondition!),Nothing},Tuple{AtmosAcousticLinearModel{AtmosModel{Float64,EarthParameterSet,NoOrientation,IsentropicVortexReferenceState{Float64},ConstantViscosityWithDivergence{Float64},CLIMA.Atmos.NoHyperDiffusion,DryModel,NoPrecipitation,NoRadiation,Nothing,Tuple{},typeof(isentropicvortex_initialcondition!),Nothing}}}},Val{2},Val{4},CLIMA.Mesh.Grids.EveryDirection,Array{Float64,3},Array{Float64,3},Array{Float64,3},Array{Float64,3},Array{Float64,3},Array{Float64,3},Float64,Array{Float64,1},Array{Float64,2},UnitRange{Int64},Bool}, ::KernelAbstractions.NDIteration.NoDynamicCheck) at /Users/charliekawczynski/.julia/packages/KernelAbstractions/cpaX2/src/backends/cpu.jl:121
   [5] (::KernelAbstractions.var"#30#31"{Tuple{KernelAbstractions.NoneEvent},Nothing,typeof(KernelAbstractions.__run),Tuple{KernelAbstractions.Kernel{KernelAbstractions.CPU,KernelAbstractions.NDIteration.StaticSize{(5, 5, 1)},KernelAbstractions.NDIteration.DynamicSize,typeof(CLIMA.DGmethods.cpu_volumerhs!)},Tuple{Int64,Int64,Int64},KernelAbstractions.NDIteration.NDRange{3,KernelAbstractions.NDIteration.DynamicSize,KernelAbstractions.NDIteration.StaticSize{(5, 5, 1)},CartesianIndices{3,Tuple{Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}},Nothing},Tuple{RemainderModel{AtmosModel{Float64,EarthParameterSet,NoOrientation,IsentropicVortexReferenceState{Float64},ConstantViscosityWithDivergence{Float64},CLIMA.Atmos.NoHyperDiffusion,DryModel,NoPrecipitation,NoRadiation,Nothing,Tuple{},typeof(isentropicvortex_initialcondition!),Nothing},Tuple{AtmosAcousticLinearModel{AtmosModel{Float64,EarthParameterSet,NoOrientation,IsentropicVortexReferenceState{Float64},ConstantViscosityWithDivergence{Float64},CLIMA.Atmos.NoHyperDiffusion,DryModel,NoPrecipitation,NoRadiation,Nothing,Tuple{},typeof(isentropicvortex_initialcondition!),Nothing}}}},Val{2},Val{4},CLIMA.Mesh.Grids.EveryDirection,Array{Float64,3},Array{Float64,3},Array{Float64,3},Array{Float64,3},Array{Float64,3},Array{Float64,3},Float64,Array{Float64,1},Array{Float64,2},UnitRange{Int64},Bool},KernelAbstractions.NDIteration.NoDynamicCheck}})() at /Users/charliekawczynski/.julia/packages/KernelAbstractions/cpaX2/src/backends/cpu.jl:20
  Stacktrace:
   [1] wait(::KernelAbstractions.CPU, ::KernelAbstractions.MultiEvent{Tuple{KernelAbstractions.CPUEvent}}, ::Nothing) at /Users/charliekawczynski/.julia/packages/KernelAbstractions/cpaX2/src/backends/cpu.jl:57
   [2] wait at /Users/charliekawczynski/.julia/packages/KernelAbstractions/cpaX2/src/backends/cpu.jl:27 [inlined]
   [3] (::KernelAbstractions.var"#30#31"{Tuple{KernelAbstractions.CPUEvent},Nothing,typeof(KernelAbstractions.__run),Tuple{KernelAbstractions.Kernel{KernelAbstractions.CPU,KernelAbstractions.NDIteration.StaticSize{(5,)},KernelAbstractions.NDIteration.DynamicSize,typeof(CLIMA.DGmethods.cpu_facerhs!)},Tuple{Int64},KernelAbstractions.NDIteration.NDRange{1,KernelAbstractions.NDIteration.DynamicSize,KernelAbstractions.NDIteration.StaticSize{(5,)},CartesianIndices{1,Tuple{Base.OneTo{Int64}}},Nothing},Tuple{RemainderModel{AtmosModel{Float64,EarthParameterSet,NoOrientation,IsentropicVortexReferenceState{Float64},ConstantViscosityWithDivergence{Float64},CLIMA.Atmos.NoHyperDiffusion,DryModel,NoPrecipitation,NoRadiation,Nothing,Tuple{},typeof(isentropicvortex_initialcondition!),Nothing},Tuple{AtmosAcousticLinearModel{AtmosModel{Float64,EarthParameterSet,NoOrientation,IsentropicVortexReferenceState{Float64},ConstantViscosityWithDivergence{Float64},CLIMA.Atmos.NoHyperDiffusion,DryModel,NoPrecipitation,NoRadiation,Nothing,Tuple{},typeof(isentropicvortex_initialcondition!),Nothing}}}},Val{2},Val{4},CLIMA.Mesh.Grids.EveryDirection,Rusanov,CentralNumericalFluxDiffusive,Array{Float64,3},Array{Float64,3},Array{Float64,3},Array{Float64,3},Array{Float64,3},Array{Float64,3},Array{Float64,4},Float64,Array{Int64,3},Array{Int64,3},Array{Int64,2},Array{Int64,1}},KernelAbstractions.NDIteration.NoDynamicCheck}})() at /Users/charliekawczynski/.julia/packages/KernelAbstractions/cpaX2/src/backends/cpu.jl:19
  Stacktrace:
   [1] wait(::KernelAbstractions.CPU, ::KernelAbstractions.MultiEvent{Tuple{KernelAbstractions.CPUEvent}}, ::Nothing) at /Users/charliekawczynski/.julia/packages/KernelAbstractions/cpaX2/src/backends/cpu.jl:57
   [2] wait(::KernelAbstractions.CPU, ::KernelAbstractions.MultiEvent{Tuple{KernelAbstractions.CPUEvent}}) at /Users/charliekawczynski/.julia/packages/KernelAbstractions/cpaX2/src/backends/cpu.jl:31
   [3] #_#2(::Bool, ::DGModel{RemainderModel{AtmosModel{Float64,EarthParameterSet,NoOrientation,IsentropicVortexReferenceState{Float64},ConstantViscosityWithDivergence{Float64},CLIMA.Atmos.NoHyperDiffusion,DryModel,NoPrecipitation,NoRadiation,Nothing,Tuple{},typeof(isentropicvortex_initialcondition!),Nothing},Tuple{AtmosAcousticLinearModel{AtmosModel{Float64,EarthParameterSet,NoOrientation,IsentropicVortexReferenceState{Float64},ConstantViscosityWithDivergence{Float64},CLIMA.Atmos.NoHyperDiffusion,DryModel,NoPrecipitation,NoRadiation,Nothing,Tuple{},typeof(isentropicvortex_initialcondition!),Nothing}}}},DiscontinuousSpectralElementGrid{Float64,2,4,25,Array,Array{Float64,1},Array{Float64,2},Array{Float64,3},Array{Float64,4},Array{Int64,1},Array{Int64,2},Array{Int64,3},BrickTopology{2,Float64}},Rusanov,CentralNumericalFluxDiffusive,CentralNumericalFluxGradient,CLIMA.MPIStateArrays.MPIStateArray{Float64,NamedTuple{(:∫dz, :∫dnz, :coord, :orientation, :ref_state, :turbulence, :hyperdiffusion, :moisture, :radiation),Tuple{NamedTuple{(:radiation,),Tuple{NamedTuple{(),Tuple{}}}},NamedTuple{(:radiation,),Tuple{NamedTuple{(),Tuple{}}}},SArray{Tuple{3},Float64,1,3},NamedTuple{(),Tuple{}},NamedTuple{(:ρ, :ρe, :p, :T),NTuple{4,Float64}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(:θ_v, :air_T),Tuple{Float64,Float64}},NamedTuple{(),Tuple{}}}},Array{Float64,3},Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true},CLIMA.MPIStateArrays.CMBuffers.CMBuffer{Float64,Array{Float64,2},Nothing}},CLIMA.MPIStateArrays.MPIStateArray{Float64,NamedTuple{(:∇h_tot, :turbulence, :hyperdiffusion, :moisture),Tuple{SArray{Tuple{3},Float64,1,3},NamedTuple{(:S,),Tuple{SHermitianCompact{3,Float64,6}}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}}}},Array{Float64,3},Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true},CLIMA.MPIStateArrays.CMBuffers.CMBuffer{Float64,Array{Float64,2},Nothing}},Tuple{CLIMA.MPIStateArrays.MPIStateArray{Float64,NamedTuple{(:hyperdiffusion,),Tuple{NamedTuple{(),Tuple{}}}},Array{Float64,3},Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true},CLIMA.MPIStateArrays.CMBuffers.CMBuffer{Float64,Array{Float64,2},Nothing}},CLIMA.MPIStateArrays.MPIStateArray{Float64,NamedTuple{(:hyperdiffusion,),Tuple{NamedTuple{(),Tuple{}}}},Array{Float64,3},Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true},CLIMA.MPIStateArrays.CMBuffers.CMBuffer{Float64,Array{Float64,2},Nothing}}},CLIMA.Mesh.Grids.EveryDirection,CLIMA.Mesh.Grids.EveryDirection,Nothing}, ::CLIMA.MPIStateArrays.MPIStateArray{Float64,NamedTuple{(:ρ, :ρu, :ρe, :turbulence, :hyperdiffusion, :moisture, :radiation),Tuple{Float64,SArray{Tuple{3},Float64,1,3},Float64,NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}}}},Array{Float64,3},Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true},CLIMA.MPIStateArrays.CMBuffers.CMBuffer{Float64,Array{Float64,2},Nothing}}, ::CLIMA.MPIStateArrays.MPIStateArray{Float64,NamedTuple{(:ρ, :ρu, :ρe, :turbulence, :hyperdiffusion, :moisture, :radiation),Tuple{Float64,SArray{Tuple{3},Float64,1,3},Float64,NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}}}},Array{Float64,3},Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true},CLIMA.MPIStateArrays.CMBuffers.CMBuffer{Float64,Array{Float64,2},Nothing}}, ::Nothing, ::Float64) at /Users/charliekawczynski/Dropbox/Caltech/work/dev/ClimateMachine/CLIMA/src/DGmethods/DGmodel.jl:459
   [4] (::Core.var"#kw#Any")(::NamedTuple{(:increment,),Tuple{Bool}}, ::DGModel{RemainderModel{AtmosModel{Float64,EarthParameterSet,NoOrientation,IsentropicVortexReferenceState{Float64},ConstantViscosityWithDivergence{Float64},CLIMA.Atmos.NoHyperDiffusion,DryModel,NoPrecipitation,NoRadiation,Nothing,Tuple{},typeof(isentropicvortex_initialcondition!),Nothing},Tuple{AtmosAcousticLinearModel{AtmosModel{Float64,EarthParameterSet,NoOrientation,IsentropicVortexReferenceState{Float64},ConstantViscosityWithDivergence{Float64},CLIMA.Atmos.NoHyperDiffusion,DryModel,NoPrecipitation,NoRadiation,Nothing,Tuple{},typeof(isentropicvortex_initialcondition!),Nothing}}}},DiscontinuousSpectralElementGrid{Float64,2,4,25,Array,Array{Float64,1},Array{Float64,2},Array{Float64,3},Array{Float64,4},Array{Int64,1},Array{Int64,2},Array{Int64,3},BrickTopology{2,Float64}},Rusanov,CentralNumericalFluxDiffusive,CentralNumericalFluxGradient,CLIMA.MPIStateArrays.MPIStateArray{Float64,NamedTuple{(:∫dz, :∫dnz, :coord, :orientation, :ref_state, :turbulence, :hyperdiffusion, :moisture, :radiation),Tuple{NamedTuple{(:radiation,),Tuple{NamedTuple{(),Tuple{}}}},NamedTuple{(:radiation,),Tuple{NamedTuple{(),Tuple{}}}},SArray{Tuple{3},Float64,1,3},NamedTuple{(),Tuple{}},NamedTuple{(:ρ, :ρe, :p, :T),NTuple{4,Float64}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(:θ_v, :air_T),Tuple{Float64,Float64}},NamedTuple{(),Tuple{}}}},Array{Float64,3},Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true},CLIMA.MPIStateArrays.CMBuffers.CMBuffer{Float64,Array{Float64,2},Nothing}},CLIMA.MPIStateArrays.MPIStateArray{Float64,NamedTuple{(:∇h_tot, :turbulence, :hyperdiffusion, :moisture),Tuple{SArray{Tuple{3},Float64,1,3},NamedTuple{(:S,),Tuple{SHermitianCompact{3,Float64,6}}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}}}},Array{Float64,3},Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true},CLIMA.MPIStateArrays.CMBuffers.CMBuffer{Float64,Array{Float64,2},Nothing}},Tuple{CLIMA.MPIStateArrays.MPIStateArray{Float64,NamedTuple{(:hyperdiffusion,),Tuple{NamedTuple{(),Tuple{}}}},Array{Float64,3},Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true},CLIMA.MPIStateArrays.CMBuffers.CMBuffer{Float64,Array{Float64,2},Nothing}},CLIMA.MPIStateArrays.MPIStateArray{Float64,NamedTuple{(:hyperdiffusion,),Tuple{NamedTuple{(),Tuple{}}}},Array{Float64,3},Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true},CLIMA.MPIStateArrays.CMBuffers.CMBuffer{Float64,Array{Float64,2},Nothing}}},CLIMA.Mesh.Grids.EveryDirection,CLIMA.Mesh.Grids.EveryDirection,Nothing}, ::CLIMA.MPIStateArrays.MPIStateArray{Float64,NamedTuple{(:ρ, :ρu, :ρe, :turbulence, :hyperdiffusion, :moisture, :radiation),Tuple{Float64,SArray{Tuple{3},Float64,1,3},Float64,NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}}}},Array{Float64,3},Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true},CLIMA.MPIStateArrays.CMBuffers.CMBuffer{Float64,Array{Float64,2},Nothing}}, ::CLIMA.MPIStateArrays.MPIStateArray{Float64,NamedTuple{(:ρ, :ρu, :ρe, :turbulence, :hyperdiffusion, :moisture, :radiation),Tuple{Float64,SArray{Tuple{3},Float64,1,3},Float64,NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}}}},Array{Float64,3},Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true},CLIMA.MPIStateArrays.CMBuffers.CMBuffer{Float64,Array{Float64,2},Nothing}}, ::Nothing, ::Float64) at ./none:0
   [5] dostep!(::CLIMA.MPIStateArrays.MPIStateArray{Float64,NamedTuple{(:ρ, :ρu, :ρe, :turbulence, :hyperdiffusion, :moisture, :radiation),Tuple{Float64,SArray{Tuple{3},Float64,1,3},Float64,NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}}}},Array{Float64,3},Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true},CLIMA.MPIStateArrays.CMBuffers.CMBuffer{Float64,Array{Float64,2},Nothing}}, ::MultirateRungeKutta{LowStorageRungeKutta2N{Float64,Float64,CLIMA.MPIStateArrays.MPIStateArray{Float64,NamedTuple{(:ρ, :ρu, :ρe, :turbulence, :hyperdiffusion, :moisture, :radiation),Tuple{Float64,SArray{Tuple{3},Float64,1,3},Float64,NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}}}},Array{Float64,3},Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true},CLIMA.MPIStateArrays.CMBuffers.CMBuffer{Float64,Array{Float64,2},Nothing}},14},StrongStabilityPreservingRungeKutta{Float64,Float64,CLIMA.MPIStateArrays.MPIStateArray{Float64,NamedTuple{(:ρ, :ρu, :ρe, :turbulence, :hyperdiffusion, :moisture, :radiation),Tuple{Float64,SArray{Tuple{3},Float64,1,3},Float64,NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}}}},Array{Float64,3},Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true},CLIMA.MPIStateArrays.CMBuffers.CMBuffer{Float64,Array{Float64,2},Nothing}},3},Float64}, ::Nothing, ::Float64, ::Nothing, ::Nothing, ::Nothing) at /Users/charliekawczynski/Dropbox/Caltech/work/dev/ClimateMachine/CLIMA/src/ODESolvers/MultirateRungeKuttaMethod.jl:106
   [6] dostep! at /Users/charliekawczynski/Dropbox/Caltech/work/dev/ClimateMachine/CLIMA/src/ODESolvers/MultirateRungeKuttaMethod.jl:90 [inlined]
   [7] #general_dostep!#9 at /Users/charliekawczynski/Dropbox/Caltech/work/dev/ClimateMachine/CLIMA/src/ODESolvers/ODESolvers.jl:61 [inlined]
   [8] #general_dostep! at ./none:0 [inlined]
   [9] #solve!#10(::Float64, ::Bool, ::Int64, ::Tuple{EveryXWallTimeSeconds}, ::typeof(solve!), ::CLIMA.MPIStateArrays.MPIStateArray{Float64,NamedTuple{(:ρ, :ρu, :ρe, :turbulence, :hyperdiffusion, :moisture, :radiation),Tuple{Float64,SArray{Tuple{3},Float64,1,3},Float64,NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}}}},Array{Float64,3},Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true},CLIMA.MPIStateArrays.CMBuffers.CMBuffer{Float64,Array{Float64,2},Nothing}}, ::MultirateRungeKutta{LowStorageRungeKutta2N{Float64,Float64,CLIMA.MPIStateArrays.MPIStateArray{Float64,NamedTuple{(:ρ, :ρu, :ρe, :turbulence, :hyperdiffusion, :moisture, :radiation),Tuple{Float64,SArray{Tuple{3},Float64,1,3},Float64,NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}}}},Array{Float64,3},Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true},CLIMA.MPIStateArrays.CMBuffers.CMBuffer{Float64,Array{Float64,2},Nothing}},14},StrongStabilityPreservingRungeKutta{Float64,Float64,CLIMA.MPIStateArrays.MPIStateArray{Float64,NamedTuple{(:ρ, :ρu, :ρe, :turbulence, :hyperdiffusion, :moisture, :radiation),Tuple{Float64,SArray{Tuple{3},Float64,1,3},Float64,NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}},NamedTuple{(),Tuple{}}}},Array{Float64,3},Array{Int64,1},SubArray{Float64,3,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Base.Slice{Base.OneTo{Int64}},UnitRange{Int64}},true},CLIMA.MPIStateArrays.CMBuffers.CMBuffer{Float64,Array{Float64,2},Nothing}},3},Float64}, ::Nothing) at /Users/charliekawczynski/Dropbox/Caltech/work/dev/ClimateMachine/CLIMA/src/ODESolvers/ODESolvers.jl:126
   [10] #solve! at ./none:0 [inlined] (repeats 2 times)
   [11] run(::MPI.Comm, ::Type, ::Int64, ::Tuple{Int64,Int64}, ::IsentropicVortexSetup{Float64}, ::Type, ::Function, ::Int64, ::Int64) at /Users/charliekawczynski/Dropbox/Caltech/work/dev/ClimateMachine/CLIMA/test/DGmethods/Euler/isentropicvortex_multirate.jl:276
   [12] macro expansion at /Users/charliekawczynski/Dropbox/Caltech/work/dev/ClimateMachine/CLIMA/test/DGmethods/Euler/isentropicvortex_multirate.jl:86 [inlined]
   [13] macro expansion at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107 [inlined]
   [14] main() at /Users/charliekawczynski/Dropbox/Caltech/work/dev/ClimateMachine/CLIMA/test/DGmethods/Euler/isentropicvortex_multirate.jl:71
   [15] top-level scope at /Users/charliekawczynski/Dropbox/Caltech/work/dev/ClimateMachine/CLIMA/test/DGmethods/Euler/isentropicvortex_multirate.jl:404
   [16] include at ./boot.jl:328 [inlined]
   [17] include_relative(::Module, ::String) at ./loading.jl:1105
   [18] include(::Module, ::String) at ./Base.jl:31
   [19] exec_options(::Base.JLOptions) at ./client.jl:287
   [20] _start() at ./client.jl:460

What used to be the interpretable line now reads

wait(::KernelAbstractions.CPU, ::KernelAbstractions.MultiEvent{Tuple{KernelAbstractions.CPUEvent}}, ::Nothing) at /Users/charliekawczynski/.julia/packages/KernelAbstractions/cpaX2/src/backends/cpu.jl:57

I understand that, technically, wait may be responsible for calling linearized_pressure. But it's now more difficult to determine what line I need to fix--the closest thing to what the user will be familiar with is dostep!, which is very far up the call graph compared to flux_nondiffusive! (the nearest familiar function call I'd prefer to see).

I understand that this may be difficult to fix, and that "lying" in the stack trace is potentially dangerous, but I think we need something more trace-friendly or some way to generate _something_ that is more trace-friendly because what we currently have is drifting away from being useful (as it was useless in this case). Otherwise, we may be stuck with reporting our stack-traces to someone who can figure out what went wrong, and we shouldn't put that burden on a single person. cc: @vchuravy ;)

Additional context

For CLIMA Developers

• [ ] This feature can be added (if it cannot be, explain why in a comment below -- lack of technical expertise, not relevant to the scope of this project, too ambitious)
• [ ] There is a timeline for when this feature can be implemented
• [ ] The feature has been (or will be) implemented (Please link the PR)

[vchuravy]

Probably related to https://github.com/JuliaGPU/KernelAbstractions.jl/issues/44 since:

   [1] macro expansion at /Users/charliekawczynski/.julia/packages/Cassette/7OymZ/src/context.jl:450 [inlined]

is simply wrong.

Julia in 1.5 is adding a mechanism to hide things in stacktraces, which would cut down on the length of the stacktrace.