Climatemachine.jl: Slowdown from using CLIMAParameters (?)

[trontrytel]

Description

@charleskawczynski and @kpamnany - It seems like using CLIMAParameters makes my code much slower. I made two branches that do the same test and I tried to make the test small. One branch is using Microphysics module with all parameters hardcoded (aj/ice_fast) and the other is using the CLIMAParameters module (aj/ice_slow).

Here is the github comparison between them: https://github.com/CliMA/ClimateMachine.jl/compare/aj/ice_fast...aj/ice_slow

For the slow and fast branches on my laptop the difference is quite big:
time julia --project test/Atmos/Parameterizations/Microphysics/warm_rain.jl

real    94m17.028s 
user    34m35.634s
sys     0m29.122s

real    10m34.746s
user    7m13.580s
sys     0m19.629s

(As a side note. For some obscure reason the github diff will show you differences in t_end and output_freq. I checked around 15 times with my eyes and git diff in terminal and in the actual files they are the same. I don't know why this is happening).

For CLIMA Developers

• [ ] This feature can be added (if it cannot be, explain why in a comment below -- lack of technical expertise, not relevant to the scope of this project, too ambitious)
• [ ] There is a timeline for when this feature can be implemented
• [ ] The feature has been (or will be) implemented (Please link the PR)

[kpamnany]

Using v0_rai() as an example:

With param_sets (aj/ice_slow):

julia> @code_llvm ClimateMachine.Microphysics.v0_rai(eps, rps, 1.0)

;  @ /central/home/kpamnany/ClimateMachine.jl/src/Atmos/Parameterizations/CloudPhysics/Microphysics.jl:41 within `v0_rai'
define double @julia_v0_rai_17951(double) {
top:
;  @ /central/home/kpamnany/ClimateMachine.jl/src/Atmos/Parameterizations/CloudPhysics/Microphysics.jl:46 within `v0_rai'
; ┌ @ float.jl:407 within `/'
   %1 = fdiv double 1.000000e+03, %0
; └
; ┌ @ float.jl:403 within `-'
   %2 = fadd double %1, -1.000000e+00
; └
; ┌ @ operators.jl:529 within `*' @ float.jl:405
   %3 = fmul double %2, 0x401364D9364D9364
   %4 = fmul double %3, 9.810000e+00
; │ @ operators.jl:529 within `*'
; │┌ @ operators.jl:516 within `afoldl'
; ││┌ @ float.jl:405 within `*'
     %5 = fmul double %4, 1.000000e-03
; └└└
; ┌ @ math.jl:557 within `sqrt'
; │┌ @ float.jl:458 within `<'
    %6 = fcmp uge double %5, 0.000000e+00
; │└
   br i1 %6, label %L10, label %L8

L8:                                               ; preds = %top
   %7 = call nonnull %jl_value_t addrspace(10)* @julia_throw_complex_domainerror_11970(%jl_value_t addrspace(10)* addrspacecast (%jl_value_t* inttoptr (i64 47611956809888 to %jl_value_t*) to %jl_value_t addrspace(10)*), double %5)
   call void @llvm.trap()
   unreachable

L10:                                              ; preds = %top
; │ @ math.jl:558 within `sqrt'
   %8 = call double @llvm.sqrt.f64(double %5)
; └
  ret double %8
}

Without param_sets (aj/ice_fast):

julia> @code_llvm Microphysics.v0_rai(1.0)

;  @ /central/home/kpamnany/ClimateMachine.jl/src/Atmos/Parameterizations/CloudPhysics/Microphysics.jl:30 within `v0_rai'
define double @julia_v0_rai_18018(double) {
top:
;  @ /central/home/kpamnany/ClimateMachine.jl/src/Atmos/Parameterizations/CloudPhysics/Microphysics.jl:35 within `v0_rai'
; ┌ @ float.jl:407 within `/'
   %1 = fdiv double 9.167000e+02, %0
; └
; ┌ @ float.jl:403 within `-'
   %2 = fadd double %1, -1.000000e+00
; └
; ┌ @ operators.jl:529 within `*' @ float.jl:405
   %3 = fmul double %2, 0x401364D9364D9364
   %4 = fmul double %3, 9.810000e+00
; │ @ operators.jl:529 within `*'
; │┌ @ operators.jl:516 within `afoldl'
; ││┌ @ float.jl:405 within `*'
     %5 = fmul double %4, 1.000000e-03
; └└└
; ┌ @ math.jl:557 within `sqrt'
; │┌ @ float.jl:458 within `<'
    %6 = fcmp uge double %5, 0.000000e+00
; │└
   br i1 %6, label %L10, label %L8

L8:                                               ; preds = %top
   %7 = call nonnull %jl_value_t addrspace(10)* @julia_throw_complex_domainerror_11970(%jl_value_t addrspace(10)* addrspacecast (%jl_value_t* inttoptr (i64 47616007475360 to %jl_value_t*) to %jl_value_t addrspace(10)*), double %5)
   call void @llvm.trap()
   unreachable

L10:                                              ; preds = %top
; │ @ math.jl:558 within `sqrt'
   %8 = call double @llvm.sqrt.f64(double %5)
; └
  ret double %8
}

I see no problem here. @vchuravy, any suggestions for understanding this?

[jkozdon]

When I try to run with Julia 1.4 I get the following errors:

jekozdon@ip-172-31-47-159:~/codes/CLIMA2 ± ● aj/ice_slow
❯ julia --project test/Atmos/Parameterizations/Microphysics/warm_rain.jl
┌ Warning: Incompatibility detected between CUDA and LLVM 8.0+; disabling debug info emission for CUDA kernels
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:114
┌ Info: Model composition
│     param_set = EarthParameterSet{MicropysicsParameterSet{LiquidParameterSet,IceParameterSet,RainParameterSet,SnowParameterSet}}(MicropysicsParameterSet{LiquidParameterSet,IceParameterSet,RainParameterSet,SnowParameterSet}(LiquidParameterSet(), IceParameterSet(), RainParameterSet(), SnowParameterSet()))
│     orientation = FlatOrientation()
│     moisture = nothing
│     precipitation = nothing
│     source = nothing
│     boundarycondition = nothing
│     init_state_conservative = init_kinematic_eddy!
└     data_config = KinematicModelConfig{Float64}(1500.0, 10.0, 1500.0, 0.6, 289.0, 101500.0, 100000.0, 0.0075, 0.0)
┌ Info: Establishing Atmos LES configuration for KinematicModel
│     precision        = Float64
│     polynomial order = 4
│     domain           = 1500.00 m x10.00 m x1500.00 m
│     resolution       = 20x1x20
│     MPI ranks        = 1
│     min(Δ_horz)      = 0.86 m
└     min(Δ_vert)      = 13.63 m
[ Info: Initializing KinematicModel
ERROR: LoadError: InvalidIRError: compiling gpu_kernel_nodal_update_auxiliary_state!(Cassette.Context{nametype(CUDACtx),KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize,KernelAbstractions.NDIteration.DynamicCheck,Nothing,CartesianIndices{1,Tuple{Base.OneTo{Int64}}},KernelAbstractions.NDIteration.NDRange{1,KernelAbstractions.NDIteration.DynamicSize,KernelAbstractions.NDIteration.StaticSize{(125,)},CartesianIndices{1,Tuple{Base.OneTo{Int64}}},Nothing}},Nothing,KernelAbstractions.var"##PassType#253",Nothing,Cassette.DisableHooks}, typeof(ClimateMachine.DGmethods.gpu_kernel_nodal_update_auxiliary_state!), KinematicModel{Float64,EarthParameterSet{MicropysicsParameterSet{LiquidParameterSet,IceParameterSet,RainParameterSet,SnowParameterSet}},FlatOrientation,Nothing,Nothing,Nothing,Nothing,typeof(init_kinematic_eddy!),KinematicModelConfig{Float64}}, Val{3}, Val{4}, typeof(kinematic_model_nodal_update_auxiliary_state!), CUDAnative.CuDeviceArray{Float64,3,CUDAnative.AS.Global}, CUDAnative.CuDeviceArray{Float64,3,CUDAnative.AS.Global}, Float64, UnitRange{Int64}, CUDAnative.CuDeviceArray{Bool,1,CUDAnative.AS.Global}) resulted in invalid LLVM IR
Reason: unsupported dynamic function invocation (call to getproperty)
Stacktrace:
 [1] call at /home/jekozdon/.julia/packages/Cassette/158rp/src/context.jl:456
 [2] fallback at /home/jekozdon/.julia/packages/Cassette/158rp/src/context.jl:454
 [3] overdub at /home/jekozdon/.julia/packages/Cassette/158rp/src/context.jl:279
 [4] m0 at /home/jekozdon/.julia/packages/CLIMAParameters/EHWaS/src/AtmosParameters.jl:54
 [5] unpack_params at /home/jekozdon/codes/CLIMA2/src/Atmos/Parameterizations/CloudPhysics/Microphysics.jl:68
 [6] terminal_velocity at /home/jekozdon/codes/CLIMA2/src/Atmos/Parameterizations/CloudPhysics/Microphysics.jl:154
 [7] kinematic_model_nodal_update_auxiliary_state! at /home/jekozdon/codes/CLIMA2/test/Atmos/Parameterizations/Microphysics/warm_rain.jl:116
 [8] macro expansion at /home/jekozdon/codes/CLIMA2/src/Numerics/DGmethods/DGmodel_kernels.jl:1542
 [9] gpu_kernel_nodal_update_auxiliary_state! at /home/jekozdon/.julia/packages/KernelAbstractions/tHbOr/src/macros.jl:80
 [10] overdub at /home/jekozdon/.julia/packages/Cassette/158rp/src/overdub.jl:0
Reason: unsupported dynamic function invocation (call to overdub)
Stacktrace:
 [1] m0 at /home/jekozdon/.julia/packages/CLIMAParameters/EHWaS/src/AtmosParameters.jl:54
 [2] unpack_params at /home/jekozdon/codes/CLIMA2/src/Atmos/Parameterizations/CloudPhysics/Microphysics.jl:68
 [3] terminal_velocity at /home/jekozdon/codes/CLIMA2/src/Atmos/Parameterizations/CloudPhysics/Microphysics.jl:154
 [4] kinematic_model_nodal_update_auxiliary_state! at /home/jekozdon/codes/CLIMA2/test/Atmos/Parameterizations/Microphysics/warm_rain.jl:116
 [5] macro expansion at /home/jekozdon/codes/CLIMA2/src/Numerics/DGmethods/DGmodel_kernels.jl:1542
 [6] gpu_kernel_nodal_update_auxiliary_state! at /home/jekozdon/.julia/packages/KernelAbstractions/tHbOr/src/macros.jl:80
 [7] overdub at /home/jekozdon/.julia/packages/Cassette/158rp/src/overdub.jl:0
Reason: unsupported dynamic function invocation (call to overdub)
Stacktrace:
 [1] unpack_params at /home/jekozdon/codes/CLIMA2/src/Atmos/Parameterizations/CloudPhysics/Microphysics.jl:68
 [2] terminal_velocity at /home/jekozdon/codes/CLIMA2/src/Atmos/Parameterizations/CloudPhysics/Microphysics.jl:154
 [3] kinematic_model_nodal_update_auxiliary_state! at /home/jekozdon/codes/CLIMA2/test/Atmos/Parameterizations/Microphysics/warm_rain.jl:116
 [4] macro expansion at /home/jekozdon/codes/CLIMA2/src/Numerics/DGmethods/DGmodel_kernels.jl:1542
 [5] gpu_kernel_nodal_update_auxiliary_state! at /home/jekozdon/.julia/packages/KernelAbstractions/tHbOr/src/macros.jl:80
 [6] overdub at /home/jekozdon/.julia/packages/Cassette/158rp/src/overdub.jl:0
Reason: unsupported call to the Julia runtime (call to jl_type_error)
Stacktrace:
 [1] unpack_params at /home/jekozdon/codes/CLIMA2/src/Atmos/Parameterizations/CloudPhysics/Microphysics.jl:68
 [2] terminal_velocity at /home/jekozdon/codes/CLIMA2/src/Atmos/Parameterizations/CloudPhysics/Microphysics.jl:154
 [3] kinematic_model_nodal_update_auxiliary_state! at /home/jekozdon/codes/CLIMA2/test/Atmos/Parameterizations/Microphysics/warm_rain.jl:116
 [4] macro expansion at /home/jekozdon/codes/CLIMA2/src/Numerics/DGmethods/DGmodel_kernels.jl:1542
 [5] gpu_kernel_nodal_update_auxiliary_state! at /home/jekozdon/.julia/packages/KernelAbstractions/tHbOr/src/macros.jl:80
 [6] overdub at /home/jekozdon/.julia/packages/Cassette/158rp/src/overdub.jl:0
Reason: unsupported dynamic function invocation (call to getproperty)
Stacktrace:
 [1] call at /home/jekozdon/.julia/packages/Cassette/158rp/src/context.jl:456
 [2] fallback at /home/jekozdon/.julia/packages/Cassette/158rp/src/context.jl:454
 [3] overdub at /home/jekozdon/.julia/packages/Cassette/158rp/src/context.jl:279
 [4] a0 at /home/jekozdon/.julia/packages/CLIMAParameters/EHWaS/src/AtmosParameters.jl:56
 [5] unpack_params at /home/jekozdon/codes/CLIMA2/src/Atmos/Parameterizations/CloudPhysics/Microphysics.jl:70
 [6] terminal_velocity at /home/jekozdon/codes/CLIMA2/src/Atmos/Parameterizations/CloudPhysics/Microphysics.jl:154
 [7] kinematic_model_nodal_update_auxiliary_state! at /home/jekozdon/codes/CLIMA2/test/Atmos/Parameterizations/Microphysics/warm_rain.jl:116
 [8] macro expansion at /home/jekozdon/codes/CLIMA2/src/Numerics/DGmethods/DGmodel_kernels.jl:1542
 [9] gpu_kernel_nodal_update_auxiliary_state! at /home/jekozdon/.julia/packages/KernelAbstractions/tHbOr/src/macros.jl:80
 [10] overdub at /home/jekozdon/.julia/packages/Cassette/158rp/src/overdub.jl:0
Reason: unsupported dynamic function invocation (call to overdub)
Stacktrace:
 [1] a0 at /home/jekozdon/.julia/packages/CLIMAParameters/EHWaS/src/AtmosParameters.jl:56
 [2] unpack_params at /home/jekozdon/codes/CLIMA2/src/Atmos/Parameterizations/CloudPhysics/Microphysics.jl:70
 [3] terminal_velocity at /home/jekozdon/codes/CLIMA2/src/Atmos/Parameterizations/CloudPhysics/Microphysics.jl:154
 [4] kinematic_model_nodal_update_auxiliary_state! at /home/jekozdon/codes/CLIMA2/test/Atmos/Parameterizations/Microphysics/warm_rain.jl:116
 [5] macro expansion at /home/jekozdon/codes/CLIMA2/src/Numerics/DGmethods/DGmodel_kernels.jl:1542
 [6] gpu_kernel_nodal_update_auxiliary_state! at /home/jekozdon/.julia/packages/KernelAbstractions/tHbOr/src/macros.jl:80
 [7] overdub at /home/jekozdon/.julia/packages/Cassette/158rp/src/overdub.jl:0
Reason: unsupported dynamic function invocation (call to overdub)
Stacktrace:
 [1] unpack_params at /home/jekozdon/codes/CLIMA2/src/Atmos/Parameterizations/CloudPhysics/Microphysics.jl:70
 [2] terminal_velocity at /home/jekozdon/codes/CLIMA2/src/Atmos/Parameterizations/CloudPhysics/Microphysics.jl:154
 [3] kinematic_model_nodal_update_auxiliary_state! at /home/jekozdon/codes/CLIMA2/test/Atmos/Parameterizations/Microphysics/warm_rain.jl:116
 [4] macro expansion at /home/jekozdon/codes/CLIMA2/src/Numerics/DGmethods/DGmodel_kernels.jl:1542
 [5] gpu_kernel_nodal_update_auxiliary_state! at /home/jekozdon/.julia/packages/KernelAbstractions/tHbOr/src/macros.jl:80
 [6] overdub at /home/jekozdon/.julia/packages/Cassette/158rp/src/overdub.jl:0
Reason: unsupported call to the Julia runtime (call to jl_type_error)
Stacktrace:
 [1] unpack_params at /home/jekozdon/codes/CLIMA2/src/Atmos/Parameterizations/CloudPhysics/Microphysics.jl:70
 [2] terminal_velocity at /home/jekozdon/codes/CLIMA2/src/Atmos/Parameterizations/CloudPhysics/Microphysics.jl:154
 [3] kinematic_model_nodal_update_auxiliary_state! at /home/jekozdon/codes/CLIMA2/test/Atmos/Parameterizations/Microphysics/warm_rain.jl:116
 [4] macro expansion at /home/jekozdon/codes/CLIMA2/src/Numerics/DGmethods/DGmodel_kernels.jl:1542
 [5] gpu_kernel_nodal_update_auxiliary_state! at /home/jekozdon/.julia/packages/KernelAbstractions/tHbOr/src/macros.jl:80
 [6] overdub at /home/jekozdon/.julia/packages/Cassette/158rp/src/overdub.jl:0
Stacktrace:
 [1] check_ir(::CUDAnative.CompilerJob, ::LLVM.Module) at /home/jekozdon/.julia/packages/CUDAnative/hfulr/src/compiler/validation.jl:116
 [2] macro expansion at /home/jekozdon/.julia/packages/CUDAnative/hfulr/src/compiler/driver.jl:193 [inlined]
 [3] macro expansion at /home/jekozdon/.julia/packages/TimerOutputs/NvIUx/src/TimerOutput.jl:245 [inlined]
 [4] codegen(::Symbol, ::CUDAnative.CompilerJob; libraries::Bool, dynamic_parallelism::Bool, optimize::Bool, strip::Bool, strict::Bool) at /home/jekozdon/.julia/packages/CUDAnative/hfulr/src/compiler/driver.jl:191
 [5] compile(::Symbol, ::CUDAnative.CompilerJob; libraries::Bool, dynamic_parallelism::Bool, optimize::Bool, strip::Bool, strict::Bool) at /home/jekozdon/.julia/packages/CUDAnative/hfulr/src/compiler/driver.jl:52
 [6] #compile#152 at /home/jekozdon/.julia/packages/CUDAnative/hfulr/src/compiler/driver.jl:33 [inlined]
 [7] macro expansion at /home/jekozdon/.julia/packages/CUDAnative/hfulr/src/execution.jl:393 [inlined]
 [8] cufunction(::typeof(Cassette.overdub), ::Type{Tuple{Cassette.Context{nametype(CUDACtx),KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize,KernelAbstractions.NDIteration.DynamicCheck,Nothing,CartesianIndices{1,Tuple{Base.OneTo{Int64}}},KernelAbstractions.NDIteration.NDRange{1,KernelAbstractions.NDIteration.DynamicSize,KernelAbstractions.NDIteration.StaticSize{(125,)},CartesianIndices{1,Tuple{Base.OneTo{Int64}}},Nothing}},Nothing,KernelAbstractions.var"##PassType#253",Nothing,Cassette.DisableHooks},typeof(ClimateMachine.DGmethods.gpu_kernel_nodal_update_auxiliary_state!),KinematicModel{Float64,EarthParameterSet{MicropysicsParameterSet{LiquidParameterSet,IceParameterSet,RainParameterSet,SnowParameterSet}},FlatOrientation,Nothing,Nothing,Nothing,Nothing,typeof(init_kinematic_eddy!),KinematicModelConfig{Float64}},Val{3},Val{4},typeof(kinematic_model_nodal_update_auxiliary_state!),CUDAnative.CuDeviceArray{Float64,3,CUDAnative.AS.Global},CUDAnative.CuDeviceArray{Float64,3,CUDAnative.AS.Global},Float64,UnitRange{Int64},CUDAnative.CuDeviceArray{Bool,1,CUDAnative.AS.Global}}}; name::String, kwargs::Base.Iterators.Pairs{Symbol,Int64,Tuple{Symbol},NamedTuple{(:maxthreads,),Tuple{Int64}}}) at /home/jekozdon/.julia/packages/CUDAnative/hfulr/src/execution.jl:360
 [9] macro expansion at /home/jekozdon/.julia/packages/CUDAnative/hfulr/src/execution.jl:179 [inlined]
 [10] (::KernelAbstractions.Kernel{KernelAbstractions.CUDA,KernelAbstractions.NDIteration.StaticSize{(125,)},KernelAbstractions.NDIteration.DynamicSize,typeof(ClimateMachine.DGmethods.gpu_kernel_nodal_update_auxiliary_state!)})(::KinematicModel{Float64,EarthParameterSet{MicropysicsParameterSet{LiquidParameterSet,IceParameterSet,RainParameterSet,SnowParameterSet}},FlatOrientation,Nothing,Nothing,Nothing,Nothing,typeof(init_kinematic_eddy!),KinematicModelConfig{Float64}}, ::Vararg{Any,N} where N; ndrange::Int64, dependencies::Tuple{KernelAbstractions.CudaEvent}, workgroupsize::Nothing, progress::Function) at /home/jekozdon/.julia/packages/KernelAbstractions/tHbOr/src/backends/cuda.jl:215
 [11] nodal_update_auxiliary_state!(::Function, ::DGModel{KinematicModel{Float64,EarthParameterSet{MicropysicsParameterSet{LiquidParameterSet,IceParameterSet,RainParameterSet,SnowParameterSet}},FlatOrientation,Nothing,Nothing,Nothing,Nothing,typeof(init_kinematic_eddy!),KinematicModelConfig{Float64}},DiscontinuousSpectralElementGrid{Float64,3,4,125,CuArrays.CuArray,CuArrays.CuArray{Float64,1,Nothing},CuArrays.CuArray{Float64,2,Nothing},CuArrays.CuArray{Float64,3,Nothing},CuArrays.CuArray{Float64,4,Nothing},CuArrays.CuArray{Int64,1,Nothing},CuArrays.CuArray{Int64,2,Nothing},CuArrays.CuArray{Int64,3,Nothing},StackedBrickTopology{3,Float64}},RusanovNumericalFlux,CentralNumericalFluxSecondOrder,CentralNumericalFluxGradient,MPIStateArray{Float64,NamedTuple{(:p, :z, :u, :w, :q_tot, :q_vap, :q_liq, :q_ice, :q_rai, :e_tot, :e_kin, :e_pot, :e_int, :T, :S, :RH, :rain_w),NTuple{17,Float64}},CuArrays.CuArray{Float64,3,Nothing},CuArrays.CuArray{Int64,1,Nothing},CuArrays.CuArray{Float64,3,CuArrays.CuArray{Float64,3,Nothing}},ClimateMachine.MPIStateArrays.CMBuffers.CMBuffer{Float64,CuArrays.CuArray{Float64,2,Nothing},Array{Float64,2}}},MPIStateArray{Float64,NamedTuple{(),Tuple{}},CuArrays.CuArray{Float64,3,Nothing},CuArrays.CuArray{Int64,1,Nothing},CuArrays.CuArray{Float64,3,CuArrays.CuArray{Float64,3,Nothing}},ClimateMachine.MPIStateArrays.CMBuffers.CMBuffer{Float64,CuArrays.CuArray{Float64,2,Nothing},Array{Float64,2}}},Tuple{MPIStateArray{Float64,NamedTuple{(),Tuple{}},CuArrays.CuArray{Float64,3,Nothing},CuArrays.CuArray{Int64,1,Nothing},CuArrays.CuArray{Float64,3,CuArrays.CuArray{Float64,3,Nothing}},ClimateMachine.MPIStateArrays.CMBuffers.CMBuffer{Float64,CuArrays.CuArray{Float64,2,Nothing},Array{Float64,2}}},MPIStateArray{Float64,NamedTuple{(),Tuple{}},CuArrays.CuArray{Float64,3,Nothing},CuArrays.CuArray{Int64,1,Nothing},CuArrays.CuArray{Float64,3,CuArrays.CuArray{Float64,3,Nothing}},ClimateMachine.MPIStateArrays.CMBuffers.CMBuffer{Float64,CuArrays.CuArray{Float64,2,Nothing},Array{Float64,2}}}},EveryDirection,EveryDirection,Nothing}, ::KinematicModel{Float64,EarthParameterSet{MicropysicsParameterSet{LiquidParameterSet,IceParameterSet,RainParameterSet,SnowParameterSet}},FlatOrientation,Nothing,Nothing,Nothing,Nothing,typeof(init_kinematic_eddy!),KinematicModelConfig{Float64}}, ::MPIStateArray{Float64,NamedTuple{(:ρ, :ρu, :ρe, :ρq_tot, :ρq_liq, :ρq_ice, :ρq_rai),Tuple{Float64,SArray{Tuple{3},Float64,1,3},Float64,Float64,Float64,Float64,Float64}},CuArrays.CuArray{Float64,3,Nothing},CuArrays.CuArray{Int64,1,Nothing},CuArrays.CuArray{Float64,3,CuArrays.CuArray{Float64,3,Nothing}},ClimateMachine.MPIStateArrays.CMBuffers.CMBuffer{Float64,CuArrays.CuArray{Float64,2,Nothing},Array{Float64,2}}}, ::Float64, ::UnitRange{Int64}; diffusive::Bool) at /home/jekozdon/codes/CLIMA2/src/Numerics/DGmethods/DGmodel.jl:745
 [12] nodal_update_auxiliary_state! at /home/jekozdon/codes/CLIMA2/src/Numerics/DGmethods/DGmodel.jl:713 [inlined]
 [13] update_auxiliary_state!(::DGModel{KinematicModel{Float64,EarthParameterSet{MicropysicsParameterSet{LiquidParameterSet,IceParameterSet,RainParameterSet,SnowParameterSet}},FlatOrientation,Nothing,Nothing,Nothing,Nothing,typeof(init_kinematic_eddy!),KinematicModelConfig{Float64}},DiscontinuousSpectralElementGrid{Float64,3,4,125,CuArrays.CuArray,CuArrays.CuArray{Float64,1,Nothing},CuArrays.CuArray{Float64,2,Nothing},CuArrays.CuArray{Float64,3,Nothing},CuArrays.CuArray{Float64,4,Nothing},CuArrays.CuArray{Int64,1,Nothing},CuArrays.CuArray{Int64,2,Nothing},CuArrays.CuArray{Int64,3,Nothing},StackedBrickTopology{3,Float64}},RusanovNumericalFlux,CentralNumericalFluxSecondOrder,CentralNumericalFluxGradient,MPIStateArray{Float64,NamedTuple{(:p, :z, :u, :w, :q_tot, :q_vap, :q_liq, :q_ice, :q_rai, :e_tot, :e_kin, :e_pot, :e_int, :T, :S, :RH, :rain_w),NTuple{17,Float64}},CuArrays.CuArray{Float64,3,Nothing},CuArrays.CuArray{Int64,1,Nothing},CuArrays.CuArray{Float64,3,CuArrays.CuArray{Float64,3,Nothing}},ClimateMachine.MPIStateArrays.CMBuffers.CMBuffer{Float64,CuArrays.CuArray{Float64,2,Nothing},Array{Float64,2}}},MPIStateArray{Float64,NamedTuple{(),Tuple{}},CuArrays.CuArray{Float64,3,Nothing},CuArrays.CuArray{Int64,1,Nothing},CuArrays.CuArray{Float64,3,CuArrays.CuArray{Float64,3,Nothing}},ClimateMachine.MPIStateArrays.CMBuffers.CMBuffer{Float64,CuArrays.CuArray{Float64,2,Nothing},Array{Float64,2}}},Tuple{MPIStateArray{Float64,NamedTuple{(),Tuple{}},CuArrays.CuArray{Float64,3,Nothing},CuArrays.CuArray{Int64,1,Nothing},CuArrays.CuArray{Float64,3,CuArrays.CuArray{Float64,3,Nothing}},ClimateMachine.MPIStateArrays.CMBuffers.CMBuffer{Float64,CuArrays.CuArray{Float64,2,Nothing},Array{Float64,2}}},MPIStateArray{Float64,NamedTuple{(),Tuple{}},CuArrays.CuArray{Float64,3,Nothing},CuArrays.CuArray{Int64,1,Nothing},CuArrays.CuArray{Float64,3,CuArrays.CuArray{Float64,3,Nothing}},ClimateMachine.MPIStateArrays.CMBuffers.CMBuffer{Float64,CuArrays.CuArray{Float64,2,Nothing},Array{Float64,2}}}},EveryDirection,EveryDirection,Nothing}, ::KinematicModel{Float64,EarthParameterSet{MicropysicsParameterSet{LiquidParameterSet,IceParameterSet,RainParameterSet,SnowParameterSet}},FlatOrientation,Nothing,Nothing,Nothing,Nothing,typeof(init_kinematic_eddy!),KinematicModelConfig{Float64}}, ::MPIStateArray{Float64,NamedTuple{(:ρ, :ρu, :ρe, :ρq_tot, :ρq_liq, :ρq_ice, :ρq_rai),Tuple{Float64,SArray{Tuple{3},Float64,1,3},Float64,Float64,Float64,Float64,Float64}},CuArrays.CuArray{Float64,3,Nothing},CuArrays.CuArray{Int64,1,Nothing},CuArrays.CuArray{Float64,3,CuArrays.CuArray{Float64,3,Nothing}},ClimateMachine.MPIStateArrays.CMBuffers.CMBuffer{Float64,CuArrays.CuArray{Float64,2,Nothing},Array{Float64,2}}}, ::Float64, ::UnitRange{Int64}) at /home/jekozdon/codes/CLIMA2/test/Atmos/Parameterizations/Microphysics/KinematicModel.jl:206
 [14] ClimateMachine.SolverConfiguration(::Float64, ::Float64, ::ClimateMachine.DriverConfiguration{Float64}; init_on_cpu::Bool, ode_solver_type::ClimateMachine.ExplicitSolverType, ode_dt::Float64, modeldata::Nothing, Courant_number::Nothing, diffdir::EveryDirection, timeend_dt_adjust::Bool, CFL_direction::EveryDirection) at /home/jekozdon/codes/CLIMA2/src/Driver/solver_configs.jl:137
 [15] main() at /home/jekozdon/codes/CLIMA2/test/Atmos/Parameterizations/Microphysics/warm_rain.jl:335
 [16] top-level scope at /home/jekozdon/codes/CLIMA2/test/Atmos/Parameterizations/Microphysics/warm_rain.jl:443
 [17] include(::Module, ::String) at ./Base.jl:377
 [18] exec_options(::Base.JLOptions) at ./client.jl:288
 [19] _start() at ./client.jl:484
in expression starting at /home/jekozdon/codes/CLIMA2/test/Atmos/Parameterizations/Microphysics/warm_rain.jl:443

[trontrytel]

Sorry I should have said. I was testing it on CPU first. And the slowdown is on CPU.

[jkozdon]

(possibly) unrelated side note: I do see in the CLIMAParameters lots of 2. and 1.0 stuff. These should all just be integers.

I believe that x^2 and x^2.0 will result in different code.

[trontrytel]

Sorry I should have said. I was testing it on CPU first. And the slowdown is on CPU.

After this bug: https://github.com/JuliaGPU/CUDA.jl/issues/39 I was testing on CPUs

[jkozdon]

All the decimals should probably be converted to rationals as well.

[trontrytel]

(possibly) unrelated side note: I do see in the CLIMAParameters lots of 2. and 1.0 stuff. These should all just be integers.

I believe that x^2 and x^2.0 will result in different code.

When I use CLIMAParameters in ClimateMachine I always take them in as Floats:
_me::FT = me(rain_param_set)

Should I define them as Integers and take them in as Integers if they are in exponents? Does that make a difference for performance?

[jkozdon]

So your code is generating some Anys:

julia> using ClimateMachine.Microphysics: unpack_params
[ Info: Precompiling ClimateMachine [777c4786-024f-11e9-21a3-85d5d4106250]
┌ Warning: Incompatibility detected between CUDA and LLVM 8.0+; disabling debug info emission for CUDA kernels
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:114

julia> include("test/Atmos/Parameterizations/Microphysics/KinematicModel.jl")
^[config_kinematic_eddy (generic function with 1 method)

julia> @code_warntype unpack_params(param_set, rain_param_set, 1.0, 0.5)
Variables
  #self#::Core.Compiler.Const(ClimateMachine.Microphysics.unpack_params, false)
  param_set::Core.Compiler.Const(EarthParameterSet{MicropysicsParameterSet{LiquidParameterSet,IceParameterSet,RainParameterSet,SnowParameterSet}}(MicropysicsParameterSet{LiquidParameterSet,IceParameterSet,RainParameterSet,SnowParameterSet}(LiquidParameterSet(), IceParameterSet(), RainParameterSet(), SnowParameterSet())), false)
  rain_param_set::Core.Compiler.Const(RainParameterSet(), false)
  ρ::Float64
  q_rai::Float64
  _n0::Float64
  _r0::Float64
  _m0::Float64
  _me::Float64
  _a0::Float64
  _ae::Float64
  _v0::Float64
  _ve::Float64
  _χm::Float64
  _Δm::Float64
  _χa::Float64
  _Δa::Float64
  _χv::Float64
  _Δv::Float64

Body::NTuple{14,Float64}
1 ─       nothing
│   %2  = ClimateMachine.Microphysics.n0(rain_param_set)::Core.Compiler.Const(1.6e7, false)
│   %3  = Base.convert($(Expr(:static_parameter, 1)), %2)::Core.Compiler.Const(1.6e7, false)
│         (_n0 = Core.typeassert(%3, $(Expr(:static_parameter, 1))))
│   %5  = ClimateMachine.Microphysics.r0(rain_param_set)::Core.Compiler.Const(0.001, false)
│   %6  = Base.convert($(Expr(:static_parameter, 1)), %5)::Core.Compiler.Const(0.001, false)
│         (_r0 = Core.typeassert(%6, $(Expr(:static_parameter, 1))))
│   %8  = ClimateMachine.Microphysics.m0(param_set, rain_param_set)::Any
│   %9  = Base.convert($(Expr(:static_parameter, 1)), %8)::Any
│         (_m0 = Core.typeassert(%9, $(Expr(:static_parameter, 1))))
│   %11 = ClimateMachine.Microphysics.me(rain_param_set)::Core.Compiler.Const(3, false)
│   %12 = Base.convert($(Expr(:static_parameter, 1)), %11)::Core.Compiler.Const(3.0, false)
│         (_me = Core.typeassert(%12, $(Expr(:static_parameter, 1))))
│   %14 = ClimateMachine.Microphysics.a0(rain_param_set)::Any
│   %15 = Base.convert($(Expr(:static_parameter, 1)), %14)::Any
│         (_a0 = Core.typeassert(%15, $(Expr(:static_parameter, 1))))
│   %17 = ClimateMachine.Microphysics.ae(rain_param_set)::Core.Compiler.Const(2, false)
│   %18 = Base.convert($(Expr(:static_parameter, 1)), %17)::Core.Compiler.Const(2.0, false)
│         (_ae = Core.typeassert(%18, $(Expr(:static_parameter, 1))))
│   %20 = ClimateMachine.Microphysics.v0_rai(param_set, rain_param_set, ρ)::Float64
│   %21 = Base.convert($(Expr(:static_parameter, 1)), %20)::Float64
│         (_v0 = Core.typeassert(%21, $(Expr(:static_parameter, 1))))
│   %23 = ClimateMachine.Microphysics.ve(rain_param_set)::Core.Compiler.Const(0.5, false)
│   %24 = Base.convert($(Expr(:static_parameter, 1)), %23)::Core.Compiler.Const(0.5, false)
│         (_ve = Core.typeassert(%24, $(Expr(:static_parameter, 1))))
│   %26 = ClimateMachine.Microphysics.χm(rain_param_set)::Core.Compiler.Const(1, false)
│   %27 = Base.convert($(Expr(:static_parameter, 1)), %26)::Core.Compiler.Const(1.0, false)
│         (_χm = Core.typeassert(%27, $(Expr(:static_parameter, 1))))
│   %29 = ClimateMachine.Microphysics.Δm(rain_param_set)::Core.Compiler.Const(0, false)
│   %30 = Base.convert($(Expr(:static_parameter, 1)), %29)::Core.Compiler.Const(0.0, false)
│         (_Δm = Core.typeassert(%30, $(Expr(:static_parameter, 1))))
│   %32 = ClimateMachine.Microphysics.χa(rain_param_set)::Core.Compiler.Const(1, false)
│   %33 = Base.convert($(Expr(:static_parameter, 1)), %32)::Core.Compiler.Const(1.0, false)
│         (_χa = Core.typeassert(%33, $(Expr(:static_parameter, 1))))
│   %35 = ClimateMachine.Microphysics.Δa(rain_param_set)::Core.Compiler.Const(0, false)
│   %36 = Base.convert($(Expr(:static_parameter, 1)), %35)::Core.Compiler.Const(0.0, false)
│         (_Δa = Core.typeassert(%36, $(Expr(:static_parameter, 1))))
│   %38 = ClimateMachine.Microphysics.χv(rain_param_set)::Core.Compiler.Const(1, false)
│   %39 = Base.convert($(Expr(:static_parameter, 1)), %38)::Core.Compiler.Const(1.0, false)
│         (_χv = Core.typeassert(%39, $(Expr(:static_parameter, 1))))
│   %41 = ClimateMachine.Microphysics.Δv(rain_param_set)::Core.Compiler.Const(0, false)
│   %42 = Base.convert($(Expr(:static_parameter, 1)), %41)::Core.Compiler.Const(0.0, false)
│         (_Δv = Core.typeassert(%42, $(Expr(:static_parameter, 1))))
│   %44 = Core.tuple(_n0::Core.Compiler.Const(1.6e7, false), _r0::Core.Compiler.Const(0.001, false), _m0, _me::Core.Compiler.Const(3.0, false), _χm::Core.Compiler.Const(1.0, false), _Δm::Core.Compiler.Const(0.0, false), _a0, _ae::Core.Compiler.Const(2.0, false), _χa::Core.Compiler.Const(1.0, false), _Δa::Core.Compiler.Const(0.0, false), _v0, _ve::Core.Compiler.Const(0.5, false), _χv::Core.Compiler.Const(1.0, false), _Δv::Core.Compiler.Const(0.0, false))::Core.Compiler.PartialStruct(NTuple{14,Float64}, Any[Core.Compiler.Const(1.6e7, false), Core.Compiler.Const(0.001, false), Float64, Core.Compiler.Const(3.0, false), Core.Compiler.Const(1.0, false), Core.Compiler.Const(0.0, false), Float64, Core.Compiler.Const(2.0, false), Core.Compiler.Const(1.0, false), Core.Compiler.Const(0.0, false), Float64, Core.Compiler.Const(0.5, false), Core.Compiler.Const(1.0, false), Core.Compiler.Const(0.0, false)])
└──       return %44

[jkozdon]

So your code is generating some Anys:

Particularly the calls to m0 and a0

[jkozdon]

Little more introspection:

julia> @code_warntype ClimateMachine.Microphysics.a0(rain_param_set)
Variables
  #self#::Core.Compiler.Const(CLIMAParameters.Atmos.Microphysics.a0, false)
  ps_rain::Core.Compiler.Const(RainParameterSet(), false)

Body::Any
1 ─ %1 = Base.getproperty(CLIMAParameters.Microphysics, :r0)::Any
│   %2 = (%1)(ps_rain)::Any
│   %3 = Base.getproperty(CLIMAParameters.Microphysics, :ae)::Any
│   %4 = (%3)(ps_rain)::Any
│   %5 = (%2 ^ %4)::Any
│   %6 = (CLIMAParameters.π * %5)::Any
└──      return %6

julia> @code_warntype ClimateMachine.Microphysics.m0(param_set, rain_param_set)
Variables
  #self#::Core.Compiler.Const(CLIMAParameters.Atmos.Microphysics.m0, false)
  ps::Core.Compiler.Const(EarthParameterSet{MicropysicsParameterSet{LiquidParameterSet,IceParameterSet,RainParameterSet,SnowParameterSet}}(MicropysicsParameterSet{LiquidParameterSet,IceParameterSet,RainParameterSet,SnowParameterSet}(LiquidParameterSet(), IceParameterSet(), RainParameterSet(), SnowParameterSet())), false)
  ps_rain::Core.Compiler.Const(RainParameterSet(), false)

Body::Any
1 ─ %1  = (4 / 3.0)::Core.Compiler.Const(1.3333333333333333, false)
│   %2  = CLIMAParameters.Planet::Core.Compiler.Const(CLIMAParameters.Planet, false)
│   %3  = Base.getproperty(%2, :ρ_cloud_liq)::Core.Compiler.Const(CLIMAParameters.Planet.ρ_cloud_liq, false)
│   %4  = (%3)(ps)::Core.Compiler.Const(1000.0, false)
│   %5  = Base.getproperty(CLIMAParameters.Microphysics, :r0)::Any
│   %6  = (%5)(ps_rain)::Any
│   %7  = Base.getproperty(CLIMAParameters.Microphysics, :me)::Any
│   %8  = (%7)(ps_rain)::Any
│   %9  = (%6 ^ %8)::Any
│   %10 = (%1 * CLIMAParameters.π * %4 * %9)::Any
└──       return %10

[charleskawczynski]

When I try to run with Julia 1.4 I get the following errors: ...

Do you still have the branch that this was run on?

[jkozdon]

Do you still have the branch that this was run on?

It's Anna's branch that she posted

[trontrytel]

So your code is generating some Anys:

Particularly the calls to m0 and a0

They are both defined as functions of other parameters. For example:
Microphysics.a0(ps_rain::AbstractRainParameterSet) = π * Microphysics.r0(ps_rain)^Microphysics.ae(ps_rain)

Or is it that I should have FT(π) there?

[jkozdon]

Try adding const here:

• https://github.com/CliMA/CLIMAParameters.jl/blob/d00dbc6804f764bb322a7e0e248d1d525cce241c/src/AtmosParameters.jl#L1
• https://github.com/CliMA/CLIMAParameters.jl/blob/d00dbc6804f764bb322a7e0e248d1d525cce241c/src/AtmosParameters.jl#L16

[jkozdon]

Since these aliases are not set const the type of the functions are not always known and constants cannot be propagated

[charleskawczynski]

Ahhhh, I'll update that now. Thanks for tracking that down!

[jkozdon]

Ahhhh, I'll update that now. Thanks for tracking that down!

Are there more of these in the CLIMAParameters?

[trontrytel]

@jkozdon - Thank you!

I'll still go over the decimals and integers that you mentioned.

[charleskawczynski]

Are there more of these in the CLIMAParameters?

Yes, there are a few.

[charleskawczynski]

Maybe this will boost performance elsewhere! Thanks @jkozdon!

[trontrytel]

I confirm that CLIMAParameters fix solves the slowdown issue.

I will keep this issue open a little longer, until I can check the getproperty error from GPU that was reported here.

[trontrytel]

Both the solwdown and the GPU getproperty error are solved by https://github.com/CliMA/CLIMAParameters.jl/pull/27