Climatemachine.jl: Held Suarez is broken

[LenkaNovak]

Fresh after a git clone ...:

elencz@hpc-22-11:/groups/esm/lenka/CLIMA$julia --project
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.3.0 (2019-11-26)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> include("experiments/Atmos_GCM/heldsuarez.jl")
[ Info: Precompiling CLIMA [777c4786-024f-11e9-21a3-85d5d4106250]
[ Info: CUDAdrv.jl failed to initialize, GPU functionality unavailable (set JULIA_CUDA_SILENT or JULIA_CUDA_VERBOSE to silence or expand this message)
┌ Warning: CUDA is installed, but fails to load (error code 100)
└ @ CUDAapi ~/.julia/packages/CUDAapi/K94wY/src/availability.jl:53
[1581632844.998379] [hpc-22-11:334807:0]         parser.c:1369 UCX  WARN  unused env variable: UCX_MEM_MMAP_RELOC (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
┌ Warning: CUDA is installed, but fails to load (error code 100)
└ @ CUDAapi ~/.julia/packages/CUDAapi/K94wY/src/availability.jl:53
[hpc-22-11:334807:0:334807] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x440000e8)
==== backtrace ====
 0 0x000000000006a42b MPI_Comm_rank()  ???:0
 1 0x000000000005d3fc _jl_invoke()  /buildworker/worker/package_linux64/build/src/gf.c:2141
 2 0x00000000001d6a8e jl_apply()  /buildworker/worker/package_linux64/build/src/julia.h:1631
 3 0x00000000001d66e0 eval_value()  /buildworker/worker/package_linux64/build/src/interpreter.c:417
 4 0x00000000001d7214 eval_stmt_value()  /buildworker/worker/package_linux64/build/src/interpreter.c:368
 5 0x00000000001d7214 eval_stmt_value()  /buildworker/worker/package_linux64/build/src/interpreter.c:369
 6 0x00000000001d7214 eval_body()  /buildworker/worker/package_linux64/build/src/interpreter.c:778
 7 0x00000000001d7a1f jl_interpret_toplevel_thunk_callback()  /buildworker/worker/package_linux64/build/src/interpreter.c:888
 8 0x00000000000748ac Lenter_interpreter_frame_start_val()  interpreter.c:0
 9 0x00000000001d83ed jl_interpret_toplevel_thunk()  /buildworker/worker/package_linux64/build/src/interpreter.c:897
10 0x0000000000091c76 jl_toplevel_eval_flex()  /buildworker/worker/package_linux64/build/src/toplevel.c:814
11 0x000000000006811e jl_parse_eval_all()  /buildworker/worker/package_linux64/build/src/ast.c:873
12 0x0000000000092ab3 jl_load()  /buildworker/worker/package_linux64/build/src/toplevel.c:878
13 0x0000000000d78086 include()  ./boot.jl:328
14 0x0000000000d78086 japi1_include_relative_3537()  ./loading.jl:1105
15 0x000000000074a5ed japi1_include_3734()  ./Base.jl:31
16 0x000000000005d79c _jl_invoke()  /buildworker/worker/package_linux64/build/src/gf.c:2135
17 0x000000000005d3fc _jl_invoke()  /buildworker/worker/package_linux64/build/src/gf.c:2141
18 0x00000000001d6a8e jl_apply()  /buildworker/worker/package_linux64/build/src/julia.h:1631
19 0x00000000001d66e0 eval_value()  /buildworker/worker/package_linux64/build/src/interpreter.c:417
20 0x00000000001d7214 eval_stmt_value()  /buildworker/worker/package_linux64/build/src/interpreter.c:368
21 0x00000000001d7214 eval_stmt_value()  /buildworker/worker/package_linux64/build/src/interpreter.c:369
22 0x00000000001d7214 eval_body()  /buildworker/worker/package_linux64/build/src/interpreter.c:778
23 0x00000000001d7a1f jl_interpret_toplevel_thunk_callback()  /buildworker/worker/package_linux64/build/src/interpreter.c:888
24 0x00000000000748ac Lenter_interpreter_frame_start_val()  interpreter.c:0
25 0x00000000001d83ed jl_interpret_toplevel_thunk()  /buildworker/worker/package_linux64/build/src/interpreter.c:897
26 0x0000000000091c76 jl_toplevel_eval_flex()  /buildworker/worker/package_linux64/build/src/toplevel.c:814
27 0x000000000009179a jl_toplevel_eval_flex()  /buildworker/worker/package_linux64/build/src/toplevel.c:764
28 0x00000000000928d5 jl_toplevel_eval_in()  /buildworker/worker/package_linux64/build/src/toplevel.c:843
29 0x000000000074ab02 japi1_eval_3749()  ./boot.jl:330
30 0x000000000005d79c _jl_invoke()  /buildworker/worker/package_linux64/build/src/gf.c:2135
31 0x0000000000ec2ad4 japi1_eval_user_input_11470()  /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/REPL/src/REPL.jl:86
32 0x0000000000ec2dc9 macro expansion()  /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/REPL/src/REPL.jl:118
33 0x0000000000ec2dc9 japi1_#26_11468()  ./task.jl:333
34 0x000000000005d79c _jl_invoke()  /buildworker/worker/package_linux64/build/src/gf.c:2135
35 0x0000000000078e79 jl_apply()  /buildworker/worker/package_linux64/build/src/julia.h:1631
===================
Segmentation fault (core dumped)
elencz@hpc-22-11:/groups/esm/lenka/CLIMA$

[kpamnany]

This looks like it's an interactive session on Central. Did you follow these instructions?

Also, you must:

$ module load cuda/10.0 openmpi/4.0.1_cuda-10.0

Before you start Julia. To make sure all is well, add a:

julia> ]build MPI

Before doing the include (not every time, just to make sure MPI.jl is built with CUDA support).

For me, on current master:

hpc-23-30:~/CLIMA$ julia --project
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.3.0 (2019-11-26)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> include("experiments/Atmos_GCM/heldsuarez.jl")
[ Info: Initializing HeldSuarez
┌ Info: Starting HeldSuarez
│ dt                      = 5.12821e-01
│ timeend                 = 6.00000e+01
└ norm(Q)                 = 8.2546082716647938e+13
┌ Info: Finished
│ norm(Q)                 = 8.2544572364610203e+13
│ norm(Q) / norm(Q₀)      = 9.9998170292292465e-01
└ norm(Q) - norm(Q₀)      = -1.5103520377343750e+09
┌ Info: Euclidean distance
│ norm(Q - Qe)            = 1.5847175661781528e+11
└ norm(Q - Qe) / norm(Qe) = 1.9197974198460008e-03
0.9999817029229247

[akshaysridhar]

@kpamnany
Is there a preference between using mpich vs openmpi ?
I imagine this shouldn't matter as long as the same module used during the build is loaded at runtime ?
This runs on CentralHPC (interactive session) with
1) julia/1.3.0 2) openmpi/4.0.1 3) cuda/10.0 4) cmake/3.10.2

[kpamnany]

OpenMPI is recommended by NVIDIA -- their CUDA-awareness is best.

[akshaysridhar]

OpenMPI is recommended by NVIDIA -- their CUDA-awareness is best.

Perhaps it's best to change the README.md instructions then (which suggest mpich usage) ?

[kpamnany]

You meant on the wiki I assume. I edited it just now.

[bischtob]

This is the same issue as #707 .

[LenkaNovak]

Akshay and I have updated all the modules, but we're still having issues:

Precompiling CLIMA
[ Info: Precompiling CLIMA [777c4786-024f-11e9-21a3-85d5d4106250]
ERROR: LoadError: LoadError: could not open file /home/elencz/.julia/packages/MPI/hcbnk/deps/consts.jl
Stacktrace:
[1] include at ./boot.jl:328 [inlined]
[2] include_relative(::Module, ::String) at ./loading.jl:1105
[3] include at ./Base.jl:31 [inlined]
[4] include(::String) at /home/elencz/.julia/packages/MPI/hcbnk/src/MPI.jl:1
[5] top-level scope at /home/elencz/.julia/packages/MPI/hcbnk/deps/deps.jl:8
[6] include at ./boot.jl:328 [inlined]
[7] include_relative(::Module, ::String) at ./loading.jl:1105
[8] include at ./Base.jl:31 [inlined]
[9] include(::String) at /home/elencz/.julia/packages/MPI/hcbnk/src/MPI.jl:1
[10] top-level scope at /home/elencz/.julia/packages/MPI/hcbnk/src/MPI.jl:40
[11] include at ./boot.jl:328 [inlined]
[12] include_relative(::Module, ::String) at ./loading.jl:1105
[13] include(::Module, ::String) at ./Base.jl:31
[14] top-level scope at none:2
[15] eval at ./boot.jl:330 [inlined]
[16] eval(::Expr) at ./client.jl:425
[17] top-level scope at ./none:3
in expression starting at /home/elencz/.julia/packages/MPI/hcbnk/deps/deps.jl:8
in expression starting at /home/elencz/.julia/packages/MPI/hcbnk/src/MPI.jl:40
ERROR: LoadError: LoadError: Failed to precompile MPI [da04e1cc-30fd-572f-bb4f-1f8673147195] to /home/elencz/.julia/compiled/v1.3/MPI/nO0XF_C91At.ji.
Stacktrace:
[1] error(::String) at ./error.jl:33
[2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1283
[3] _require(::Base.PkgId) at ./loading.jl:1024
[4] require(::Base.PkgId) at ./loading.jl:922
[5] require(::Module, ::Symbol) at ./loading.jl:917
[6] include at ./boot.jl:328 [inlined]
[7] include_relative(::Module, ::String) at ./loading.jl:1105
[8] include at ./Base.jl:31 [inlined]
[9] include(::String) at /central/groups/esm/lenka/CLIMA/src/CLIMA.jl:1
[10] top-level scope at /central/groups/esm/lenka/CLIMA/src/CLIMA.jl:14
[11] include at ./boot.jl:328 [inlined]
[12] include_relative(::Module, ::String) at ./loading.jl:1105
[13] include(::Module, ::String) at ./Base.jl:31
[14] top-level scope at none:2
[15] eval at ./boot.jl:330 [inlined]
[16] eval(::Expr) at ./client.jl:425
[17] top-level scope at ./none:3
in expression starting at /central/groups/esm/lenka/CLIMA/src/Arrays/MPIStateArrays.jl:9
in expression starting at /central/groups/esm/lenka/CLIMA/src/CLIMA.jl:14

[LenkaNovak]

The problem is that /home/elencz/.julia/packages/MPI/hcbnk/deps/ doesn't have consts.jl. It only has consts_msmpi.jl. Changing this filename in /home/elencz/.julia/packages/MPI/hcbnk/deps/deps.jl:8 then lets you precompile, but then you get the segmentation error above when doing include("heldsuarez.jl")

[charleskawczynski]

The good news is that this does seem to be a configuration problem since @kpamnany has this running with his configuration.

[kpamnany]

Wipe and recreate the Julia package repo maybe? rm -rf ~/.julia and then instantiate while on a node, to make sure the correct libraries are picked up.

[akshaysridhar]

Wipe and recreate the Julia package repo maybe? rm -rf ~/.julia and then instantiate while on a node, to make sure the correct libraries are picked up.

Doesn't seem to help: returns to the error pointed out above with /home/<username>/.julia/packages/MPI/hcbnk/deps/consts.jl

[bischtob]

Worked for me.

[akshaysridhar]

Broken node ? >:D
Successfully able to build MPI on specific nodes :D

[bischtob]

Looks like a cluster-specific issue. Need to set MPI paths and modules and it works.

[simonbyrne]

The problem is that /home/elencz/.julia/packages/MPI/hcbnk/deps/ doesn't have consts.jl. It only has consts_msmpi.jl. Changing this filename in /home/elencz/.julia/packages/MPI/hcbnk/deps/deps.jl:8 then lets you precompile, but then you get the segmentation error above when doing include("heldsuarez.jl")

That won't work (those constants are for a different MPI implementation). In most cases this issue can be fixed by running ]build MPI.

[simonbyrne]

This is something I want to fix in MPI.jl