Oceananigans.jl: Run on GPU without network access

[pnavaro]

Dear Oceananigans developers,
I want to run an example on a HPC server where GPU nodes have no network access. Is it possible to force downloads on the frontend even if GPU is not available ?
Thanks

[glwagner]

@pnavaro I believe we have been able to run Oceananigans in this scenario. I can't remember exactly what was done (@ali-ramadhan may have more insight).

Have you tried running Pkg.instantiate() in an environment that contains Oceananigans on the login node? I believe this should download the software that you need (despite that precompilation will fail if the login node does not have a GPU).

If you can get access to the GPU node interactively at the same time that you have access to the login node I think that might help a lot with debugging.

[pnavaro]

Yes i tried that, i launch the tests on the login node and everything works fine. But when i am connected interactively to the GPU, if i do

julia> using Oceananigans
[ Info: Precompiling Oceananigans [9e8cae18-63c1-5223-a75c-80ca9d6e9a09]
         Downloading artifact: CUDA101

and the download is not possible...

[ali-ramadhan]

Hi @pnavaro thanks for opening this issue. Yeah this is an issue on some clusters so it might be worth adding a note in the documentation.

In my case I think the login node had a GPU so I was able to build on the login node then just precompile and run on the compute node without internet access. But this was also before CUDA.jl which very nicely downloads CUDA artifacts for you so it just relied on the local CUDA toolkit.

Do you have access to the CUDA toolkit on the login node via something like module load cuda? Might allow you to build on the login node without a GPU but might be weird to have CUDA but no GPU.

Your compute node must have a CUDA installation. Looks like the CUDA.jl documentation has a section on how to make use of a local CUDA installation: https://juliagpu.gitlab.io/CUDA.jl/installation/overview/#CUDA-toolkit

If you can find where the CUDA toolkit is installed on your compute node, then maybe all you have to do is set one of the CUDA_HOME, CUDA_ROOT or CUDA_PATH environment variables?

Looks like creating a container with CUDA.jl is an option but it still requires a CUDA toolkit at runtime: https://juliagpu.gitlab.io/CUDA.jl/installation/overview/#Containers

Let us know if this helps or if you're able to get up and running on your compute node.

Would be good to figure this out and add to the documentation. The #gpu channel on Julia's Slack or the GPU section of the Julia Discourse might also be good places to ask if we can't figure it out.

[pnavaro]

Thanks for your answer, i made some progress by using the JULIA_CUDA_USE_BINARYBUILDER env variable.
The download problem is fixed and CUDA.jl tests passed on the gpu node but i still have some errors with Oceaningans.

module load julia cuda
env JULIA_CUDA_USE_BINARYBUILDER=false julia —project
julia> using Oceananigans
julia> grid = RegularCartesianGrid(size=(100, 100, 50), extent=(2π, 2π, 1))
RegularCartesianGrid{Float64, Periodic, Periodic, Bounded}
                   domain: x ∈ [-5.496153587253255e-18, 6.283185307179586], y ∈ [-5.496153587253255e-18, 6.283185307179586], z ∈ [-1.0, 1.7080354225002348e-17]
                 topology: (Periodic, Periodic, Bounded)
  resolution (Nx, Ny, Nz): (100, 100, 50)
   halo size (Hx, Hy, Hz): (1, 1, 1)
grid spacing (Δx, Δy, Δz): (0.06283185307179587, 0.06283185307179587, 0.02)

julia> model = IncompressibleModel(architecture=GPU(), grid=grid)
ERROR: MethodError: no method matching plan_forward_transform(::CUDA.CuArray{Complex{Float64},3}, ::BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}, ::Array{Int64,1})

[ali-ramadhan]

Ah can't say I've seen that error before.

Which version of Julia are you using? Can you also post the details of your environment, i.e. the output of ] status at the REPL or using Pkg; Pkg.status()?

[pnavaro]

~~~
(Oceananigans) pkg> status
Project Oceananigans v0.40.0
Status /gpfs7kw/linkhome/rech/genbes01/use25mx/Oceananigans.jl/Project.toml
[79e6a3ab] Adapt v2.0.2
[052768ef] CUDA v1.3.3
[a8cc5b0e] Crayons v4.0.4
[7a1cc6ca] FFTW v1.2.4
[033835bb] JLD2 v0.1.14
[63c18a36] KernelAbstractions v0.4.0
[85f8d34a] NCDatasets v0.10.4
[6fe1bfb0] OffsetArrays v1.1.3
[bac558e1] OrderedCollections v1.3.0
[1bc83da4] SafeTestsets v0.0.1
[d496a93d] SeawaterPolynomials v0.2.0
[90137ffa] StaticArrays v0.12.4
[ade2ca70] Dates
[b77e0a4c] InteractiveUtils
[37e2e46d] LinearAlgebra
[56ddb016] Logging
[44cfe95a] Pkg
[de0858da] Printf
[9a3f8284] Random
[10745b16] Statistics

shell> module list
Currently Loaded Modulefiles:
1) cuda/10.2 2) julia/1.5.2
~~~

[pnavaro]

~~~
bash-4.4$ nvidia-smi
Tue Oct 13 13:55:11 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00 Driver Version: 440.64.00 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... Off | 00000000:88:00.0 Off | 0 |
| N/A 45C P0 45W / 300W | 0MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
~~~

[ali-ramadhan]

Ah I see you're using the master branch of Oceananigans.jl with Julia 1.5.2. We'd usually recommend using a tagged release installed via the package manager, ] add Oceananigans, since the master branch could break if we accidentally merge in a bug.

Also, since you're running on a GPU I think you have to use Julia 1.4 for now. See #828 which should be fixed soon (see #997). So hopefully we'll release a new version, v0.41.0, soon which will work with Julia 1.5.

But your error is not due to Julia 1.5.

Can you post the output of

using CUDA
CUDA.has_cuda()

and

using Oceananigans.Architectures: @hascuda
@hascuda "Hi"

?

[pnavaro]

Sorry but sometimes i must wait a long time to get gpu resources :
I created a TestOcean project with Oceananigans as dependency. I used Julia v1.4
and it works perfectly, thank you very much!

~~~
module load julia/1.4.0 cuda
cd TestOcean
env JULIA_CUDA_USE_BINARYBUILDER=false julia --project
julia> using Oceananigans
[ Info: Precompiling Oceananigans [9e8cae18-63c1-5223-a75c-80ca9d6e9a09]
julia> using CUDA
julia> CUDA.has_cuda()
true
julia> using Oceananigans.Architectures: @hascuda
julia> @hascuda "Hi"
"Hi"
julia> grid = RegularCartesianGrid(size=(100, 100, 50), extent=(2π, 2π, 1))
RegularCartesianGrid{Float64, Periodic, Periodic, Bounded}
domain: x ∈ [-5.496153587253255e-18, 6.283185307179586], y ∈ [-5.496153587253255e-18, 6.283185307179586], z ∈ [-1.0, 1.7080354225002348e-17]
topology: (Periodic, Periodic, Bounded)
resolution (Nx, Ny, Nz): (100, 100, 50)
halo size (Hx, Hy, Hz): (1, 1, 1)
grid spacing (Δx, Δy, Δz): (0.06283185307179587, 0.06283185307179587, 0.02)

julia> model = IncompressibleModel(architecture=GPU(), grid=grid)
IncompressibleModel{GPU, Float64}(time = 0.000 s, iteration = 0)
├── grid: RegularCartesianGrid{Float64, Periodic, Periodic, Bounded}(Nx=100, Ny=100, Nz=50)
├── tracers: (:T, :S)
├── closure: IsotropicDiffusivity{Float64,NamedTuple{(:T, :S),Tuple{Float64,Float64}}}
├── buoyancy: SeawaterBuoyancy{Float64,LinearEquationOfState{Float64},Nothing,Nothing}
└── coriolis: Nothing

julia> simulation = Simulation(model, Δt=60, stop_time=3600)
Simulation{IncompressibleModel{GPU, Float64}}
├── Model clock: time = 0.000 s, iteration = 0
├── Next time step (Int64): 1.000 min
├── Iteration interval: 1
├── Stop criteria: Any[Oceananigans.Simulations.iteration_limit_exceeded, Oceananigans.Simulations.stop_time_exceeded, Oceananigans.Simulations.wall_time_limit_exceeded]
├── Run time: 0.000 s, wall time limit: Inf
├── Stop time: 1.000 hr, stop iteration: Inf
├── Diagnostics: OrderedCollections.OrderedDict with no entries
└── Output writers: OrderedCollections.OrderedDict with no entries

julia> run!(simulation)
[ Info: Simulation is stopping. Model time 1.000 hr has hit or exceeded simulation stop time 1.000 hr.
~~~

[ali-ramadhan]

Oh haha that's great, glad you got it working!

Usually the IncompressibleModel constructor will throw an error if you try to create a GPU model without a GPU.

But I think the reason you hit

ERROR: MethodError: no method matching plan_forward_transform(::CUDA.CuArray{Complex{Float64},3}, ::BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}, ::Array{Int64,1})

is because maybe you precompiled Oceananigans.jl in an environment without a GPU (so @hascuda does not evaluate expressions) but then ran in an environment with a GPU. So IncompressibleModel did not throw an error but plan_forward_transform(::CuArray, ...) was never defined.

This is probably our fault for still defining functions inside @hascuda macros, a relic from when CUDA did not work on machines without a GPU.

Now that CUDA.jl can be imported on machines without a GPU I think we can stop defining GPU-specific methods in @hascuda.

[pnavaro]

So JULIA_CUDA_USE_BINARYBUILDER=false solves the issue. Fantastic work, thanks, wish you the best.