Climatemachine.jl: GPU scalability results

[smarras79]

Description

I ran several scalability tests and documented the multi-GPU performance for problems of increasing size. Please, look at the table in my gist link:

https://gist.github.com/smarras79/30c3d6fee4ade890e82dbbaf4fae88a0

In the gist I give all the details of the specific branch that was used and that can be used to reproduce the results.

@lcw @kpamnany @simonbyrne @vchuravy can you please help with performance?

Additional context

What the table is telling me is that we need a proper way to automatically optimize the problem setup to maximize the GPU performance. As of now, scaling will be difficult to achieve for any user with little GPU experience and new to Clima performance.

For CLIMA Developers

• [ ] This feature can be added (if it cannot be, explain why in a comment below -- lack of technical expertise, not relevant to the scope of this project, too ambitious)
• [ ] There is a timeline for when this feature can be implemented
• [ ] The feature has been (or will be) implemented (Please link the PR)

[lcw]

What help do you need? I am not sure what you want to optimize about the problem setup.

For context, I did some LES strong scaling tests when @vchuravy and I were working on the overlap of communication and computation the results of which can be found here https://gist.github.com/lcw/9d567e9036f4a748cabd9c8d7e3d74ab#file-commit-md. Basically, you need a large enough problem to see strong scaling (see the changes I made to dycoms in the gist). It also helps if you run with CUDA aware mpi.

[tapios]

@lcw Are the necessary changes to see strong scaling merged into master yet? It seems they should be part of the default configurations for the LES.

[smarras79]

thanks for info @lcw They are useful. I was trying to get to that point with my runs because I didn't know that you had run those tests before.

How do we activate CUDA aware mpi?

[lcw]

@lcw Are the necessary changes to see strong scaling merged into master yet? It seems they should be part of the default configurations for the LES.

They are merged in and part of the default config.

How do we activate CUDA aware mpi?

MPI.jl tries to determine if your MPI implementation has CUDA aware MPI automatically, which works for at least OpenMPI. You can override the automatic detection by setting the environment variable JULIA_MPI_HAS_CUDA to true.

[simonbyrne]

On the cluster currently

module load cuda/10.0 openmpi/4.0.3_cuda-10.0

gives you a CUDA-aware MPI.

[tapios]

On the cluster currently

module load cuda/10.0 openmpi/4.0.3_cuda-10.0

gives you a CUDA-aware MPI.

Is this in the docs somewhere?

[smarras79]

@lcw how do you get those number? Where to set your timing flags in the code?

Using the setup from your gist, I get 1+ second/time step versus yours 0.2 s:

Info: Establishing Atmos LES configuration for DYCOMS
│     precision        = Float64
│     polynomial order = 4
│     domain           = 3820.00 m x3820.00 m x1500.00 m
│     resolution       = 35x35x5
│     MPI ranks        = 4
│     min(Δ_horz)      = 24.43 m
└     min(Δ_vert)      = 3.45 m
[ Info: Initializing DYCOMS
[ Info: Initializing DYCOMS
 22.103780 seconds ( 22.103780 seconds ( 22.053430 seconds ( 22.103780 seconds (128.54 M allocations: 5.446 GiB, 6.77% gc time)
128.98 M allocations: 5.415 GiB, 6.96% gc time)
128.54 M allocations: 5.436 GiB, 7.91% gc time)
128.54 M allocations: 5.416 GiB, 6.95% gc time)
┌ Info: Starting DYCOMS
│     dt              = 1.66667e-02
│     timeend         =     1.00
│     number of steps = 60
└     norm(Q)         = 4.9920518150982323e+09
 65.068015 seconds (103.16 M allocations: 4.726 GiB, 1.56% gc time)
 64.955730 seconds (108.32 M allocations: 4.825 GiB, 1.64% gc time)
 65.068033 seconds (102.97 M allocations: 4.722 GiB, 1.75% gc time)
 65.068038 seconds (108.87 M allocations: 4.840 GiB, 1.72% gc time)

[jkozdon]

I believe that the problem you are running is likely too small to see good scaling.

If you look at the gist @lcw pointed to he is using on the order of 90K elements, whereas you are using at most 3K elements in the gist that you posted.

To see strong scaling on the GPU you likely need to have significantly more elements on each GPU.

[jkozdon]

If you use nvidia-smi you can see how much memory you are using on the GPU, and you want this to be large (I'd start by choosing a resolution that maxes out the memory then throw more GPUs are it to see how it scales).

[smarras79]

If you use nvidia-smi you can see how much memory you are using on the GPU, and you want this to be large (I'd start by choosing a resolution that maxes out the memory then throw more GPUs are it to see how it scales).

I am using @lcw 's setup.

[jkozdon]

If you use nvidia-smi you can see how much memory you are using on the GPU, and you want this to be large (I'd start by choosing a resolution that maxes out the memory then throw more GPUs are it to see how it scales).

I am using @lcw 's setup.

I was pointing out in general and responding to your original posts gist.

[lcw]

My runs we done on an Amazon EC2 instance with 8 V100's. I am not sure what hardware you are trying to run on.

They have also changed the flag names since I ran those results. So now you are going to need something like mpirun -np 2 julia --projectexperiments/AtmosLES/dycoms.jl --monitor-timestep-duration 10steps to get the average wall-clock time per time-step output.

[smarras79]

My runs we done on an Amazon EC2 instance with 8 V100's. I am not sure what hardware you are trying to run on.

They have also changed the flag names since I ran those results. So now you are going to need something like mpirun -np 2 julia --projectexperiments/AtmosLES/dycoms.jl --monitor-timestep-duration 10steps to get the average wall-clock time per time-step output.

let me try with the --monitor-timestep-duration 10steps flag. I'll let you know what I get.
It would be useful if you cloned my code version and do a try with your hardware and see whether you can reproduce the same timing of your gist.

[lcw]

As @jkozdon said, your problem setup is too small to see strong scaling on our hardware.

[smarras79]

As @jkozdon said, your problem setup is too small to see strong scaling on our hardware.

I am reproducing your gist setup. Are the info there incorrect?

[smarras79]

As @jkozdon said, your problem setup is too small to see strong scaling on our hardware.

I am reproducing your gist setup. Are the info there incorrect?

Ok, I see. your top is 2500 which is the old dycoms. I am using 1500. That's a factor of two smaller than yours. My fault. Thanks for checking.

[smarras79]

@lcw @jkozdon @tapios I confirm that I am getting values that are consistent with what Lucas sees (considering the different hardware and the fact that I set up a slightly larger problem):

 Info: Wall-clock time per time-step (statistics across MPI ranks)
│    maximum (s) =    7.5873062420000004e-01
│    minimum (s) =    7.5812997469999999e-01
│    median  (s) =    7.5872877884999990e-01
└    std     (s) =    2.9971089788065455e-04

thanks all for your help

[smarras79]

Closing this because the scalability is referenced in the gist by @lcw