Climatemachine.jl: Interpolation fails at Float32 precision

[kpamnany]

Change this to Float32 and the brick interpolation test fails. Change this to Float32 and the cubed sphere interpolation test fails.

Interpolation must work with either single or double precision.

[kpamnany]

Cc: @bischtob

[simonbyrne]

We should make sure to run the tests at both precisions.

[jkozdon]

The tolerance is probably set too small for Float32.

Also probably want a relative error check, or to use isapprox with appropriately chosen atol and rtol.

In general, based on previous Float32 problems, I think we need to be careful to build up confidence that the code can be trusted running in Float32. We've seen some weirdness before with Float32 (#505, #713, #669, #81)

[skandalaCLIMA]

Yes. Tolerance needs to be increased slightly for Float32 runs. Tests for Float64 and Float32 are now explicitly added. Multiple MPI process runs work fine on the CPU for both Float32 and Float64. It is also tested successfully for both Float32 and Float64 on a single GPU.

I will run these tests on multiple GPUs tomorrow.

[smarras79]

Based on what is done in CFD, I would suggest that anything that deals with interpolation and grid generation or adaptivity should be done in Float64 regardless of what precision is used for the calculations.

[skandalaCLIMA]

The tests run successfully for Float32 and Float64 on 1, 2 & 4 GPUs on the cluster. I believe this resolves the issue.

[simonbyrne]

Fixed by #782.