Climatemachine.jl: NetCDF: cannot open shared object file

[akshaysridhar]

Description

When using NCDatasets within an experiment file on the Caltech Central HPC system, occasional failures with the following stack trace are seen:

2 ERROR: LoadError: error compiling main: error compiling get_gcm_info: error compiling #NCDataset#9: error compiling nc_open: could not load library "/lib64/libnetcdf.so"
  3 /lib64/libnetcdf.so: cannot open shared object file: No such file or directory

Steps to reproduce

On branch
as/hadgem-sandbox
run
sbatch run_hadgem.sh
which launches an array job on the SLURM queue. This has been reproduced on Caltech cluster node 89-38. The detailed stack trace is here : https://gist.github.com/akshaysridhar/210cff7924d8feca8b752163ad5e089c

Expected behavior

The same experiment source file is being used to launch (in this specific instance) 7 jobs in a SLURM-ARRAY, I expect all of those to be launched identically [In this instance of the issue on hpc-89-38, I see 4 jobs progress and 3 fail with the above trace].

Expected fix

Include instruction module load hdf5/1.10.1 netcdf-c/4.6.1. (On most nodes on Caltech HPC we can run CLIMA with NCDatasets without any of this). Working on testing this as a general solution, following a conversation with @kpamnany. Will close this issue if the fix works across multiple Caltech Central nodes.

Add any other context about the problem here.

For CLIMA developers

• [ ] The bug can be reproduced
• [ ] Labels have been added and developers assigned
• [ ] The bug can be fixed (if not, please explain why not in a comment below)
• [ ] There is a timeline to fix the bug
• [ ] The bug has been fixed (Please link the PR)

[simonbyrne]

Blegh, NCDatasets keeps causing issues...

I think what should work is to make sure that you

module load hdf5/1.10.1 netcdf-c/4.6.1

_before_ calling Pkg.build(), as that is when it resolves the path. What you should then see is:

julia> using NCDatasets
[ Info: Precompiling NCDatasets [85f8d34a-cbdd-5861-8df4-14fed0d494ab]

julia> NCDatasets.libnetcdf
"/central/software/netcdf-c/4.6.1//lib/libnetcdf.so"

The library in /lib64/libnetcdf.so appears to be older (4.3.3.1): I'm not sure why that is installed on the images.

[simonbyrne]

I've updated https://github.com/CliMA/ClimateMachine.jl/wiki/Caltech-Central-Cluster#sample-sbatch-scripts to reflect this.

[kpamnany]

IIRC there was also a problem with running ldconfig at some point in relation to this issue, for which we added /usr/sbin to the PATH?

Cc: @leios

[simonbyrne]

@akshaysridhar can you post a link to your sbatch script?

[simonbyrne]

IIRC there was also a problem with running ldconfig at some point in relation to this issue, for which we added /usr/sbin to the PATH?

I think that had to do with how cron jobs were launched.

[kpamnany]

Akshay's script.

I suggested:

setup_hadgem_gcm_les.sh:

#!/bin/bash

#SBATCH --job-name=setup_hadgem_gcm_les
#SBATCH --gres=gpu:1
#SBATCH --time=1:00:00
#SBATCH [email protected]
#SBATCH --mail-type=ALL
#SBATCH --output=/groups/esm/asridhar/GCP/setup_hadgem_gcm_les.out

set -euo pipefail
set -x
hostname

module load julia/1.3.0
module load cuda/10.0 openmpi/4.0.3_cuda-10.0 hdf5/1.10.1 netcdf-c/4.6.1

export JULIA_DEPOT_PATH=/home/asridhar/CLIMA/.julia_depot
export JULIA_MPI_BINARY=system
export OPENBLAS_NUM_THREADS=1
export PATH="/usr/sbin:$PATH"

julia --project=/home/asridhar/CLIMA -e 'using Pkg; Pkg.instantiate(); Pkg.build()'
julia --project=/home/asridhar/CLIMA -e 'using Pkg; Pkg.precompile()'

cat /home/asridhar/CLIMA/Manifest.toml

And:

run_hadgem_gcm_les.sh:

#!/bin/bash

#SBATCH --job-name=site%a_minnodedist
#SBATCH --gres=gpu:1
#SBATCH --time=12:10:00
#SBATCH [email protected]
#SBATCH --mail-type=ALL
#SBATCH --array=17,19,20
#SBATCH --output=/groups/esm/asridhar/GCP/out_%j_site%a_MR5-10.5_V.out

set -euo pipefail
set -x
hostname

module load julia/1.3.0
module load cuda/10.0 openmpi/4.0.3_cuda-10.0 hdf5/1.10.1 netcdf-c/4.6.1

export JULIA_DEPOT_PATH=/home/asridhar/CLIMA/.julia_depot
export JULIA_MPI_BINARY=system
export OPENBLAS_NUM_THREADS=1
export PATH="/usr/sbin:$PATH"

mpirun julia --project=/home/asridhar/CLIMA /home/asridhar/CLIMA/experiments/AtmosLES/hadgem_gcm_les.jl --diagnostics 5smins --output-dir=/central/groups/esm/asridhar/GCP/Site${SLURM_ARRAY_TASK_ID}_MR5-10_V --group-id=site${SLURM_ARRAY_TASK_ID} --vtk 30smins

[simonbyrne]

One thing I've noticed is that the PATH variables can get messed up if modules are loaded on the login node. I would suggest using module purge before loading the job-specific modules.

[akshaysridhar]

@simonbyrne @kpamnany Thanks, your suggested changes appear to have fixed this issue across multiple runs [on a range of nodes including the problematic one listed above].

[trontrytel]

The feeling when you google an issue you have and the first thing that pops up is a solution waiting for you. Already there in the codebase you help write. This is awesome. Thank you guys!