Chapel: slurm-* launcher is calculating an incorrect default value for CHPL_LAUNCHER_CORES_PER_LOCALE

[BryantLam]

Related #5095

Chapel 1.20.0 on Cray XC, comm=ugni, launcher=slurm-srun

CHPL_LAUNCHER_CORES_PER_LOCALE isn't set explicitly and is undefined. I think that running a Chapel program like ./app -nl <any number of locales> is causing this launcher error

srun: error: Unable to allocate resources: Requested node configuration is not available

because the launcher is likely calculating an incorrect default value.

This is hard to debug as an error message.

On my system, sinfo reports CPUS=64 and S:C:T=2:16:2 (32 cores per locale). Running ./app -nl <#> --verbose shows that --cpus-per-task=64 is being used, which oversubscribes the node and emits the launcher error.

Workaround is to set CHPL_LAUNCHER_CORES_PER_LOCALE explicitly to 32 or less.

[bradcray]

@gbtitus , @ronawho : Is this behavior familiar to either of you?

[ronawho]

We gather the cores per locale from sinfo. We grab the first value in sinfo --exact --format=%c --sort=+=#c --noheader --responding, which should give us the least number of cores per node. We usually have the opposite problem where we're not asking slurm for enough cores. @BryantLam can you send the output of that sinfo command?

[BryantLam]

@ronawho -- That sinfo command reports "64"

I believe that command is grabbing the CPUS value, which isn't correct because there's two hyperthreads per core in that value. I don't think CPUS is an accurate value for the number of cores per locale, but I could be mistaken.

[ronawho]

slurm "CPU"s are typically hardware threads / hyperthreads -- https://slurm.schedmd.com/faq.html#cpu_count. We try to ask slurm for the whole node and then we choose which parts of it we want to use (running without hyperthreads by default)

It sounds like hyperthreading is disabled on this system, but sinfo doesn't reflect that?

I think in the past we've tried to set --cpus-per-task=<num_physical_cores> but that has bit us on other systems where it would give us say 16 physical cores and 16 HT, instead of 32 physical cores.

[ronawho]

I'll take a look through some of the slurm docs to see if there's a programmatic way to check if hyperthreading has been disabled.

[BryantLam]

It sounds like hyperthreading is disabled on this system, but sinfo doesn't reflect that?

This is likely true. Do you know how I would verify hyperthreading was disabled on the system?

[ronawho]

scontrol show node | grep ThreadsPerCore would be my current guess. But if sinfo is still showing 2 threads, I'm assuming that will show 2 as well.

Something in slurm must know that hyperthreads are disabled though since the submission with --cpus-per-task=64 failed.

[gbtitus]

scontrol show node | grep ThreadsPerCore would be my current guess. But if sinfo is still showing 2 threads, I'm assuming that will show 2 as well.

@BryantLam can you confirm that this scontrol command does indeed show 2 threads per core on the system in question?

[BryantLam]

scontrol show node | grep ThreadsPerCore reports "2"

[gbtitus]

What does srun lstopo show? And also (just for confirmation, but more complicated) what does srun $(which srun | sed 's,/bin/srun,/sbin/slurmd,') -C show? (This is asking the slurm compute node daemon what its actual hardware configuration is, but that daemon is in a place not normally in PATH, thus the complication.)

Edit: actually all we really need to know is whether lstopo shows 1 or 2 threads per core, and also what the ThreadsPerCore value printed by that slurmd command is. There's no need to quote the entire output in either case, I don't believe.

[BryantLam]

Both are as expected. lstopo -s and slurmd -C on the compute node show 2 PUs or ThreadsPerCore.

The system has this environment variable being set: SLURM_HINT=nomultithread. I don't have a good intuition for handling this case in the slurm launcher.

[gbtitus]

The system has this environment variable being set: SLURM_HINT=nomultithread.

Ah, that's what's doing it. Thanks! We're discussing solutions.

[ronawho]

@BryantLam could you see if this PR works for you: https://github.com/chapel-lang/chapel/pull/15176

(I was able to reproduce your error message by manually setting SLURM_HINT=nomultithread and CHPL_LAUNCHER_CORES_PER_LOCALE, but it's harder for me to test the default calculation since we don't have access to homogenous slurm systems (internal machines tend to have a wide range of processors so we can test more configs/cpus.)

I can provide the full branch or the diff against 1.20 if that's easier.

[BryantLam]

15176 looks like it works. I didn't test that rigorously though. Mostly just ./some_program -nl 4 --verbose to verify that the correct --cpus-per-task was being passed through.