Aws-parallelcluster: sbatch: error: Memory specification can not be satisfied. Found 1 host with 8 cores and 0 GB memory under Slurm control.

[chandanbfx]

Environment:

• AWS ParallelCluster / CfnCluster version [e.g. aws-parallelcluster-2.5.0]
• OS: [ ubuntu1804]
• Scheduler: [e.g. slurm 19.05.3-2]
• Master instance type: [e.g. md5.2xlarge]
• Compute instance type: [e.g. md5.2xlarge]

Hi

I tried modifying slurm.conf and added SelectType=select/cons_res
SelectTypeParameters=CR_CPU_Memory DefMemPerCPU=3000 but the same error persists either from the command line or script file.

from the script: sbatch: error: Memory specification can not be satisfied
sbatch: error: Batch job submission failed: Requested node configuration is not available

from the command line: Found 1 host with 8 cores and 0 GB memory under Slurm control.

$scontrol show partition
PartitionName=compute
AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
AllocNodes=ALL Default=YES QoS=N/A
DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO
MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED
Nodes=dummy-compute[1-19],ip-172-31-83-72
PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO
OverTimeLimit=NONE PreemptMode=OFF
State=UP TotalCPUs=160 TotalNodes=20 SelectTypeParameters=NONE
JobDefaults=(null)
DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED

parallel cluster configuration file: pcluster_config.txt
slurm configuration file: slurm.conf.txt
slurm controller log: slurmctld.log

I am new to parallel cluster and slurm, any suggestion and help very much appreciated. if any other log or details required in order to understand the issue, I will provide.

pcluster_config.txt
slurm.conf.txt
slurmctld.log

thanks

[rexcsn]

Hi,

ParallelCluster is currently not configured to support scheduling with memory options with slurm.
With that said, here is how we could setup slurm manually to use memory options:

• Since you are using ParallelCluster 2.5.0 with slurm 19.05, there is no need to change SelectType from select/cons_tres to select/cons_res.
• Determine the RealMemory available in the compute instance. We can get this by logging into the compute and running /opt/slurm/sbin/slurmd -C, we should see something like RealMemory=491805.
• Edit /opt/slurm/etc/slurm.conf and add NodeName=DEFAULT RealMemory=[RealMemory for compute nodes] BEFORE include slurm_parallelcluster_nodes.conf. Note that ideally we should just use the RealMemory we got from /opt/slurm/sbin/slurmd -C, but RealMemory might be different even for different instances of the same instance type. I would round down RealMemory slightly just to avoid any potential error when the cluster is scaling and launching new instances.
• Restart slurmd on compute nodes and slurmctld on master node, and we should see that memory is configured in scontrol show nodes

Hope that helps! I will mark this issue as an enhancement. Thanks!

[chandanbfx]

Hi, rexcsn

thank you so much for your help. I tried your suggestion and follow the steps you recommend and it's working now. issue fixed!

thanks

[gmoneyomics]

I am having this same issue, could you be more specific about how to login to the compute node? Are you changing the slurm.conf file in the master node or in each compute node?

[rakarnik]

I have been following the instructions in this thread successfully for the past month or so. To answer your specific questions:

1. Login to the compute node using "srun --pty bash". There is no mem specification, so this should give you an interactive session on the compute node.
2. slurm.conf is on the master node. I believe "/opt/slurm" is NFS mounted on the compute nodes, so the changes are visible there.

[rexcsn]

@gmoneyomics I don't know how I missed this thread. Thanks @rakarnik for providing the help!

Suggestion to use srun --pty bash to interact with compute node is super fancy!
I was just referring to something like running sinfo to get a compute node hostname and doing ssh ip-xx-xx-xxx to ssh into the compute and run /opt/slurm/sbin/slurmd -C, but either way works.

Yes /opt/slurm is exported from Master to Compute, so you can edit slurm.conf from any node, just make sure to do it as root.

[rakarnik]

Is there a way to implement this workaround as a custom bootstrap action?

[alexiswl]

Took me a long time to figure out but if you overestimate the RealMemory you will set the partition into drain with the error -

See /var/log/slurmctld.log

[2020-08-13T10:20:03.698] error: Node ip-10-2-1-120 has low real_memory size (63622 < 64000)
[2020-08-13T10:20:03.698] error: Setting node ip-10-2-1-120 state to DRAIN
[2020-08-13T10:20:03.698] drain_nodes: node ip-10-2-1-120 state set to DRAIN

Which then needs to be brought back up manually with:

sudo /opt/slurm/bin/scontrol update node=ip-10-2-1-218 state=RESUME

I think this can be brought into a post-install with:

#!/bin/bash

. "/etc/parallelcluster/cfnconfig"

# Security updates
yum update -y

case "${cfn_node_type}" in
    MasterServer)
        # Update slurm conf to expect memory capacity of ComputeFleet based on input
        # Should be 64 Gbs of real memory for a m5.4xlarge partition
        # However we lose around 800mb on average with overhead.
        # Take off 2GB to be safe
        REAL_MEM="62000"
        SLURM_CONF_FILE="/opt/slurm/etc/slurm.conf"
        REAL_MEM_LINE="NodeName=DEFAULT RealMemory=${REAL_MEM}"
        INCLUDE_CLUSTER_LINE="include slurm_parallelcluster_nodes.conf"
        # Get line of INCLUDE_CLUSTER_LINE
        include_cluster_line_num=$(grep -n "${INCLUDE_CLUSTER_LINE}" "${SLURM_CONF_FILE}" | cut -d':' -f1)
        # Prepend with REAL_MEM_LINE
        sed -i "${include_cluster_line_num}i${REAL_MEM_LINE}" /opt/slurm/etc/slurm.conf
        # Restart slurm database with changes to conf file
        systemctl restart slurmctld
    ;;
    ComputeFleet)
    ;;
    *)
    ;;
esac

Furthermore, I would treat this as a bug / re-open this issue over the enhancement label - memory allocation is essential for running workflows on slurm.