Aws-parallelcluster: Documentation on 2.9.x "Multiple queue mode"

[rems0]

Hi all,

I've read the Multiple queue mode documentation , but I still don't understand which compute_resource will be selected on a queue with more than one compute_resources.

Let's say I have a queue with 3 compute_resources and so 3 different instance_types.
If I send a job to that queue, which compute_resource (instance_type) will get started?
Am I missing something obvious here?

Kind regards,
Richard

[demartinofra]

Hi Richard,

I'm sorry that our documentation still lacks of some important details and examples on how to use the new multiple instance types functionality and we are working to address this asap.

In the new architecture the compute resources you define in your cluster configuration are all added to the scheduler queues as individual CLOUD scheduler nodes and the scheduler is given the responsibility to select what nodes to assign to each job based on the available compute nodes and on the job submission options.

For example here is the scheduler status for a cluster having 4 different queues:

[ec2-user@ip-10-0-0-162 ~]$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
efa          up   infinite     20  idle~ efa-dy-c5n18xlarge-[1-20]
gpu          up   infinite     10  idle~ gpu-dy-g38xlarge-[1-10]
ondemand*    up   infinite     29  idle~ ondemand-dy-c52xlarge-[1-9],ondemand-dy-t2xlarge-[1-20]
ondemand*    up   infinite      1   idle ondemand-st-c52xlarge-1
spot         up   infinite     12  idle~ spot-dy-c5xlarge-[1-9],spot-dy-t2large-[1-3]
spot         up   infinite      3   idle spot-st-c5xlarge-1,spot-st-t2large-[1-2]

As you can see all nodes are available to the scheduler, however some of them are in a power down state (idle~). When you submit a job the scheduler will select the node that is most suitable based on the options used at submission time. ParallelCluster then will take care of launching the nodes selected by the scheduler.

Each of these nodes also has a set of features attached to it:

[ec2-user@ip-10-0-0-162 ~]$ scontrol show nodes gpu-dy-g38xlarge-1 | grep Features
   AvailableFeatures=dynamic,g3.8xlarge,gpu
   ActiveFeatures=dynamic,g3.8xlarge,gpu

One of these features is the instance type of the node. This allows you to require specific instance types by using the Slurm constraint option documented here. For example say you want to submit a job to the spot queue which requires 2 c5.xlarge nodes and 2 t2.large nodes you can do something like:

sbatch -N 4 --constraint "[c5.xlarge*2&t2.large*2]" --partition spot --wrap "hostname"

In general you can use any submission option (such as the number of slots or the number of gpus) which would make the scheduler pick a specific node over another.
In case you do not specify any option Slurm will pick the node for you based on the configured scheduling logic.

As a side node it is useful to familiarize with the status codes that Slurm uses with CLOUD nodes. The list of Slurm status codes and available special suffixes is here: https://slurm.schedmd.com/sinfo.html#lbAG

[rems0]

Hi Franceso,

thanks for your thorough answer. It helps a lot understanding how to use it.

In our case what we want to achieve is to have two queues, sp-12 and od-12, each containing c5a.12xlarge, c5ad.12xlarge and c5.12xlarge and then submit with these two options:
1- send a job to either sp-12 or od-12 and let the system get one of those 3 instance_types, prioritizing the selection in that order (try to get c5a until some timeout, then switch to c5ad, then to c5a)
2- similar to above, but try first spot and if all 3 requests fail after N retries, try ondemand queue.

Is such a behaviour doable?
If we submit so Slurm to the above queue sp-12 with c5...12xlarge spot instances, requesting just 24 cores (we run one thread per core), which instance_type will then Slurm request since all 3 types are "equal" in this case? Can priorities be given somehow?

I hope that was all not that confusing.

Kind regards,
Richard

[demartinofra]

Hi Richard,

sorry for the delayed answer. With this initial version of multiple queues and instance types ParallelCluster does not provide a native mechanism to assign priorities to instances. At the same time, when a node associated to a specific instance type is not available, ParallelCluster marks it as unavailable for the time defined by SuspendTimeout in Slurm config (it defaults to 2 minutes), which means if ParallelCluster fails to launch a given instance type, by default Slurm has 2 minutes to try a different node before the previous one can be retried again. Now depending on the cluster configuration and job requirements this might not be enough to let Slurm loop over all the remaining instance types.

The best thing you can do to achieve what you are describing in point 1 is adding some customizations on top of what ParallelCluster provides. Keep in mind this is not a fully supported path but we can provide some pointer if you would like to go this way. Shortly you can modify Slurm config under /opt/slurm/etc/pcluster/ and configure a Weight for each node so that Slurm will try instance types in a given order. ParallelCluster will not override configs under /opt/slurm/etc/pcluster/ until you perform a cluster update. Then you might need to increase the SuspendTimeout in slurm.conf to keep nodes down for a little longer so that Slurm can cycle over the remaining capacity when some instance types are not available. Keep in mind that increasing the SuspendTimeout too much will also affect the time to have an instance available to be re-launched after a scaling down operation so you don't want it to be too big.
The general best practice here is trying to select a max_count that is not very far from the max number of instances you are able to usually get.

Regarding point 2 I don't think there is an easy mechanism to achieve that. You need to somehow build some custom logic that requeues the job to a different queue when the first one cannot run it after a certain amount of time.

Let me know if you need further details on any of the points above.

Francesco

[demartinofra]

As promised an official guide for multiple queues in Slurm has been added to our docs: https://docs.aws.amazon.com/parallelcluster/latest/ug/multiple-queue-mode-slurm-user-guide.html

Resolving this..Feel free to reopen if I missed anything