Aws-parallelcluster: --norollback parameter causing issues when additional compute nodes are created

[alexiswl]

Environment:

• AWS ParallelCluster / CfnCluster version

2.8.1

• Configuration file (i.e. ~/.parallelcluster/config) without any credentials or personal data.
  ```
  [aws]
  aws_region_name = ap-southeast-2

[global]
cluster_template = umccr_dev
update_check = true
sanity_check = true

[cluster umccr_dev]
base_os = alinux2
vpc_settings = umccr_dev_network

Definitely experimental

fsx_settings = umccr_dev_lustrefs
s3_read_resource = *
cluster_type = spot
key_name = alexis-wfh-dev

Need something substantial to hold the slurm database

master_instance_type = t2.medium

16 CPUs and 64 Gb

compute_instance_type = m5.4xlarge

Using additional_iam_policies over ec2_iam_role

Add SSM policy, so we can keep port 22 closed

additional_iam_policies = arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
scheduler = slurm
initial_queue_size = 1

Basic ami with installations of

R, python3.8, conda, pip, ruby, golang, rust

conda has been initialised to the ec2-user

Currently built manually - create image builder for this.

custom_ami = ami-02f2ae59ffab4fbfb
post_install = s3://umccr-temp-dev/Alexis_parallel_cluster_test/bootstrap/bootstrap-slurm-cromwell.sh

[vpc umccr_dev_network]

Default vpc for dev account

vpc_id = vpc-00eafc63c0dfca266

Compute subnet ids take the same subnet as the master

Our default public subnet

master_subnet_id = subnet-0fab038b0341872f1

Elastic IP address is associated to the master instance.

use_public_ips = true

Security group where the slurm accounting database sits

additional_sg = sg-0ca5bdaab39885649

[aliases]
ssh = ssh {CFN_USER}@{MASTER_IP} {ARGS}

[fsx umccr_dev_lustrefs]
shared_dir = /fsx
storage_capacity = 1200
imported_file_chunk_size = 1024

FIXME how to mount this folder at /fsx

import_path = s3://umccr-temp-dev/Alexis_parallel_cluster_test/input_folder
```

Bug description and how to reproduce:
Running pcluster create with the --norollback flag generates a situation where additional compute nodes cannot be registered. The compute nodes are spun up successfully when the --norollback flag is NOT specified.

How to reproduce:

• Spin up the cluster with pcluster create
• Wait ten minutes for the first compute node to terminate.
• Then run srun --pty bash. It will try to spin up a new compute node.
  The node will start running but an error will be seen from slurm

Slurm error:

srun --pty bash
srun: Required node not available (down, drained or reserved)
srun: job 5 queued and waiting for resources
srun: job 5 has been allocated resources
srun: error: fwd_tree_thread: can't find address for host ip-10-2-1-121, check slurm.conf
srun: error: Task launch for 5.0 failed on node ip-10-2-1-121: Can't find an address, check slurm.conf
srun: error: Application launch failed: Can't find an address, check slurm.conf
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: Timed out waiting for job step to complete

Snippet from /var/log/messages on compute node:

Aug 19 07:34:46 ip-10-2-1-121 cloud-init: + cfn-signal --exit-code=0 '--reason=ComputeServer setup complete' --stack=parallelcluster-AlexisFirstBcBioCondaInstallTest --resource=ComputeFleet --region=ap-sou
theast-2
Aug 19 07:34:46 ip-10-2-1-121 ec2net: [get_meta] Trying to get http://169.254.169.254/latest/meta-data/network/interfaces/macs/02:42:e2:af:00:9f/local-ipv4s
Aug 19 07:34:46 ip-10-2-1-121 cloud-init: ValidationError: Stack arn:aws:cloudformation:ap-southeast-2:843407916570:stack/parallelcluster-AlexisFirstBcBioCondaInstallTest/3d0b0240-e1c5-11ea-bd59-06fa27f652
ae is in CREATE_COMPLETE state and cannot be signaled
Aug 19 07:34:46 ip-10-2-1-121 cloud-init: Aug 19 07:34:46 cloud-init[3948]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-002 [1]
Aug 19 07:34:46 ip-10-2-1-121 cloud-init: Aug 19 07:34:46 cloud-init[3948]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
Aug 19 07:34:46 ip-10-2-1-121 cloud-init: Aug 19 07:34:46 cloud-init[3948]: util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/site-packages/
cloudinit/config/cc_scripts_user.pyc'>) failed

Please let me know if you need any more logs.

Alexis.

[tilne]

Hi Alexis.

This issue isn't related to pcluster create's --no-rollback flag. This flag is passed as the DisableRollback parameter to CloudFormation's CreateStack API. Its only function is to prevent CloudFormation from deleting all of the stack resources if cluster creation fails.

Those messages from /var/log/messages are expected. They are caused by the fact that the signaling process (used to make sure the minimum number of initial compute nodes is ready before a cluster is considered to have been successfully created) is no longer necessary.

Instead I think the issue you're seeing is caused by the fact that ParallelCluster doesn't (currently) support the use of interactive commands like srun to scale a cluster up. Use this type of interactive command only after a cluster has scaled up to the desired size.

For example, if you wanted to use srun to run a job interactively across three nodes on a cluster that had an initial size of zero compute nodes, you could do :

• scale the cluster up: sbatch -N 3 --wrap='sleep 60'
• loop until the job is finished running: while scontrol show job ${JOB_ID_FROM_PREVIOUS_STEP} | grep 'JobState=COMPLETED'; do sleep 10; done
• run the interactive command: srun [args] /path/to/app

(The 'duplicate' label added due to the similarity of this issue to #1347)

[alexiswl]

Thank you, I have been able to reproduce this.

[brainstorm]

Thanks for the response, @tilne ... since you seem to have good insight into this, what's the real underlying issue? Simply SLURM not waiting for long enough for instances to come up (default seems to be 32 seconds?) or something else more related to AWS's parallelcluster instead?

If you were to fix this issue (other than using your workaround), would you start looking at SLURM's codebase or somewhere else?

[alexiswl]

scale the cluster up: sbatch -N 3 --wrap='sleep 60'
loop until the job is finished running: while scontrol show job ${JOB_ID_FROM_PREVIOUS_STEP} | grep 'JobState=COMPLETED'; do sleep 10; done

This could be shortened to
sbatch --wait --wrap='sleep 2'

The --wait parameter prevents sbatch from returning until the job has terminated.

[tilne]

Thanks for the response, @tilne ... since you seem to have good insight into this, what's the real underlying issue? Simply SLURM not waiting for long enough for instances to come up (default seems to be 32 seconds?) or something else more related to AWS's parallelcluster instead?

It's related to how ParallelCluster reconfigures slurm as new compute nodes come online and are added to a cluster.

If you were to fix this issue (other than using your workaround), would you start looking at SLURM's codebase or somewhere else?

It should be fixed in v2.9.0.

[tilne]

This could be shortened to
sbatch --wait --wrap='sleep 2'

The --wait parameter prevents sbatch from returning until the job has terminated.

I did not know about that flag. It's very useful. Thanks for the tip @alexiswl!

[alexiswl]

Work around with sinteractive

sinteractive.sh

#!/usr/bin/env bash

check_nodes(){
  # Check a node is ready
  # 1 for yes, 0 for no
  node_length=$(scontrol show nodes --oneliner | grep -v "DRAIN" | grep -cv "No nodes in the system")
  echo "${node_length}"
}

if [[ "$(check_nodes)" == "0" ]]; then
  sbatch --wait --wrap "sleep 2" >/dev/null 2>&1
fi

# tcsh implementation adds -l param
if [[ ! "${SHELL}" == "/bin/tcsh" ]]; then
  eval srun "$*" --pty -u "${SHELL}" -i -l
else
  eval srun "$*" --pty -u "${SHELL}" -i
fi

Link to known path dir

ln -s /path/to/sinteractive.sh /usr/bin/sinteractive

Usage

sinteractive --cpus-per-task=2 --mem=10G

[demartinofra]

I'm going to resolve this one since starting from version 2.9 ParallelCluster supports the usage of srun command with Slurm scheduler

[alexiswl]

Yes thank you, we've been able to reproduce this on our side