Aws-parallelcluster: Compute Fleet Received 2 FAILURE signals out of 2 and restarts endlessly

[MRA-Cambridge]

Environment:

• AWS ParallelCluster: aws-parallelcluster-2.5.0
• OS: Ubuntu 18.04
• Scheduler: Slurm
• Master instance type: T2.Micro
• Compute instance type: T2.Micro

Created a custom AMI for the cluster based on the supplied AMI spun up as Unbuntu 1804 from the new 2.5.0 release. The customised Master Instance comes up fine and has the required addtional Ubuntu packages, but the Compute instances terminate after being up about five minutes and get restarted as new instances because of the AWS autoscaling group. The compute instances are up just long enouch to see that they too have got the required addtional packages.

Cannot see the resaon for the failure in the compute node logs. I may well be missing something.

Log bundle and scaling logs are to be attached and the error message from the pcluster create command is below:
Status: parallelcluster-Cluster-C - CREATE_FAILED
Cluster creation failed. Failed events:

• AWS::CloudFormation::Stack parallelcluster-Cluster-C The following resource(s) failed to create: [ComputeFleet].
• AWS::AutoScaling::AutoScalingGroup ComputeFleet Received 2 FAILURE signal(s) out of 2. Unable to satisfy 100% MinSuccessfulInstancesPercent requirement

The configuraton file is below:

[cluster Cluster-C]
key_name = a-key-pair
vpc_settings = a-private-subnet-2C
s3_read_write_resource = arn:aws:s3:::nnnnnnnn/*
base_os = ubuntu1804
custom_ami = ami-nnnnnnnnnnnnn
scheduler = slurm
initial_queue_size = 2
max_queue_size = 10
maintain_initial_size = true
scaling_settings = custom

[vpc a-private-subnet-2C]
vpc_id = vpc-nnnnnnnnnnnnnnn
master_subnet_id = subnet-nnnnnnnnnnn
use_public_ips = false
vpc_security_group_id = sg-nnnnnnnnnnnnn

[global]
cluster_template = Cluster-C
update_check = true
sanity_check = true

[aliases]
ssh = ssh {CFN_USER}@{MASTER_IP} {ARGS}

[ebs]
encrypted = true
ebs_kms_key_id = nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnn

[scaling custom]
scaledown_idletime = 10

attaching /var/log/nodewatcher from the Compute node
and /var/log/jobwatcher and /var/log/sqswatcher from the Master node

[MRA-Cambridge]

Log files attached
i-0cab7dd64a35c2a3c.tar.gz
i-0fab423126552813e.tar.gz
i-065c47135fe4f133a.tar.gz
i-009798b3112628e90.tar.gz

watcherlogs.zip

[demartinofra]

Apparently compute nodes are failing during their bootstrap phase but it's not clear why from the logs.

I found the following error in the cfn-init.log for one of the compute nodes:

[2019-11-25T14:38:44+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: service[nfs-kernel-server] (nfs::server line 42) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /bin/systemctl --system restart nfs-kernel-server ----
STDOUT: 
STDERR: Job for nfs-server.service canceled.
---- End output of /bin/systemctl --system restart nfs-kernel-server ----
Ran /bin/systemctl --system restart nfs-kernel-server returned 1
[2019-11-25T14:38:44+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: service[nfs-kernel-server] (nfs::server line 42) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /bin/systemctl --system restart nfs-kernel-server ----
STDOUT: 
STDERR: Job for nfs-server.service canceled.
---- End output of /bin/systemctl --system restart nfs-kernel-server ----
Ran /bin/systemctl --system restart nfs-kernel-server returned 1

Not sure what is causing this. Here is how I would suggest to proceed:

• Can you try to create a cluster without a custom-ami and check if it succeeds?
  
  
  
  • In case it does we need to understand what is causing this failure in your custom ami, hence if you could share additional details on the software you are installing
  
  
  • If it fails we need to dive deep a bit more in your cluster settings, so that we can try to reproduce this on our side.
  
  

As a side node, if you want to customise your EBS settings you need to set a label for your ebs section and reference it from the cluster section:

[cluster Cluster-C]
...
ebs_settings = my_ebs

[ebs my_ebs]
shared_dir = shared
encrypted = true
ebs_kms_key_id = nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnn

[MRA-Cambridge]

Thanks. I will run with the custom EBS settings.
Yes, a cluster without customised AMI succeeds. Before creating a custom ami, I ran up a cluster from the default ami for 2.5.0 and all was OK. I purposfully chose to add just one small package "rolldice" then created the AMI following the steps on pages 89/90 of the user guide (Modify an AWS ParallelCluster AMI). I customised the run up Master instance since the compute nodes terminate when stopped.

[demartinofra]

Thanks for the additional details.

If I understood correctly you are executing the following steps:

• Create a cluster without a custom AMI
• Install new packages on the master instance
• Create a custom AMI out of the customized Master instance
• Create a new cluster with the obtained custom AMI

If so the problem is that you are using the Master instance as starting point rather than using a clean ParallelCluster AMI taken from this list. When the master boots some additional runtime configurations are executed and those are specific to the master node, hence do not need to be part of a custom ami. That is why you need to start an instance with a clean ParallelCluster AMI rather than starting the customization from a cluster instance.

Just let me know if that makes sense to you and if you have any additional question. Is there anything we can improve from the official docs in order to make this more clear?

[MRA-Cambridge]

That could well be it, Yes. I believe I used the 2.5.0 AMI from the list to run up a first cluster and then customised from the cluster that spun up fine. Let me test again adhering better to the official docs and report back. Thanks.

[MRA-Cambridge]

I remember now why I wanted to build a custom AMI from the clustered instances. The single AMI you deliver from this list above is AMI 2 Linux, but we need to run Ubuntu 18.04 to have access to install from the wider range of Ubuntu 18.04 compatible packages. Ubnutu 18.04 is now supported in ParallelCluster 2.5.0 but how can I make an Ubuntu 18.04 customised AMI?

[demartinofra]

Same approach you would use for any other OS. As described here either you launch an instance with an existing ParallelCluster AMI taken from this list or you start from an official Ubuntu 18 AMI, add your package and then make it compatible with ParallelCluster with the pcluster createami command.
Just make sure that when you use such custom ami with the custom_ami directive you also specify base_os = ubuntu1804

[MRA-Cambridge]

Got it. Thank-you. The doc was steering me towards the "safest option". This info request can be closed off.

[MRA-Cambridge]

Good Morning,

I’ve been away from this for a few days, but now back…….

The case is closed so this is just a for info message.

I happy to report that the Cloud formation generated from the Pcluster create command now works fine using a customised Ubuntu 18.04 AMI.

Thanks again for your help and advice. Our research project which will probably want to use AWS Parallel Cluster for some processing at least has the option.

Regards,
Mark.

Mark Andrews
IT Specialist

MRC Biostatistics Unit
University of Cambridge, School of Clinical Medicine
Cambridge Institute of Public Health
Forvie Site
Robinson Way
Cambridge Biomedical Campus
Cambridge CB2 0SR

Tel: [phone]
Email: mark.[email protected][e-mail]
Website: www.mrc-bsu.cam.ac.ukhttp://www.mrc-bsu.cam.ac.uk/
Follow us on Twitter: @MRC_BSUhttps://twitter.com/MRC_BSU

From: Francesco De Martino [e-mail]
Sent: 28 November 2019 14:01
To: aws/aws-parallelcluster [e-mail]
Cc: Andrews, Mark [e-mail]; Author [e-mail]
Subject: Re: [aws/aws-parallelcluster] Compute Fleet Received 2 FAILURE signals out of 2 and restarts endlessly (#1492)

Same approach you would use for any other OS. As described herehttps://docs.aws.amazon.com/parallelcluster/latest/ug/tutorials_02_ami_customization.html#modify-an-aws-parallelcluster-ami either you launch an instance with an existing ParallelCluster AMI taken from this listhttps://github.com/aws/aws-parallelcluster/blob/v2.5.0/amis.txt or you start from an official Ubuntu 18 AMI, add your package and then make it compatible with ParallelCluster with the pcluster createami command.
Just make sure that when you use such custom ami with the custom_ami directive you also specify base_os = ubuntu1804

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/aws/aws-parallelcluster/issues/1492?email_source=notifications&email_token=AK4MBTF3H2ZH3PGSBGPNVNLQV7FI5A5CNFSM4JRJYJBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFMV63I#issuecomment-559505261, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK4MBTBFVYPBA35C2ZVMBWTQV7FI5ANCNFSM4JRJYJBA.