Aws-parallelcluster: Unable to launch cluster

[armaneshaghi]

Environment:

• AWS ParallelCluste: current dev version on Github
• OS: OS X Mojave
• Scheduler:SGE
• Master instance type: C4.large
• Compute instance type: C4.xlarge

My cluster fails at the very beginning. I am using a minimally customized AMI built on top of the default AMI that pcluster creates (ALinux).

$ pcluster create cluster2
 Beginning cluster creation for cluster: cluster2 
Creating stack named: parallelcluster-cluster2 Status: AssociateEIP - CREATE_COMPLETE                                            
Status: parallelcluster-cluster2 - ROLLBACK_IN_PROGRESS                          
Cluster creation failed.  Failed events:
   - AWS::EC2::Instance MasterServer Received FAILURE signal with UniqueId i-0cf58d9102fbe780b

I can create my cluster with no problems if I do not use the customized AMI.

Additional context:

My config file is as follows:

[global]
cluster_template = default
update_check = true
sanity_check = true

[aws]
aws_region_name = eu-west-2
aws_access_key_id = <REMOVED>
aws_secret_access_key = <REMOVED>

[cluster default]
vpc_settings = public
key_name = ipmsa
master_instance_type = c4.large
compute_instance_type = c4.xlarge
cluster_type = spot
spot_price = 0.2
master_root_volume_size = 17
initial_queue_size = 2
custom_ami = ami-<REMOVED>

[vpc public]
vpc_id = vpc-<REMOVED>
master_subnet_id = subnet-<REMOVED>
ssh_from = <REMOVED>

[aliases]
ssh = ssh {CFN_USER}@{MASTER_IP} {ARGS}
vpc_settings = public

[scaling custom]
scaledown_idletime = 10

#[ebs custom1]
shared_dir = software
volume_size = 60
ebs_volume_id = vol-<REMOVED>

Any input would be appreciated.

[lukeseawalker]

Hi @armaneshaghi,
please re-execute create action using --norollback option and attach /var/log/cfn-init.log, /var/log/cloud-init.log and /var/log/cloud-init-output.log files from the Master node

[armaneshaghi]

Thank you @lukeseawalker .

$ pcluster create cluster --norollback 
Beginning cluster creation for cluster: cluster
Creating stack named: parallelcluster-cluster
Status: parallelcluster-cluster - CREATE_FAILED                                 
Cluster creation failed.  Failed events:
  - AWS::CloudFormation::Stack parallelcluster-cluster The following resource(s) failed to create: [MasterServer]. 
  - AWS::EC2::Instance MasterServer Received FAILURE signal with UniqueId i-02bbce3c6f6193397                       

The Master node, despite the message, is created. Logs are as follows:

cloud-init-output.log
cloud-init.log
cloud-init.log

[lukeseawalker]

/var/log/cfn-init.log is missing
thanks

[armaneshaghi]

Sorry.

cfn-init.log

[lukeseawalker]

How did you create the custom AMI?
From the logs looks like that you used for the current cluster (parallelcluster-cluster) an AMI created from a previously running cluster (parallelcluster-testcluster).

The AMI you want to customize needs to be launched independently from AWS ParallelCluster (e.g. from the EC2 console).

The procedure is described here Modify an AWS ParallelCluster AMI

In your case you must start creating your custom AMI picking up one AMI from this list https://github.com/aws/aws-parallelcluster/blob/v2.3.1/amis.txt

[armaneshaghi]

Thanks Luke. I followed the link you sent, but it is true that the AMI I was customising was created as a cluster by pcluster.

I will use EC2 Console and will reopen this thread if the issue persists.

[lukeseawalker]

Perfect.
Thank you for the update