Aws-parallelcluster: Cloudformation fails and cfncluster rollbacks during cluster creation

[bonetblai]

Dear all,

I had been unable to create a stable cluster with cfncluster. The cluster is created and then, immediately after all nodes come up, the cluster is destroyed. After looking the log file and wandering around in AWS, I found the following events in the event log of cloudformation:

14:26:13 UTC-0450 ROLLBACK_IN_PROGRESS AWS::CloudFormation::Stack cfncluster-slam The following resource(s) failed to create: [ComputeFleet]. . Rollback requested by user.

14:26:11 UTC-0450 CREATE_FAILED AWS::AutoScaling::AutoScalingGroup ComputeFleet Received FAILURE signal with UniqueId i-624e0bb6

14:18:47 UTC-0450 CREATE_IN_PROGRESS AWS::AutoScaling::Auto Resource creation Initiated

It seems that I am missing some kind of permission but I have no idea how to solve this problem. My config file (edited) is attached below.

Any help is appreciated!

Thanks

Blai

[dougalb]

Taking a look at your last stack launch, I think you issue is in the post_install script. I suspect the ln -s ... is failing on the Compute instances as it already is created on the MasterServer. I would suggest that you want to only execute certain commands once. Below is a small snippet that might help you understand how to do that.

. /opt/cfncluster/cfnconfig

if [ "$cfn_node_type" == "MasterServer" ]; then
  # Do something on the MasterServer
fi

if [ "$cfn_node_type" == "ComputeFleet" ]; then
  # Do something on a Compute instance
fi

[dougalb]

Another thing, if you use the cfncluster create --norollback mycluster when launching a cluster, it will keep the nodes running and you can then login and look at the log file. Typically errors will be reported in /var/log/cfn-init.log.

[bonetblai]

Dear Dougal,

Thanks for the information about --norollback and the location of the logs. It is very useful. As you said, the problem was related to the postinstall script.