Aws-parallelcluster: allow heterogeneous configurations (instance type and subnets)

[jprobichaud]

Recently, I'm running into the problem that there's ins't a lot of c5 instances available in the specific availability region the master subnet locks me in.

I'm wondering if it wouldn't be possible to be able to enable users to define clusters that can use a variety of instance types (c5.4xlarge or c5.9xlarge, or ...) as the compute nodes instances type and also have the ability to specify more than one master-subnet so that we can scale more easily.

Right now, my grid is stuck at 2 compute nodes because there's ins't any extra c5.9xlarge instances available in us-west-2b... This is is somewhat a show stopper for a HPC setup.

I'd be happy to testdrive stuff.

[bwbarrett]

Heterogeneous configurations is on our roadmap, but not something likely to be done in the short term. It also looks like changing the instance type for compute nodes in a running cluster (via cfncluster update) isn't functioning in 1.4.2.

Are you using placement groups? If so, are you setting the placement = option to cluster or compute? If set to cluster, the entire cluster, including the master node, has to be in the same placement group. It's possible (likely, really) that there are C5s available in the az, but not in the placement group that the master node is placed in. You may have better luck with placement = compute (or, if you're not networking bound, just not using placement groups at all), as that will give EC2's placement logic more freedom in finding the compute resources your application is requesting.

[bwbarrett]

This should have occurred to me sooner, but have you looked at why scaling up is failing? If you look at the autoscaling group associated with the cluster, the activity history will have the failure reason. The reason I bring this up is that most accounts will have a default limit of 2 c5.9xlarge instances in a given region; if that's your problem (instead of capacity), the error message in the failed scale-up should be pretty clear.

[jprobichaud]

Sorry, I should have replied earlier. The failing is caused by the lack of availability of c5.4xlarge instances in us-west-2b. Our quota for such machines is 55 but right now the grid can't get more than 31 instances.

I see that placement is set to cluster but placement_group = NONE. Does that means that we don't have any placement group set?

The fact that the template restrict the vpc to only one subnet makes it difficult to span across more availability zones. Am I missing something obvious here? Our production services auto-scaling groups can scale across multiple AZ (us-west-2*)

[bwbarrett]

placement_group = NONE means that you're not using placement groups. So the issue is likely not related to placement groups. For more information about why you can't launch more than the ~31 instances, you'll have to go through AWS support.

We chose to limit the autoscaling group to a single AZ because it matches what the majority of CfnCluster users want: an HPC-oriented cluster in AWS. Multi-AZ adds cost (in cross-AZ bandwidth charges) and latency (because intra-AZ latency is lower than inter-AZ latency). Placement Groups lower the upper bound of cross-instance latency, but at the cost of tighter capacity constraints. While we never tested it before this morning, it appears that you can add subnets to the ASG associated with CfnCluster and the system works cross-AZ. See this page for instructions: https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-add-availability-zone.html. Note that cross-AZ ASG is really meant for load balancing, so the ASG is going to try to keep a balanced number of instances in each AZ. This might work for your use case or it might not, but it likely would help with capacity.

[JiaweiZhuang]

Here's how I create heterogeneous compute nodes (tested with pcluster 2.1.1):

1. Spin up a cluster with a number of compute nodes (sayc5.4xlarge).
2. In config file, change compute node type, for example compute_instance_type = c5.9xlarge; and then run pcluster update {cluster-name}. This creates a new version of EC2 Launch Template for your compute nodes, and sets the Auto Scaling group to use that version.
3. Further scale up the compute nodes. The quickest way is to increase Desired Capacity in the Auto Scaling Groups console (https://github.com/aws/aws-parallelcluster/issues/692#issuecomment-439534581).

This is an undocumented practice so I am not sure if I am abusing Auto Scaling group :) But it does seem to work well.

[sean-smith]

@JiaweiZhuang While this works initially, if you were to scale down the c5.4xlarge nodes, they would come back up as c5.9xlarge. Just something to keep in mind, if the solution works for you, keep using it.

[MMichael-S]

Hi guys,
Any progress or update for the Heterogeneous configurations feature??
My customers hope to use AWS parallelcluster to create/manage multiple instance queue in a slurm cluster.
Thank you.

[aqi-mph]

Any further update on this feature?

[nstornetta]

@aqi-mph: Thanks for your inquiry into heterogeneous instance configurations. This feature is currently on the AWS ParallelCluster team's roadmap.

[demartinofra]

Resolving this in favour of https://github.com/aws/aws-parallelcluster/issues/1066
Both issues are tracking the same enhancement