Aws-parallelcluster: Stopping cluster doesn't remove compute nodes from SGE

[huffmabc]

Unless there's something I'm missing, it appears that stopping the cluster (using the cfncluster stop command) will not remove the compute fleet from SGE because it doesn't wait for the compute nodes to be terminated before stopping the master.

Since the message sent to SQSwatcher to remove a queue node doesn't get sent until the compute node has completely terminated, and by that time the master is gone, this message never gets received or acted upon.

[bwbarrett]

@huffmabc stop only kind of works. If you restart the cluster within 4 days, the node remove messages will still be in the SQS queue when the master restarts and should be processed at the same time as the add node messages for the new nodes. However, after the messages age out of SQS, then the master will always be out of sync. SQS + stop/start probably isn't the right design, so this isn't going to be a quick fix, but we'll talk internally about design options.

[adavanisanti]

I have a similar problem. Stopping and starting a cluster doesn't add new compute nodes. Old nodes are still in the queue. Do you recommend not using stop/start at this time? or Is there a workaround?

Thanks
Santi

[bwbarrett]

@adavanisanti, I would not use stop/start. It's fundamentally broken in design. In 1.4.0 (coming real soon now), we will be changing start/stop to not stop the master node (only stop the compute nodes), because of the design issues with stop/start.

[adavanisanti]

@bwbarrett Got it! Does this mean in 1.4.0, compute nodes will remain the same after cfncluster stop/start? Could you give some details on the new design for stop/start?

Thanks!

[bwbarrett]

@adavanisanti, no the compute nodes will still be terminated (sorry, I accidentally used stopped earlier). The problem with start/stop today is that it relies on SQS messages from the compute nodes to the master being reliably delivered, but that's only true in a (rather small) period of time after the messages are generated. If the master is shut down, it will lose those messages. Essentially, start/stop becomes the same thing as setting the max/min/desired number of compute nodes to 0 in the autoscaling group, then restoring it after the start. We can do better than that, but it requires some significant architectural changes we haven't been able to fit into our roadmap.

[adavanisanti]

@bwbarrett Thanks! If I understand right, in 1.4.0, start/stop works the same for cfncluster user but the internals is different. Will the new changes make start/stop more reliable along with slurm/sge schedulers to know which nodes are part of the cluster?

[bwbarrett]

No, the user functionality will change. In 1.3.x, when the user ran cfncluster stop, all compute nodes were terminated and the master node was stopped (in the EC2 sense of the words terminated and stopped). So the user was left incurring charges for the EBS volumes and EIP associated with the master node, but not the instance itself.

In 1.4.x, when the user runs cfncluster stop, all compute nodes are terminated, but the master is not stopped. So the user will continue incurring charges for the EBS volumes and the EC2 instance hosting the master node. The user could, in theory, make sure the SQS queue is drained and then manually stop the master node if the master node cost was a concern.

The issue with the 1.3.x behavior is that SQS messages have a limited lifetime and the master is the only source of truth about what compute nodes are running, but that source of truth is updated by the SQS messages generated when a compute node is started or terminated. So if we stop the master at the same time we terminate the compute nodes (the old behavior of CfnCluster's stop/start) and the master is stopped for longer than the SQS message retention time, the SQS messages are lost and the master's database about nodes in the cluster is forever wrong.

There are a couple long term solutions. One would be to move the source of truth from the master to DynamoDB. You can then imagine having lambda functions (which didn't exist when CfnCluster was designed...) that took the SQS messages and modified the DynamoDB database. This would slightly add to the cost of running a cluster, but probably not enough to matter. The master node would then poll the DynamoDB table to get the list of running compute nodes and update the scheduler. There's a bunch of little changes here, and like I said earlier, we don't have space in our roadmap for it just yet.

[adavanisanti]

@bwbarrett Got it! Thanks for the detailed response. Looking forward to 1.4.x.

[adavanisanti]

@bwbarrett Readme has a section of known issues with 1.4.x. Is 1.4.x released?

Thanks!

[bwbarrett]

@adavanisanti no; there's no way to start a release process without some readme confusion. Clearly, we're working on a release, no time table for it to ship.

[enrico-usai]

It has been fixed with CfnCluster 1.4.1 version.
Documentation here: https://cfncluster.readthedocs.io/en/latest/commands.html#stop.