Aws-parallelcluster: OpenFOAM case read takes long time.

[sumedhjain]

config_eu_new.pdf

Required Info:

• AWS ParallelCluster version: 2.6.1
• Full cluster configuration: check config file attached
• Cluster name: CFDEU31c

Bug description and how to reproduce:
Although this cannot qualify as bug, this seems to be serious issue. I run large HPC CFD applications on my cluster using c5 instances. I have big files (30-40GB) that need to be read into the compute node everytime from my EBS Volume mounted on my master node. Often as I start my simulation, I see the solver (application) takes endless amount of time to open the file and read it into the compute node. Many a times, this surpasses the idle time for my compute instances and they immdiately shut-down before data is read into them. Can you please suggest the best, cost-effective way to tackle this issue? Shall I increase IOPS of my master-node EBS volume or Switching to FSx. Will switching to EFA adaptor solve this issue?

Thank you in advance

[tilne]

Hi @sumedhjain.

I have big files (30-40GB) that need to be read into the compute node everytime from my EBS Volume mounted on my master node.
Often as I start my simulation, I see the solver (application) takes endless amount of time to open the file and read it into the compute node

As you alluded to, these definitely suggest a network bandwidth bottleneck. Especially given the size of the files you mentioned that the compute nodes are reading from an NFS drive exported by the head node.

Can you please suggest the best, cost-effective way to tackle this issue? Shall I increase IOPS of my master-node EBS volume or Switching to FSx.

Having the computes read the large data files from an FSx file system would be my first recommendation. Another approach would be to upgrade the head node instance type to one with more networking capabilities. You can see the network performance of the various instance types here.

The nice thing about the FSx approach is that you can avoid having to upgrade to a more powerful head node instance type, which can cost a lot depending on the demands of your application. This is because the FSx file system is mounted on the compute nodes, and so they can fetch data without routing it through the head node.

If you'd like to look more into pricing, you can find the pricing for different EC2 instance types here and FSx pricing information here.

Will switching to EFA adaptor solve this issue?

No, not if the problem is the one we think it is. I would definitely start with either having the computes read the data from an FSx file system or upgrading to a head node with better networking capabilities.

[sumedhjain]

Hello @tilne
Thank you for your reply.

I will try the FSx only suggestion first.
When you say head node with better NC, do you mean EFA or there is some other feature??

[tilne]

When you say head node with better NC, do you mean EFA or there is some other feature?

It's separate form EFA. On this page you can see that many of the instance types generally have a Network Bandwidth and EBS Bandwidth. If you're looking to select a head node that's better able to serve large quantities of data to compute nodes via NFS, I would strongly recommend selecting one with a higher Network Bandwidth rating.

Sorry for any confusion. I tried to link to this in my previous comment, but I linked to the FSx pricing document twice instead.

[sumedhjain]

Hello @tilne
FSx suggestion worked. Have not tried to change my head node type (felt it is not cost effective compared to FSx)
Thank you for your help.