Aws-parallelcluster: Should Elastic Filesystem replace EBS for /shared?

[chambm]

I have run into major bottlenecks with using EBS from a hundred compute nodes. I haven't had the time to implement Lustre; it seems using EFS would obviate the need for me to learn how to set that up.

[sajp]

I would also be interested in this, it should be easy enough to create a EFS file system and mount it on the cluster nodes as part of the post_install process but it would be nice to have it as a option within cfncluster itself.

Also not sure if you are aware but intel have release luster as a service on aws markerplace @chambm, this seems to be a useful doc showing how to set it up http://www.intel.com/content/dam/www/public/us/en/documents/reference-architectures/ICEL-RA.pdf

[dougalb]

I’ve made a post_install script for mounting EFS public and you can use this to test EFS. As usage of EFS grows and we get more feedback on how customers would like to see CfnCluster integration, we’ll look at what the best approaches are.

post_install = https://s3.amazonaws.com/cfncluster-global/scripts/mount-efs.sh
post_install_args = fs-a1b2c3d4

You should also be able to use s3://cfncluster-global/scripts/mount-efs.sh for the post_install parameter and in addition you can provide the mount point as a second argument to the script. By default, it uses _/efs_.

#!/bin/bash

. /etc/cfncluster/cfnconfig

fsid=$1
mnt=$2

if [ "x$fsid" == "x" ]; then
  echo "ERROR: You must provide the EFS File System Id i.e. fs-a1b2c3d4"
  exit 1
fi
if [ "x$mnt" == "x" ]; then
  mnt=/efs
fi

mkdir -p $mnt
mount -t nfs4 -o nfsvers=4.1 $(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone).${fsid}.efs.$cfn_region.amazonaws.com:/ $mnt

This post_install script has been tested on Amazon Linux. In addition, mounting Intel Cloud Edition Lustre, is very easy for v1.2 onwards, as they have an install script. More details can be found here: https://wiki.hpdd.intel.com/display/PUB/Intel+Cloud+Edition+for+Lustre+software+-+Client+Setup

[chambm]

Hi Doug, mounting Lustre may be easy, but what about creating the Lustre filesystem?

Requirements

A Cloud Edition Lustre filesystem must be running and accessible by the client instances

A scalable shared filesystem should be part of cfncluster by default - i.e. create and destroy it along with the rest of the cluster. Persisting that filesystem would be the job of more advanced options.

Thanks for the EFS script. I'll try that out soon.

[skoocda]

Hi @dougalb , I've tried this script and repeatedly received the following error:

CREATE_FAILED Status reason: The following resource(s) failed to create: [MasterServer]

with no additional info, from what I can find in the logs. My best guess is a post_install failure from the comparable issues I've seen.

Your answer in this thread suggested adding a line to the post_install script:
https://forums.aws.amazon.com/thread.jspa?threadID=239613

'. /etc/profile'

Will this solve this issue when mounting an EFS? Or is it possible that it's a network error of some sort?

EDIT: I believe this is caused by Issue 139

[adavanisanti]

@dougalb Is there a plan to include EFS creation and mount as part of cfncluster? I am also interested in using EFS instead of EBS.

[jprobichaud]

For our own setup, mounting our 3TB EFS onto the master/compute nodes is easy but the performance is abysmal. If 20 machines with 16 cores each try to access randomly files on EFS, the performance just dies.

We have much better success with /shared being a big fat EBS (but simple gp2) on the master node and exported through NFS. Each compute node takes a set of relatively large zip files from either the NFS volume or EFS one, copies it locally onto /scratch, do the work, zip the results and push it out.

HPC will always put pressure on the file system. GPFS, GlusterFS, Lustre, all these fail at some point if you try to allow lots of random access into a huge filesystem.

[dkmichaels]

I tried to find this script but it's no longer there. Can someone point me to its current location?
post_install = https://s3.amazonaws.com/cfncluster-global/scripts/mount-efs.sh

[sean-smith]

It's no longer hosted in cfncluster-global bucket. You'll need to host it yourself. Just copy the script above https://github.com/awslabs/cfncluster/issues/116#issuecomment-230842111 and upload to S3.

[dkmichaels]

Thanks!

[enrico-usai]

I'm going to close this issue since we added the EFS support with ParallelCluster 2.1.0 and the FSx Lustre support for Centos 7 with ParallelCluster 2.2.1.
Stay tuned for other OS.