Aws-parallelcluster: Master nfs export issue

[microbioticajon]

pcluster version: 2.3.1
Deployment OS: Ubuntu 18.04

Hi Guys,

Im having an issue with mounting shared volumes on the compute nodes on create. I checked the nfs exports file and noticed that allowable ip-range is pointing to an unexpected subnet. I was expecting it to be set to the supplied "compute_subnet_id" value but it is pointing to our public subnet instead.

I have verified that the master node and compute node are starting in the correct subnet (172.30.3.xx) but the export file is restricting to (172.30.1.0/24 - see below).

I cannot work out from where pcluster is picking up the value of "172.30.1.0".

I have supplied my config below in case anyone can spot something I have done wrong...

.parallelcluster/config:

[global]
cluster_template = default
update_check = true
sanity_check = true

[aws]
aws_region_name = eu-west-2

[aliases]
ssh = ssh {CFN_USER}@{MASTER_IP} {ARGS}

[vpc bob]
vpc_id = vpc-0bc2be.......
master_subnet_id = subnet-076f5a...... (as: 172.30.3.0/24)
compute_subnet_id = subnet-076f5a...... (as: 172.30.3.0/24)
ssh_from = 172.30.3.10/32
additional_sg = sg-054ad07......
use_public_ips = false

[cluster default]
key_name = .......
compute_instance_type = t2.micro
master_instance_type = t2.micro
initial_queue_size = 1
max_queue_size = 3
maintain_initial_size = true
scheduler = slurm
cluster_type = ondemand
ephemeral_dir = /scratch
base_os = ubuntu1604
#ec2_iam_role = NONE
ebs_settings = shared, software
vpc_settings = bob
scaling_settings = custom

[ebs software]
shared_dir = software
volume_type = gp2
volume_size = 100
encrypted = true

[ebs shared]
shared_dir = shared
volume_type = gp2
volume_size = 250
encrypted = true

Master: /etc/exports:

# /etc/exports: the access control list for filesystems which may be exported
#               to NFS clients.  See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
#
/shared 172.30.1.0/24(rw,sync,no_root_squash)
/software 172.30.1.0/24(rw,sync,no_root_squash)
/home 172.30.1.0/24(rw,sync,no_root_squash)
/opt/slurm 172.30.1.0/24(rw,sync,no_root_squash)

[enrico-usai]

Hi @microbioticajon ,

in the _master_base_config recipe we are exporting the FS for the whole VPC and not only for the compute subnet.

So the value 172.30.1.0/24 should be the CIDR of your VPC vpc_id = vpc-0bc2be....

Let us know if it helps.

[microbioticajon]

Hi @enrico-usai,

Thanks for the very quick reply! That now makes sense. However, our VPC has been configured with 5 CIDR blocks, the first of which is our public subnet. Im guessing in the case of multiple CIDR blocks it simply picks the first?

Best,
J

[enrico-usai]

Yes, we are getting the Primary IPv4 CIDR block of the VPC through the vpc-ipv4-cidr-block EC2 metadata.

Probably, we should use the vpc-ipv4-cidr-blocks metadata (with the final _s_) to retrieve all the IPv4 CIDR blocks for the VPC and use them in the export command, or only export the file system for the specific compute subnet.

Is this a blocking issue?

[microbioticajon]

Thanks @enrico-usai,

"Is this a blocking issue", I was thinking about this - I believe a work around could be to rewrite the /etc/exports file and relaunch nfs via the post_install script and replace the CIDR block that way.

Its a little bit dirty but should get us moving. At the moment we only have the one VPC so we have to use what we have been given.

Going forward:
retrieve all the IPv4 CIDR blocks for the VPC and use them in the export command: the problem I could see with this option is that if you have restrictive subnets (like a public subnet) then you might not want to expose your cluster volumes to that layer. I would add my vote to your second option.

I will let you know how I get on with this approach.

Thanks again,
J

[microbioticajon]

Ok, so I have has some success with the work around!

It's an ugly perl hack but it does the trick for the moment:

post_init.sh

#!/bin/bash

. /etc/parallelcluster/cfnconfig

case "${cfn_node_type}" in
  MasterServer)
    IPR=${2} perl -pi -e 's/172.\d{1,3}\.\d{1,3}\.\d{1,3}\/\d{2}/$ENV{IPR}/' /etc/exports;
    exportfs -ra;
  ;;
  ComputeFleet)
    # code to run on the compute node
  ;;
esac

[sean-smith]

Any difficulties getting it working with Ubuntu 18.04? Did you make any other changes (besides networking)?

[microbioticajon]

Hi @sean-smith ,

Sorry, my header was a little bit misleading. I wasn't using Ubuntu 18.04 for the Master/Compute nodes, it was the operating system used on the "launch/deploy" system running pcluster.
I can't see why there would be problems running the cluster on 18.04, I just need to build an ami and test. I haven't had time to dig into that yet /:).

Best,
J

[microbioticajon]

Hi @sean-smith ,

Regarding ubuntu1804, I have been hacking around with aws-parallelcluster-cookbook to build an ami with ubuntu1804. I had to make a few changes to the recipes (not entirely sure they are the correct changes) to progress the build, however I have hit a problem I cannot get past:

Error executing action `start` on resource 'service[ec2blkdev]'

If you are interested in taking a look, I have a fork of the repo here: https://github.com/microbioticajon/aws-parallelcluster-cookbook

Im calling ./build_ami.sh directly to simplify the process:
AWS_FLAVOR_ID=t2.large AWS_VPC_ID=vpc-12345 AWS_SUBNET_ID=subnet-54321 ./build_ami.sh --os ubuntu1804 --partition region --region eu-west-2

Otherwise, I will just keep hacking away :)

Best,
J

[sean-smith]

Thanks for the update @microbioticajon !

We're looking at this for future releases. Stay tuned to github page for updates. Thank for your excellent work so far.

[demartinofra]

Ubuntu 18 is now available with ParallelCluster 2.5.0.

The bug described here https://github.com/aws/aws-parallelcluster/issues/993#issuecomment-480837238 is still an issue. I'm going to mark this as a duplicate of https://github.com/aws/aws-parallelcluster/issues/1383