Aws-parallelcluster: Nfs mount directory naming issue

Created on 8 Apr 2019  路  7Comments  路  Source: aws/aws-parallelcluster

Environment:

  • aws-parallelcluster-2.3.1
  • OS: Ubuntu
  • Scheduler: slurm

Bug description and how to reproduce:
When specifying more than one shared ebs volume, the compute node attempts to name the mount dir as a comma concatenation of all the shared volume paths (see below)...

.parallelcluster/config:

[global]
cluster_template = default
update_check = true
sanity_check = true

[aws]
aws_region_name = eu-west-2

[aliases]
ssh = ssh {CFN_USER}@{MASTER_IP} {ARGS}

[vpc bob]
vpc_id = vpc-0bc2be.......
master_subnet_id = subnet-076f5a......
compute_subnet_id = subnet-076f5a......
ssh_from = 172.30.3.10/32
additional_sg = sg-054ad07......
use_public_ips = false

[cluster default]
key_name = .......
compute_instance_type = t2.micro
master_instance_type = t2.micro
initial_queue_size = 1
max_queue_size = 3
maintain_initial_size = true
scheduler = slurm
cluster_type = ondemand
ephemeral_dir = /scratch
base_os = ubuntu1604
#ec2_iam_role = NONE
ebs_settings = shared, software
vpc_settings = bob
scaling_settings = custom

[ebs software]
shared_dir = software
volume_type = gp2
volume_size = 100
encrypted = true

[ebs shared]
shared_dir = shared
volume_type = gp2
volume_size = 250
encrypted = true

compute: ll /

drwxr-xr-x  25 root root  4096 Apr  8 11:29 ./
drwxr-xr-x  25 root root  4096 Apr  8 11:29 ../
drwxr-xr-x   2 root root  4096 Apr  2 21:22 bin/
drwxr-xr-x   3 root root  4096 Apr  2 21:24 boot/
drwxr-xr-x  14 root root  3240 Apr  8 11:27 dev/
drwxr-xr-x 110 root root  4096 Apr  8 11:29 etc/
drwxr-xr-x   5 root root  4096 Apr  2 21:34 home/
lrwxrwxrwx   1 root root    33 Apr  2 21:24 initrd.img -> boot/initrd.img-4.4.0-145-generic
lrwxrwxrwx   1 root root    30 Apr  2 21:24 initrd.img.old -> boot/initrd.img-4.4.0-1079-aws
drwxr-xr-x  22 root root  4096 Apr  2 21:23 lib/
drwxr-xr-x   2 root root  4096 Mar 20 08:11 lib64/
drwx------   2 root root 16384 Mar 20 08:22 lost+found/
drwxr-xr-x   2 root root  4096 Mar 20 08:10 media/
drwxr-xr-x   2 root root  4096 Mar 20 08:10 mnt/
drwxr-xr-x   7 root root  4096 Apr  2 21:34 opt/
dr-xr-xr-x 140 root root     0 Apr  8 11:27 proc/
drwx------   4 root root  4096 Apr  2 21:17 root/
drwxr-xr-x  29 root root  1060 Apr  8 12:11 run/
drwxr-xr-x   2 root root 12288 Apr  8 11:29 sbin/
drwxrwxrwt   2 root root  4096 Apr  8 11:29 scratch/
drwxrwxrwt   2 root root  4096 Apr  8 11:29 shared,software/
drwxr-xr-x   5 root root  4096 Apr  2 21:17 snap/
drwxr-xr-x   2 root root  4096 Mar 20 08:10 srv/
dr-xr-xr-x  13 root root     0 Apr  8 11:27 sys/
drwxrwxrwt   8 root root  4096 Apr  8 12:11 tmp/
drwxr-xr-x  10 root root  4096 Mar 20 08:10 usr/
drwxr-xr-x  14 root root  4096 Apr  2 21:21 var/
lrwxrwxrwx   1 root root    30 Apr  2 21:24 vmlinuz -> boot/vmlinuz-4.4.0-145-generic
lrwxrwxrwx   1 root root    27 Apr  2 21:24 vmlinuz.old -> boot/vmlinuz-4.4.0-1079-aws

Up until this point, I didnt even realise that comma's were valid characters in directory paths!! /:)

Clearly, I cannot be the first person to come across this, so Im putting money on that I have a configuration problem - just cannot see what it is...

Best,
J

bug
Source
picture microbioticajon

All 7 comments

Update:

Looks like this issue was related to #993. As the compute nodes are now able to see the nfs exports on the Master node, the compute volume mounts are now appearing. Im still getting the weird /shared,software/ folder which is a little confusing...

As the mounts are now appearing Im going to close this issue

microbioticajon picture microbioticajon on 9 Apr 2019

I'm re-opening this issue so we track the /shared,software bug. Thanks for reporting this!

sean-smith picture sean-smith on 9 Apr 2019
👍1

I tried to reproduce your issue by using your configuration with no luck.
I see the two different mount points:

drwxrwxrwt   3 root root  4096 Apr  9 07:40 shared/
drwxrwxrwt   3 root root  4096 Apr  9 07:40 software/
...

I can also confirm the code is splitting by comma.

If you're able to reproduce the issue could you please send us the /var/log/cfn-init.log, /var/log/cloud-init.log and /var/log/cloud-init-output.log files from the Master node?

Thank you

enrico-usai picture enrico-usai on 9 Apr 2019

Thanks @enrico-usai ,

Just to confirm I have just rebuilt the cluster and am getting the same weird folder on the compute node. The master node seems to be fine.

As for the log files, they are relatively big and likely contain various resource identifiers Im not sure I want to post on the open web :) Is there a way I can get them over to you offline?

Otherwise I have searched through the logs on the compute node and spotted the following:

cfn-init.log

...
Compiling Cookbooks...
Converging 7 resources
Recipe: aws-parallelcluster::_prep_env
  * directory[/etc/parallelcluster] action create (up to date)
  * directory[/opt/parallelcluster] action create (up to date)
  * directory[/opt/parallelcluster/scripts] action create (up to date)
  * template[/etc/parallelcluster/cfnconfig] action create
    - create new file /etc/parallelcluster/cfnconfig
    - update content in file /etc/parallelcluster/cfnconfig from none to e2385e
    --- /etc/parallelcluster/cfnconfig  2019-04-09 08:04:32.486577437 +0000
    +++ /etc/parallelcluster/.chef-cfnconfig20190409-1531-vy9t9r        2019-04-09 08:04:32.486577437 +0000
    @@ -1 +1,18 @@
    +stack_name=parallelcluster-test
    +cfn_preinstall=NONE
    +cfn_preinstall_args=NONE
    +cfn_postinstall=s3://my-s3-bucket/aws-parallel-cluster-config/post_install.sh
    +cfn_postinstall_args=""172.30.3.0/24""
    +cfn_region=eu-west-2
    +cfn_scheduler=slurm
    +cfn_scheduler_slots=vcpus
    +cfn_instance_slots=1
    +cfn_encrypted_ephemeral=false
    +cfn_ephemeral_dir=/scratch
    +cfn_shared_dir=shared,software
    +cfn_proxy=NONE
    +cfn_node_type=ComputeFleet
    +cfn_cluster_user=ubuntu
    +cfn_sqs_queue=https://sqs.eu-west-2.amazonaws.com/112345/parallelcluster-test-SQS-1O9J7AA6871S5
    +cfn_master=ip-172-30-3-159.eu-west-2.compute.internal
    - change mode from '' to '0644'

...

Recipe: aws-parallelcluster::base_config
  * execute[setup ephemeral] action run
    - execute /usr/local/sbin/setup-ephemeral-drives.sh
Recipe: aws-parallelcluster::_compute_base_config
  * directory[shared,software] action create
    - create new directory shared,software
    - change mode from '' to '01777'
    - change owner from '' to 'root'
    - change group from '' to 'root'
  * mount[/home] action mount
    - mount ip-172-30-3-159.eu-west-2.compute.internal:/home to /home
  * mount[/home] action enable
    - enable ip-172-30-3-159.eu-west-2.compute.internal:/home
  * linux_user[ubuntu] action create
    - alter user ubuntu
  * directory[/shared] action create
    - create new directory /shared
    - change mode from '' to '01777'
    - change owner from '' to 'root'
    - change group from '' to 'root'
  * mount[/shared] action mount
    - mount ip-172-30-3-159.eu-west-2.compute.internal:/shared to /shared
  * mount[/shared] action enable
    - enable ip-172-30-3-159.eu-west-2.compute.internal:/shared
  * directory[/software] action create
    - create new directory /software
    - change mode from '' to '01777'
    - change owner from '' to 'root'
    - change group from '' to 'root'
  * mount[/software] action mount
    - mount ip-172-30-3-159.eu-west-2.compute.internal:/software to /software
  * mount[/software] action enable
    - enable ip-172-30-3-159.eu-west-2.compute.internal:/software
  * template[/etc/nodewatcher.cfg] action create
    - create new file /etc/nodewatcher.cfg
    - update content in file /etc/nodewatcher.cfg from none to 588802
    --- /etc/nodewatcher.cfg    2019-04-09 08:05:04.870370901 +0000
    +++ /etc/.chef-nodewatcher20190409-1963-qx1u16.cfg  2019-04-09 08:05:04.870370901 +0000
    @@ -1 +1,7 @@
    +[nodewatcher]
    +region = eu-west-2
    +scheduler = slurm
    +proxy = NONE
    +scaledown_idletime = 5
    +stack_name = parallelcluster-test
    - change mode from '' to '0644'
    - change owner from '' to 'root'
    - change group from '' to 'root'

...

cloud-init-output.log attached: cloud-init-output.log

microbioticajon picture microbioticajon on 9 Apr 2019

A difference I can see between _master_base_config.rb and _compute_base_config.rb is:

https://github.com/aws/aws-parallelcluster-cookbook/blob/7568a23625c6dd5599fe6f04da1663d243191b4c/recipes/_compute_base_config.rb#L17

and:

https://github.com/aws/aws-parallelcluster-cookbook/blob/7568a23625c6dd5599fe6f04da1663d243191b4c/recipes/_master_base_config.rb#L35

EDIT:
There is an extra step in the _compute_base_config.rb that makes a folder without splitting node['cfncluster']['cfn_shared_dir']

microbioticajon picture microbioticajon on 9 Apr 2019
👍1

Thank you for the analysis, you found the point.
The problem is only in the compute nodes. I'm marking it as a bug.

enrico-usai picture enrico-usai on 9 Apr 2019
👍1

The patch has been merged in the _develop_ branch of the cookbook repo.
It will be available with the next release. Thank you.

enrico-usai picture enrico-usai on 12 Apr 2019
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

afernandezody picture afernandezody  路  4Comments
chandanbfx picture chandanbfx  路  7Comments
calebwin picture calebwin  路  4Comments
tom-dyar picture tom-dyar  路  5Comments
demartinofra picture demartinofra  路  4Comments