Aws-parallelcluster: Leverage AWS EC2 Image Builder instead of custom pcluster ami-create

[brainstorm]

I am setting up and maintaining a pcluster for @umccr and after trying to upgrade to the new 2.8.0 release (from 2.6.1) I'm faced with:

% ./bin/start_cluster.sh testcluster5
Beginning cluster creation for cluster: testcluster5
Creating stack named: parallelcluster-testcluster5
Status: parallelcluster-testcluster5 - ROLLBACK_IN_PROGRESS
Cluster creation failed.  Failed events:
  - AWS::AutoScaling::AutoScalingGroup ComputeFleet Resource creation cancelled
  - AWS::CloudFormation::WaitCondition MasterServerWaitCondition Received FAILURE signal with UniqueId i-0aefe545051e1fba5

Which I haven't digged through hard with it with --norollback and grabbing the relevant logfiles as issue #1902 suggests, but I suspect it's going to be related with custom AMI issues, as many issues on this bugtracker hint towards

Now, since customers will continue doing customs AMIs anyway, wouldn't it be nice to have an EC2 AMI builder component that ensures that the latest configuration on the parallel cluster AMIs (custom or managed) are properly deployed and sound? Here's mine (in progress) for instance:

As @demartinofra points out:

To make this even more clear for customers we are working on introducing additional validations so that this is prevented by the CLI at cluster creation time.

Wouldn't it make sense to leverage the (managed) EC2 AMI builder instead of baking further custom logic into the pcluster chef & co CLI? At least as a base configuration/validation, later on customers can indeed go nuts with Chef and other IaC tools.

[brainstorm]

For reference, here's the current deployment and operation scripts (running on Amazon Linux 2020-07 AMI):

https://github.com/umccr/infrastructure/tree/master/parallel_cluster

And the current logfiles after launching with --no-rollback so that I could examine the issue post-Cfn-failure:

cfn-init.log
cloud-init-output.log
cloud-init.log

The underlying errors might be unrelated with 2.8.0 upgrade (since I've been performing other changes) but the EC2 Image builder point still stands: gives cluster power users easy access to install new software via rolling new AMIs on the web console, super handy!

[demartinofra]

Hi Roman,

thanks for opening this and proposing a potential approach to improve the custom AMI creation functionalities in ParallelCluster.

You are right in the fact that adding a validation to prevent a user from using an incorrect AMI with ParallelCluster is not going to improve the overall story of building you own custom AMI. However this is a simple validation/check that can save customers hours of debugging and that we can quickly introduce in the upcoming version.

Now going back to your suggestion of improving the custom AMI building workflow by eventually leveraging EC2 Image builder as the tool, I think that makes a lot of sense and it is something we are already evaluating as part of our broader goal that aims to improve the customer experience when customizing ParallelCluster AMIs. For this I'm going to mark the issues as a request for enhancement so that we can make sure we consider your use-case when designing the new feature. Thank you for sharing with us the work you've been doing for your customer, it'll be for sure a valid source of inspiration!

Last but not least, looking at the error from the logs you shared, it seems like there is something broken with the python virtualenv that ParallelCluster configures for recipes execution and this is most likely related to a custom_ami you are using.

---- Begin output of /opt/parallelcluster/pyenv/versions/3.6.9/envs/cookbook_virtualenv/bin/aws fsx --region ap-southeast-2 describe-file-systems --file-system-ids fs-09a50ce6a2a522df2 --query 'FileSystems[0].LustreConfiguration.MountName' --output text ----
STDOUT:
STDERR: sh: /opt/parallelcluster/pyenv/versions/3.6.9/envs/cookbook_virtualenv/bin/aws: No such file or directory

If you need further support on that we need additional info on your cluster config and custom_ami setup.

Cheers,
Francesco

[brainstorm]

Thanks for the feedback!

You are right in the fact that adding a validation to prevent a user from using an incorrect AMI with ParallelCluster is not going to improve the overall story of building you own custom AMI. However this is a simple validation/check that can save customers hours of debugging and that we can quickly introduce in the upcoming version.

Hey, I wasn't arguing against, I was agreeing with the approach: give customers a "known, working and validated base" of pcluster and let them carry on with custom on top additions thereafter ;)

Now my cluster config and current status (as of today):

cluster config: https://github.com/umccr/infrastructure/blob/master/parallel_cluster/conf/config
custom_ami (component) setup:

name: Bioinformatics
description: Bioinformatics software and dependencies
schemaVersion: 1.0

phases:
  - name: build
    steps:
      - name: base-extras
        action: ExecuteBash
        inputs:
          commands:
            - yum install -y amazon-linux-extras
            - yum install -y tmux htop vim
            - amazon-linux-extras enable R3.4 epel vim docker=latest rust1
            - pip3 install aws-parallelcluster --upgrade --user
            - mkdir -p /scratch && chown ec2-user:ec2-user /scratch
      # https://github.com/dcbark01/AnacondaInstallEC2/blob/master/install.sh
      - name: anaconda
        action: ExecuteBash
        inputs:
          commands:
            - su - ec2-user -c "wget https://repo.anaconda.com/archive/Anaconda3-2020.07-Linux-x86_64.sh && chmod +x *.sh"
            - su - ec2-user -c "./Anaconda3-2020.07-Linux-x86_64.sh -b -p /home/ec2-user/.anaconda3"
            - su - ec2-user -c "rm /home/ec2-user/Anaconda3-*.sh"
            - su - ec2-user -c "echo '. /home/ec2-user/.anaconda3/etc/profile.d/conda.sh' >> /home/ec2-user/.bashrc"
      - name: facets-biocontainer
        action: ExecuteBash
        inputs:
          commands:
            - docker pull "quay.io/biocontainers/r-facets:0.5.14--r40he991be0_3"

Current errors after further fixing/debugging seem to be related to parallelcluster trying to install dpkg-dev???... on an Amazon Linux 2 instance? (arn:aws:imagebuilder:ap-southeast-2:aws:image/amazon-linux-2-x86/x.x.x which is latest 2020-06 Amazon Linux 2, the 2020-07, present yesterday, seems to be gone today from the EC2 Image Builder web console):

(...)
Recipe: aws-parallelcluster::base_install
  * yum_package[vim, ksh, tcsh, zsh, openssl-devel, ncurses-devel, pam-devel, net-tools, openmotif-devel, libXmu-devel, hwloc-devel, tcl-devel, automake, autoconf, pyparted, libtool, httpd, boost-devel, mlocate, atlas-devel, fftw-devel, libffi-devel, dkms, mysql-devel, libedit-devel, postgresql-devel, postgresql-server, sendmail, cmake, byacc, libglvnd-devel, mdadm, libgcrypt-devel, libevent-devel, glibc-static, libdb-devel, system-lsb, libxml2-devel, perl-devel, dpkg-dev, tar, gzip, bison, flex, gcc, gcc-c++, patch, rpm-build, rpm-sign, system-rpm-config, cscope, ctags, diffstat, doxygen, elfutils, gcc-gfortran, git, indent, intltool, patchutils, rcs, subversion, swig, systemtap, curl, jq, wget, python-pip] action install
    * No candidate version available for dpkg-dev
    * No candidate version available for dpkg-dev
    * No candidate version available for dpkg-dev
    * No candidate version available for dpkg-dev
    * No candidate version available for dpkg-dev
    * No candidate version available for dpkg-dev
    * No candidate version available for dpkg-dev
    * No candidate version available for dpkg-dev
    * No candidate version available for dpkg-dev
    * No candidate version available for dpkg-dev
    * No candidate version available for dpkg-dev
    ================================================================================
    Error executing action `install` on resource 'yum_package[vim, ksh, tcsh, zsh, openssl-devel, ncurses-devel, pam-devel, net-tools, openmotif-devel, libXmu-devel, hwloc-devel, tcl-devel, automake, autoconf, pyparted, libtool, httpd, boost-devel, mlocate, atlas-devel, fftw-devel, libffi-devel, dkms, mysql-devel, libedit-devel, postgresql-devel, postgresql-server, sendmail, cmake, byacc, libglvnd-devel, mdadm, libgcrypt-devel, libevent-devel, glibc-static, libdb-devel, system-lsb, libxml2-devel, perl-devel, dpkg-dev, tar, gzip, bison, flex, gcc, gcc-c++, patch, rpm-build, rpm-sign, system-rpm-config, cscope, ctags, diffstat, doxygen, elfutils, gcc-gfortran, git, indent, intltool, patchutils, rcs, subversion, swig, systemtap, curl, jq, wget, python-pip]'
    ================================================================================

    Chef::Exceptions::Package
    -------------------------
    No candidate version available for dpkg-dev

    Resource Declaration:
    ---------------------
    # In /etc/chef/local-mode-cache/cache/cookbooks/aws-parallelcluster/recipes/base_install.rb

     56: package node['cfncluster']['base_packages'] do
     57:   retries 10
     58:   retry_delay 5
     59: end
     60:

    Compiled Resource:
    ------------------
    # Declared in /etc/chef/local-mode-cache/cache/cookbooks/aws-parallelcluster/recipes/base_install.rb:56:in `from_file'

cfn-init.log
cloud-init-output.log
cloud-init.log

Any hints on the origin of those dpkg-dev errors @demartinofra ?

[brainstorm]

@demartinofra I just pushed the YAML component files I use with EC2 AMI Builder here:

https://github.com/umccr/infrastructure/tree/master/parallel_cluster/ami

I hope that helps diagnosing the problem from your end? :)

[brainstorm]

@demartinofra Our team is running through other Parallelcluster 2.8.0 Chef deployment issues on https://github.com/umccr/infrastructure/pull/64, this time is related to awscli and FSx mounting:

  ================================================================================
  Recipe Compile Error in /etc/chef/local-mode-cache/cache/cookbooks/aws-parallelcluster/recipes/slurm_config.rb
  ================================================================================

  Mixlib::ShellOut::ShellCommandFailed
  ------------------------------------
  Expected process to exit with [0], but received '127'
  ---- Begin output of /opt/parallelcluster/pyenv/versions/3.6.9/envs/cookbook_virtualenv/bin/aws fsx --region ap-southeast-2 describe-file-systems --file-system-ids fs-07a9895a51552e844 --query 'FileSystems[0].LustreConfiguration.MountName' --output text ----
  STDOUT: 
  STDERR: sh: /opt/parallelcluster/pyenv/versions/3.6.9/envs/cookbook_virtualenv/bin/aws: No such file or directory
  ---- End output of /opt/parallelcluster/pyenv/versions/3.6.9/envs/cookbook_virtualenv/bin/aws fsx --region ap-southeast-2 describe-file-systems --file-system-ids fs-07a9895a51552e844 --query 'FileSystems[0].LustreConfiguration.MountName' --output text ----
  Ran /opt/parallelcluster/pyenv/versions/3.6.9/envs/cookbook_virtualenv/bin/aws fsx --region ap-southeast-2 describe-file-systems --file-system-ids fs-07a9895a51552e844 --query 'FileSystems[0].LustreConfiguration.MountName' --output text returned 127

  Cookbook Trace:
  ---------------
    /etc/chef/local-mode-cache/cache/cookbooks/aws-parallelcluster/recipes/fsx_mount.rb:35:in `from_file'
    /etc/chef/local-mode-cache/cache/cookbooks/aws-parallelcluster/recipes/base_config.rb:78:in `from_file'
    /etc/chef/local-mode-cache/cache/cookbooks/aws-parallelcluster/recipes/slurm_config.rb:18:in `from_file'

  Relevant File Content:
  ----------------------
  /etc/chef/local-mode-cache/cache/cookbooks/aws-parallelcluster/recipes/fsx_mount.rb:

   28:      owner 'root'
   29:      group 'root'
   30:      mode '1777'
   31:      action :create
   32:    end
   33:  
   34:    require 'chef/mixin/shell_out'
   35>>   mountname = shell_out!("#{node['cfncluster']['cookbook_virtualenv_path']}/bin/aws fsx --region #{node['cfncluster']['cfn_region']} describe-file-systems --file-system-ids #{node['cfncluster']['cfn_fsx_fs_id']} --query 'FileSystems[0].LustreConfiguration.MountName' --output text", :user=>'root').stdout.strip
   36:  
   37:    mount_options = %w[defaults _netdev flock user_xattr noatime]
   38:  
   39:    mount_options.push(%w[noauto x-systemd.automount]) if node['init_package'] == 'systemd'
   40:  
   41:    # Mount FSx over NFS
   42:    mount fsx_shared_dir do
   43:      device "#{node['cfncluster']['cfn_fsx_fs_id']}.fsx.#{node['cfncluster']['cfn_region']}.#{node['cfncluster']['aws_domain']}@tcp:/#{mountname}"
   44:      fstype 'lustre'

  System Info:
  ------------
  chef_version=15.11.8
  platform=amazon
  platform_version=2
  ruby=ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
  program_name=/opt/cinc/bin/cinc-client
  executable=/opt/cinc/bin/cinc-client

More details on the ongoing Draft PR outlined above.

/cc @alexiswl @reisingerf @victorskl

[lukeseawalker]

Hi,
as @demartinofra pointed out "there is something broken with the python virtualenv that ParallelCluster configures for recipes execution and this is most likely related to a custom_ami you are using", in particular no aws cli is found inside the pcluster cookbook virtual env at /opt/parallelcluster/pyenv/versions/3.6.9/envs/cookbook_virtualenv/bin/aws

I have briefly looked at your AMI build process and I think your not triggering any pcluster AMI build process: your are starting the EC2 image build pipeline from a vanilla AMI, installing some packages defined here and than using the output AMI as value for the custom_ami parameter. This AMI is still not "parallelcluster-ized" as it doesn't contains all the software packages and configuration required by ParallelCluster. So, when you will spin up a cluster creation, all the softwares and configurations will be done "at runtime", increasing the cluster creation time, the scaling up time and, in your case, having conflicts during the installation/configuration of such components which is driving the cluster creation to fail.

I would suggest you to try another approach which is basically a modification of the one describe here in the section Modify an AWS ParallelCluster AMI: try start your image build process from a ParallelCluster AMI.

Retrieve the ParallelCluster AMI from running the query

aws ec2 describe-images --region <region> --owners "amazon" --filters "Name=name,Values=aws-parallelcluster-<version>-amzn2-*" "Name=architecture,Values=x86_64" --query 'reverse(sort_by(Images, &CreationDate))[:1].ImageId'  --output text

where <region> is your region and <version> is the version you want to use. Then use that AMI as the input of your EC2 image build process.

Hope this could help.
L

[alexiswl]

After commenting out the custom_ami attribute and also specifying custom_ami to be the ami-056a324d5f89ea398 pcluster ready AMI, we are getting this issue:

Aug 13 03:54:12 cloud-init[3903]: util.py[DEBUG]: Failed running /var/lib/cloud/instance/scripts/part-002
 [1]
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 910, in runparts
    subp(prefix + [exe_path], capture=False, shell=True)
  File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 2105, in subp
    cmd=args)
ProcessExecutionError: Unexpected error while running command.
Command: ['/var/lib/cloud/instance/scripts/part-002']
Exit code: 1
Reason: -
Stdout: -
Stderr: -
Aug 13 03:54:12 cloud-init[3903]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
Aug 13 03:54:12 cloud-init[3903]: handlers.py[DEBUG]: finish: modules-final/config-scripts-user: FAIL: running config-scripts-user with frequency once-per-instance
Aug 13 03:54:12 cloud-init[3903]: util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
Aug 13 03:54:12 cloud-init[3903]: util.py[DEBUG]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cloudinit/stages.py", line 851, in _run_modules
    freq=freq)
  File "/usr/lib/python2.7/site-packages/cloudinit/cloud.py", line 54, in run
    return self._runners.run(name, functor, args, freq, clear_on_fail)
  File "/usr/lib/python2.7/site-packages/cloudinit/helpers.py", line 187, in run
    results = functor(*args)
  File "/usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.py", line 45, in handle
    util.runparts(runparts_path)
  File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 917, in runparts
    % (len(failed), len(attempted)))
RuntimeError: Runparts: 1 failures in 1 attempted commands

I dug into this a little further and ran the script /var/lib/cloud/instance/scripts/part-002 manually through the sudo user.
It failed at the line:

cfn-init -s parallelcluster-AlexisFirstCluster -vvv -c default -r MasterServerLaunchTemplate --region ap-southeast-2

with:

Error occurred during build: Command chef failed

After looking at the logs at /var/log/cfn-init.log I found this file:

/etc/chef/local-mode-cache/cache/cinc-stacktrace.out

which had the following error:

Generated at 2020-08-13 04:17:05 +0000
Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '255'
---- Begin output of /opt/parallelcluster/pyenv/versions/3.6.9/envs/cookbook_virtualenv/bin/aws fsx --region ap-southeast-2 describe-file-systems --file-system-ids fs-0a7f9e967c25c67e8 --query 'FileSystems[0].LustreConfiguration.MountName' --output text ----
STDOUT: 
STDERR: An error occurred (AccessDeniedException) when calling the DescribeFileSystems operation: User: arn:aws:sts::843407916570:assumed-role/parallelcluster-ec2-instance-role/i-020c1d3067192c9c4 is not authorized to perform: fsx:DescribeFileSystems on resource: arn:aws:fsx:ap-southeast-2:843407916570:file-system/*
---- End output of /opt/parallelcluster/pyenv/versions/3.6.9/envs/cookbook_virtualenv/bin/aws fsx --region ap-southeast-2 describe-file-systems --file-system-ids fs-0a7f9e967c25c67e8 --query 'FileSystems[0].LustreConfiguration.MountName' --output text ----

Updating the user role to have AmazonFSxFullAccess resolved the issue

[lukeseawalker]

Hi @alexiswl,
the DescribeFileSystems policy is present in the default instance role created by ParallelCluster, see here.
Are you using a custom role?
Apart from that, were you able to create the cluster?

Thanks,
L

[alexiswl]

Hi Luke,

Yes after moving to using additional_iam_policies over ec2_instance_role and basing our installation off a base parallel cluster we were able to get this working.
Thank you for your assistance, it is much appreciated.
Could I clarify if AmazonSSMManagedInstanceCore is a policy that is included in the default instance role you describe above?
If not, is there a reason it isn't. It's the only one I'm currently adding into the additional_iam_policies in the config attribute.

Alexis.

[lukeseawalker]

It's not included as you can see from https://github.com/aws/aws-parallelcluster/blob/v2.8.0/cloudformation/aws-parallelcluster.cfn.json#L1753-L1921.
Reason is because it's not a service currently leveraged through ParallelCluster by default so it is left to the user to add it as additional policy or through custom role.

L

[alexiswl]

@brainstorm I think we can close this yes?

[brainstorm]

Yup, works for us (tm) ;)