Aws-parallelcluster: ubuntu1604 with SGE fails to deploy compute fleet when ganglia is enabled

[chrisdag]

Summary:

When I turn on Ganglia via extra_json the v2.5.1 deploy fails because the compute nodes bomb out on a chef error related to signaling apache. Running with "-nr" for no rollback causes an endless loop of compute node creation|termination so it is hard to get the full logs and stacks out of the system

The relevant error appears here in /var/log/cfn-init.log on a compute fleet node:

[2020-01-23T20:20:50+00:00] FATAL: Chef::Exceptions::ResourceNotFound: resource execute[copy ganglia apache conf] is configured to notify resource service[apache2] with action reload, but service[apache2] cannot be found in the resource collection. execute[copy ganglia apache conf] is defined in /etc/chef/local-mode-cache/cache/cookbooks/aws-parallelcluster/recipes/_ganglia_install.rb:146:in `from_file'

Environment:

• aws-parallelcluster-2.5.1
• Config

[aws]
aws_region_name = us-east-1

[global]
cluster_template = XXREDACTEDXX
update_check = true
sanity_check = true

[aliases]
ssh = ssh {CFN_USER}@{MASTER_IP} {ARGS}

[cluster XXREDACTEDXX]
cluster_type = ondemand
placement_group = DYNAMIC

ebs_settings = OLD-MASTER-ROOT-VOL, OLD-MASTER-SHARED-VOL, NEW-SHARED-VOL, NEW-

HOME-VOL
ebs_settings = NEW-SHARED-VOL, NEW-DATA-VOL
key_name = XXREDACTEDXX
ec2_iam_role = ParallelCluster_Role
base_os = ubuntu1604
compute_instance_type = c5.9xlarge
master_instance_type = m5.large
scheduler = sge
initial_queue_size = 1
max_queue_size = 8
maintain_initial_size = false
custom_ami = ami-08190b7e8bce18017
# 23-January- Custom AMI
master_root_volume_size = 100
compute_root_volume_size = 50
ephemeral_dir = /scratch
tags = { "Platform" : "HPC", "Business" : XXREDACTEDXX", "Project": "XXREDACTEDXX", "Business Owner" : "XXREDACTEDXX", "IT
Owner" : "XXREDACTEDXX", "Description" : "Auto-scaling HPC grid based off of
AWS ParallelCluster v2.5.1 as deployed Jan 2020" }
vpc_settings = XXREDACTEDXX
extra_json = { "cluster" : { "ganglia_enabled" : "yes" } }

[ebs NEW-SHARED-VOL]
shared_dir = shared
ebs_snapshot_id = snap-XXREDACTEDXX
# 23-January snapshot

[ebs NEW-DATA-VOL]
shared_dir = data
ebs_snapshot_id = snap-XXREDACTEDXX
# 23-January snapshot

[vpc XXREDACTEDXX]
vpc_id = vpc-25ada25c
vpc_security_group_id = sg-XXREDACTEDXX
use_public_ips = false
master_subnet_id = subnet-XXREDACTEDXX
compute_subnet_id = subnet-XXREDACTEDXX

Bug description and how to reproduce:
Launch v2.5.1 with ubuntu1604 and SGE as the schedule with Ganglia enabled via extra_json

Additional context:

Here are the log entries I was able to pull from the compute node before it terminated on me (despite launching with -nr )

2020-01-23 20:20:35,236 [INFO] Running config chefPrepEnv
2020-01-23 20:20:35,237 [DEBUG] No packages specified
2020-01-23 20:20:35,237 [DEBUG] No groups specified
2020-01-23 20:20:35,237 [DEBUG] No users specified
2020-01-23 20:20:35,237 [DEBUG] No sources specified
2020-01-23 20:20:35,237 [DEBUG] No files specified
2020-01-23 20:20:35,237 [DEBUG] Running command chef
2020-01-23 20:20:35,237 [DEBUG] No test for command chef
2020-01-23 20:20:47,680 [INFO] Command chef succeeded
2020-01-23 20:20:47,680 [DEBUG] Command chef output: Starting Chef Client, version 14.2.0
[2020-01-23T20:20:45+00:00] WARN: Run List override has been provided.
[2020-01-23T20:20:45+00:00] WARN: Run List override has been provided.
[2020-01-23T20:20:45+00:00] WARN: Original Run List: [recipe[aws-parallelcluster::sge_config]]
[2020-01-23T20:20:45+00:00] WARN: Original Run List: [recipe[aws-parallelcluster::sge_config]]
[2020-01-23T20:20:45+00:00] WARN: Overridden Run List: [recipe[aws-parallelcluster::_prep_env]]
[2020-01-23T20:20:45+00:00] WARN: Overridden Run List: [recipe[aws-parallelcluster::_prep_env]]
resolving cookbooks for run list: ["aws-parallelcluster::_prep_env"]
Synchronizing Cookbooks:

• aws-parallelcluster (2.5.1)
• poise-python (1.7.0)
• tar (2.1.1)
• selinux (2.1.1)
• nfs (2.6.3)
• yum (5.1.0)
• yum-epel (3.1.0)
• openssh (2.6.3)
• apt (7.0.0)
• hostname (0.4.2)
• line (2.4.1)
• ulimit (1.0.0)
• pyenv (3.1.1)
• kernel_module (1.1.2)
• poise (2.8.2)
• poise-languages (2.1.2)
• hostsfile (3.0.1)
• iptables (4.5.0)
• poise-archive (1.5.0)
  Installing Cookbook Gems:
  Compiling Cookbooks...
  Converging 8 resources
  Recipe: aws-parallelcluster::_prep_env
• directory[/etc/parallelcluster] action create (up to date)
• directory[/opt/parallelcluster] action create (up to date)
• directory[/opt/parallelcluster/scripts] action create (up to date)
• directory[/var/log/parallelcluster/] action create
  
  
  
  • create new directory /var/log/parallelcluster/
  
  
  • change mode from '' to '01777'
  
  
  • change owner from '' to 'root'
  
  
• template[/etc/parallelcluster/cfnconfig] action create
  
  
  
  • create new file /etc/parallelcluster/cfnconfig
  
  
  • update content in file /etc/parallelcluster/cfnconfig from none to ce669b
    
    --- /etc/parallelcluster/cfnconfig 2020-01-23 20:20:47.650868553 +0000
    
    +++ /etc/parallelcluster/.chef-cfnconfig20200123-2582-1y0vvhe 2020-01-23 20:20:47.650868553 +0000
    
    @@ -1 +1,18 @@
    
    +stack_name=XXREDACTEDXX
    
    +cfn_preinstall=NONE
    
    +cfn_preinstall_args=NONE
    
    +cfn_postinstall=NONE
    
    +cfn_postinstall_args=NONE
    
    +cfn_region=us-east-1
    
    +cfn_scheduler=sge
    
    +cfn_scheduler_slots=vcpus
    
    +cfn_instance_slots=36
    
    +cfn_encrypted_ephemeral=false
    
    +cfn_ephemeral_dir=/scratch
    
    +cfn_shared_dir=shared,data,NONE,NONE,NONE
    
    +cfn_proxy=NONE
    
    +cfn_node_type=ComputeFleet
    
    +cfn_cluster_user=ubuntu
    
    +cfn_sqs_queue=https://sqs.us-east-1.amazonaws.com/327751404256/XXREDACTEDXX
    
    +cfn_master=ip-10-237-127-251.ec2.internal
  
  
  • change mode from '' to '0644'
  
  
• link[/opt/parallelcluster/cfnconfig] action create
  
  
  
  • create symlink at /opt/parallelcluster/cfnconfig to /etc/parallelcluster/cfnconfig
  
  
• template[/opt/parallelcluster/scripts/fetch_and_run] action create
  
  
  
  • create new file /opt/parallelcluster/scripts/fetch_and_run
  
  
  • update content in file /opt/parallelcluster/scripts/fetch_and_run from none to be1416
    
    --- /opt/parallelcluster/scripts/fetch_and_run 2020-01-23 20:20:47.654868527 +0000
    
    +++ /opt/parallelcluster/scripts/.chef-fetch_and_run20200123-2582-bob3yb 2020-01-23 20:20:47.654868527 +0000
    
    @@ -1 +1,66 @@
    
    +#!/bin/bash
    
    +
    
    +. /etc/parallelcluster/cfnconfig
    
    +
    
    +# Error exit function
    
    +function error_exit () {
  
  
  • script=basename $0
  
  
  • echo "parallelcluster: $script - $1"
  
  
  • logger -t parallelcluster "$script - $1"
  
  
  • exit 1
    
    +}
    
    +
    
    +function download_run (){
  
  
  • url=$1
  
  
  • scheme=$(echo "${url}"| cut -d: -f1)
  
  
  • tmpfile=$(mktemp)
  
  
  • trap "/bin/rm $tmpfile" RETURN
  
  
  • if [ "${scheme}" == "s3" ]; then
  
  
  • /root/.pyenv/versions/3.6.9/envs/cookbook_virtualenv/bin/aws --region ${cfn_region} s3 cp ${url} - > $tmpfile || return 1
  
  
  • else
  
  
  • wget -qO- ${url} > $tmpfile || return 1
  
  
  • fi
  
  
  • chmod +x $tmpfile || return 1
  
  
  • $tmpfile $@ || error_exit "Failed to run boot_as_master $ACTION, $file failed with non 0 return code: $?"
    
    +}
    
    +
    
    +function run_preinstall () {
  
  
  • if [ "${cfn_preinstall}" != "NONE" ]; then
  
  
  • file="${cfn_preinstall}"
  
  
  • if [ "${cfn_preinstall_args}" != "NONE" ]; then
  
  
  • download_run ${cfn_preinstall} ${cfn_preinstall_args}
  
  
  • else
  
  
  • download_run ${cfn_preinstall}
  
  
  • fi
  
  
  • fi || error_exit "Failed to run boot_as_master preinstall"
    
    +}
    
    +
    
    +function run_postinstall () {
  
  
  • RC=0
  
  
  • if [ "${cfn_postinstall}" != "NONE" ]; then
  
  
  • file="${cfn_postinstall}"
  
  
  • if [ "${cfn_postinstall_args}" != "NONE" ]; then
  
  
  • download_run ${cfn_postinstall} ${cfn_postinstall_args}
  
  
  • else
  
  
  • download_run ${cfn_postinstall}
  
  
  • fi
  
  
  • fi || error_exit "Failed to run boot_as_master postinstall"
    
    +}
    
    +
    
    +ACTION=${1#?}
    
    +
    
    +case $ACTION in
  
  
  • preinstall)
  
  
  • run_preinstall
  
  
  • ;;
    
    +
  
  
  • postinstall)
  
  
  • run_postinstall
  
  
  • ;;
    
    +
  
  
  • *)
  
  
  • echo "Unknown action. Exit gracefully"
  
  
  • exit 0
    
    +
    
    +esac
  
  
  • change mode from '' to '0755'
  
  
  • change owner from '' to 'root'
  
  
  • change group from '' to 'root'
  
  
• template[/opt/parallelcluster/scripts/compute_ready] action create
  
  
  
  • create new file /opt/parallelcluster/scripts/compute_ready
  
  
  • update content in file /opt/parallelcluster/scripts/compute_ready from none to ff2d4c
    
    --- /opt/parallelcluster/scripts/compute_ready 2020-01-23 20:20:47.658868501 +0000
    
    +++ /opt/parallelcluster/scripts/.chef-compute_ready20200123-2582-9j92td 2020-01-23 20:20:47.658868501 +0000
    
    @@ -1 +1,14 @@
    
    +#!/bin/bash
    
    +set -e
    
    +
    
    +. /etc/parallelcluster/cfnconfig
    
    +
    
    +# Notify compute is ready
    
    +instance_id_url="http://169.254.169.254/latest/meta-data/instance-id"
    
    +instance_id=$(curl --retry 3 --retry-delay 0 --silent --fail ${instance_id_url})
    
    +instance_type_url="http://169.254.169.254/latest/meta-data/instance-type"
    
    +instance_type=$(curl --retry 3 --retry-delay 0 --silent --fail ${instance_type_url})
    
    +local_hostname_url="http://169.254.169.254/latest/meta-data/local-hostname"
    
    +local_hostname=$(curl --retry 3 --retry-delay 0 --silent --fail ${local_hostname_url})
    
    +/root/.pyenv/versions/3.6.9/envs/cookbook_virtualenv/bin/aws --region ${cfn_region} sqs send-message --queue-url ${cfn_sqs_queue} --message-body '{"Type" : "Notification", "Message" : "{\"StatusCode\":\"Complete\",\"Description\":\"Succesfully launched '${instance_id}'\",\"Event\":\"parallelcluster:COMPUTE_READY\",\"EC2InstanceId\":\"'${instance_id}'\",\"Slots\":\"'${cfn_instance_slots}'\",\"LocalHostname\":\"'${local_hostname}'\",\"EC2InstanceType\":\"'${instance_type}'\"}"}'
  
  
  • change mode from '' to '0755'
  
  
  • change owner from '' to 'root'
  
  
  • change group from '' to 'root'
    
    [2020-01-23T20:20:47+00:00] WARN: Skipping final node save because override_runlist was given
    
    [2020-01-23T20:20:47+00:00] WARN: Skipping final node save because override_runlist was given
  
  

Running handlers:
Running handlers complete
Chef Client finished, 5/8 resources updated in 03 seconds

2020-01-23 20:20:47,680 [DEBUG] No services specified
2020-01-23 20:20:47,682 [INFO] Running config shellRunPreInstall
2020-01-23 20:20:47,682 [DEBUG] No packages specified
2020-01-23 20:20:47,682 [DEBUG] No groups specified
2020-01-23 20:20:47,682 [DEBUG] No users specified
2020-01-23 20:20:47,682 [DEBUG] No sources specified
2020-01-23 20:20:47,682 [DEBUG] No files specified
2020-01-23 20:20:47,682 [DEBUG] Running command runpreinstall
2020-01-23 20:20:47,682 [DEBUG] No test for command runpreinstall
2020-01-23 20:20:47,686 [INFO] Command runpreinstall succeeded
2020-01-23 20:20:47,686 [DEBUG] Command runpreinstall output:
2020-01-23 20:20:47,686 [DEBUG] No services specified
2020-01-23 20:20:47,747 [INFO] Running config chefConfig
2020-01-23 20:20:47,748 [DEBUG] No packages specified
2020-01-23 20:20:47,748 [DEBUG] No groups specified
2020-01-23 20:20:47,748 [DEBUG] No users specified
2020-01-23 20:20:47,748 [DEBUG] No sources specified
2020-01-23 20:20:47,748 [DEBUG] No files specified
2020-01-23 20:20:47,748 [DEBUG] Running command chef
2020-01-23 20:20:47,748 [DEBUG] No test for command chef
2020-01-23 20:20:50,413 [ERROR] Command chef (chef-client --local-mode --config /etc/chef/client.rb --log_level auto --force-formatter --no-color --chef-zero-port 8889 --json-attributes /etc/chef/dna.json) failed
2020-01-23 20:20:50,413 [DEBUG] Command chef output: Starting Chef Client, version 14.2.0
resolving cookbooks for run list: ["aws-parallelcluster::sge_config"]
Synchronizing Cookbooks:

• poise-python (1.7.0)
• tar (2.1.1)
• selinux (2.1.1)
• nfs (2.6.3)
• yum (5.1.0)
• yum-epel (3.1.0)
• openssh (2.6.3)
• aws-parallelcluster (2.5.1)
• apt (7.0.0)
• hostname (0.4.2)
• line (2.4.1)
• ulimit (1.0.0)
• pyenv (3.1.1)
• kernel_module (1.1.2)
• poise (2.8.2)
• poise-languages (2.1.2)
• iptables (4.5.0)
• hostsfile (3.0.1)
• poise-archive (1.5.0)
  Installing Cookbook Gems:
  Compiling Cookbooks...
  Converging 134 resources

Running handlers:
[2020-01-23T20:20:50+00:00] ERROR: Running exception handlers
[2020-01-23T20:20:50+00:00] ERROR: Running exception handlers
Running handlers complete
[2020-01-23T20:20:50+00:00] ERROR: Exception handlers complete
[2020-01-23T20:20:50+00:00] ERROR: Exception handlers complete
Chef Client failed. 0 resources updated in 01 seconds
[2020-01-23T20:20:50+00:00] FATAL: Stacktrace dumped to /etc/chef/local-mode-cache/cache/chef-stacktrace.out
[2020-01-23T20:20:50+00:00] FATAL: Stacktrace dumped to /etc/chef/local-mode-cache/cache/chef-stacktrace.out
[2020-01-23T20:20:50+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2020-01-23T20:20:50+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2020-01-23T20:20:50+00:00] FATAL: Chef::Exceptions::ResourceNotFound: resource execute[copy ganglia apache conf] is configured to notify resource service[apache2] with action reload, but service[apache2] cannot be found in the resource collection. execute[copy ganglia apache conf] is defined in /etc/chef/local-mode-cache/cache/cookbooks/aws-parallelcluster/recipes/_ganglia_install.rb:146:in `from_file'

[2020-01-23T20:20:50+00:00] FATAL: Chef::Exceptions::ResourceNotFound: resource execute[copy ganglia apache conf] is configured to notify resource service[apache2] with action reload, but service[apache2] cannot be found in the resource collection. execute[copy ganglia apache conf] is defined in /etc/chef/local-mode-cache/cache/cookbooks/aws-parallelcluster/recipes/_ganglia_install.rb:146:in `from_file'

2020-01-23 20:20:50,413 [ERROR] Error encountered during build of chefConfig: Command chef failed
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 542, in run_config
CloudFormationCarpenter(config, self._auth_config).build(worklog)
File "/usr/local/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 260, in build
changes['commands'] = CommandTool().apply(self._config.commands)
File "/usr/local/lib/python2.7/dist-packages/cfnbootstrap/command_tool.py", line 117, in apply
raise ToolError(u"Command %s failed" % name)
ToolError: Command chef failed
2020-01-23 20:20:50,414 [ERROR] -----------------------BUILD FAILED!------------------------
2020-01-23 20:20:50,415 [ERROR] Unhandled exception during build: Command chef failed
Traceback (most recent call last):
File "/usr/local/bin/cfn-init", line 171, in
worklog.build(metadata, configSets)
File "/usr/local/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 129, in build
Contractor(metadata).build(configSets, self)
File "/usr/local/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 530, in build
self.run_config(config, worklog)
File "/usr/local/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 542, in run_config
CloudFormationCarpenter(config, self._auth_config).build(worklog)
File "/usr/local/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 260, in build
changes['commands'] = CommandTool().apply(self._config.commands)
File "/usr/local/lib/python2.7/dist-packages/cfnbootstrap/command_tool.py", line 117, in apply
raise ToolError(u"Command %s failed" % name)
ToolError: Command chef failed
2020-01-23 20:20:52,244 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.us-east-1.amazonaws.com
2020-01-23 20:20:52,244 [DEBUG] Signaling resource ComputeFleet in stack parallelcluster-XXREDACTEDXX with unique ID i-0a6c067015d2876c6 and status FAILURE
Connection to 10.237.127.189 closed by remote host.
Connection to 10.237.127.189 closed.

[demartinofra]

Hi @chrisdag

thanks for reporting this. The issue has been already fixed by https://github.com/aws/aws-parallelcluster-cookbook/pull/469 but hasn't been released yet.
I'm going to provide you a workaround to get unblocked and enable Ganglia on Ubuntu16 until this gets fixed in the upcoming release. I'll update this soon.

[demartinofra]

To get Ganglia working on Ubuntu 16 and ParallelCluster 2.5.1 please use the following as pre_install script in your cluster. Instructions on how to do it are here (tl;dr; upload this script into an s3 bucket and make sure the cluster nodes have access to it, then use pre_install cluster config option to enable this action)

#!/bin/bash

. "/etc/parallelcluster/cfnconfig"

if [ "${cfn_node_type}" = "ComputeFleet" ]; then
  sed -i -e '/Setup ganglia-web.conf apache config/,+6d' /etc/chef/cookbooks/aws-parallelcluster/recipes/_ganglia_install.rb
fi

[lukeseawalker]

AWS ParallelCluster 2.6.0 with the fix mentioned above has been released