Aws-parallelcluster: ComputeFleet:Create fails in Cloudformation, but Compute Auto-Scaling Group is still created?

[jcpasion]

I am setting up a pcluster, and I have repeatedly gotten this error when I run create cluster:

• AWS::CloudFormation::Stack parallelcluster-galaxy-HPC The following resource (s) failed to create: [ComputeFleet].
• AWS::AutoScaling::AutoScalingGroup ComputeFleet Received 2 FAILURE signal(s) out of 2. Unable to satisfy 100% MinSuccessfulInstancesPercent requirement

This error message is reproducible throughout various changes to private and public subnets, OS, schedulers, cluster types, and groups. Sometimes I get 1 FAILURE signal, sometimes I get 2, I cannot predict when either case happens. Regardless, the resource is still created and I can see the resource, and it's activity, in the console.

Below is the environment I'm trying to use:

Environment:

AWS ParallelCluster version 2.4

[aws]
aws_region_name = us-east-1

[cluster HPC]
key_name = *****
vpc_settings = vpc
base_os: alinux
scheduler: slurm
initial_queue_size = 2
maintain_initial_size = true
placement_group = DYNAMIC
placement = compute
master instance type = c5.large
compute instance type = c5.large

[vpc vpc]
vpc_id: vpc-*****
master_subnet_id = subnet-*****
vpc_security_group_id = sg-*****

[global]
cluster_template = galaxy-HPC
update_check = true
sanity_check = true

[aliases]
ssh = ssh {CFN_USER}@{MASTER_IP} {ARGS}

Additional context:

• Currently I am only trying to get the stack to create successfully, there are no pre or post install scripts.
• My security group allows inbound traffic from ports 22, 80, 443, 8080, 8081, 8443. It also allows all traffic on any port coming from EC2's sharing the same security group.
  
  
  
  • As of right now, my Master node creates successfully; there is no termination and re initializing so I can SSH in. However, in the compute fleet, all my EC2 instances are created, fail a health check, are terminated, and new EC2's are created, over and over. Thus I cannot SSH into my compute nodes at the moment.
  
  

Attached are my log files as requested, as well as my jobwatcher, slurmctld, and sqswatcher logs. I would greatly appreciate any help!

cloud-init.txt
cloud-init-output.txt
cfn-init.txt
jobwatcher.txt
sqswatcher.txt
slurmctld.txt

[lukeseawalker]

Hi,
could you please send at least one of the archive inside the /home/logs/compute folder in the master node?
What are the outbound rules of your custom security group?

[jcpasion]

Hey there,

I navigated to the /home directory, but there is no folder titled logs in that directory. I see the folders ec2-user, sgeadmin, and slurm, but none of these directories have files in them.

These are my outbound rules for the security group:
`

Ports | Protocol | Destination | Custom SG
-- | -- | -- | --
All | All | 0.0.0.0/0, ::/0 | ✔

`

[lukeseawalker]

The compute nodes are failing even before mounting the /home folder shared by the master.

To understand why the computes are failing we need you to create a cluster with the --norollback option then connect to one of the compute node from your client, using your local key. You said that your custom SG is allowing traffic from ports 22, so you should be able to connect to the compute node through its public IP.

If not, another possibility is to stop one of the compute, detach its volume, attach that volume to the master node and then mount the volume in the master to a given mountpoint, e.g. /mnt/vol_compute. Then retrieve /mnt/vol_compute/var/log/cfn-init.log.

Also, Did you try to create a cluster without using vpc_security_group_id?

[jcpasion]

I halted termination on my autoscaling, SSHed into a compute node, went to /home/logs/compute , and found the file i-02fb18da9162c7e3c.tar. The attached file is parts of the log that had errors.

I will try without a custom SG and will report back.

Attached is the compute nodes cfn-init.log found in /var/log .

compute node cfn_init log.txt

i-02fb18da9162c7e3c.txt

[lukeseawalker]

Relevant error
STDERR: mount.nfs: Failed to resolve server ip-10-42-65-52.ec2.internal: Name or service not known
Compute node is note able to resolve the Master.
How is the VPC configured? Are you using a custom DHPC options sets?

[jcpasion]

I just finished talking to the people who run our AWS account and infrastructure. They told me that the DNS for that VPC is non-functional. Is there a way for me to specify use of the private IP of the master node, instead of the private DNS? Is there a way to even use something like Route53 instead of the autp-generated private DNS to reroute it to the master node?

[jcpasion]

Hey there, just checking in again on this issue. Is there anything I can do about the mount target?

[demartinofra]

Hi,

sorry for the delayed response. Unfortunately the cluster configuration relies on the private FQDNs resolved through the VPC DNS and we don't really have a way at the moment to support private IPs.
One thing you could try to do is to check whether that VPC has enableDnsSupport to true and in that case you could try to force the cluster instances to use the VPC DNS by running this command:
awk '!found && /nameserver/{ print "nameserver 169.254.169.253"; found=1}1' /etc/resolv.conf > /etc/resolv.conf.new and then by replacing /etc/resolv.conf with the new file.

However this might not be enough since you might have to deal with hostnames issues and search domain. Could you paste here the content of /etc/resolv.conf from the cluster master server?

Useful docs: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#vpc-dns-support

[jcpasion]

Hey again,

Here is the output of my etc/resolv.conf file

options timeout:2 attempts:5
; generated by /sbin/dhclient-script
search @@.@@@.@@@@.com
nameserver 170.54.240.117
nameserver 170.54.240.116

@@.@@@@.@@@@.com is our custom domain appended to ip-@@-@@-@@-@@@ instead of ".ec2.internal" due to our custom rules.

The VPC I am using has DNS resolution and DNS hostnames enabled.

Do I apply the command you've written above as a pre-install boostrap script? Do pre-install scripts run before cfn-init?

[demartinofra]

If you are able to modify the DHCP Options Set could you try adding the following:

• if you are in us-east-1 add ec2.internal to domain-name, otherwise specify region.compute.internal (for example, ap-northeast-1.compute.internal).
• add 169.254.169.253 to domain-name-servers

If we switched to using private IPs it might just work for most of the features but I'm afraid not all the schedulers will have a correct configuration (hence some work and testing is required for this). I will try run some tests to verify this. In any case this should be tracked as a feature enhancements since at the moment it is not something we natively supported in ParallelCluster and as I can see you already opened an issue for that.

[jcpasion]

I'd love to be updated on whether or not private IP's are feasible!

I ended up running a pre-install bootstrap script that installed the package unbound onto all instances in the cluster. I set unbound to forward any requests to DNS's that end in our domain-name to domain-name-servers, and all other traffic to 169.254.169.253 . I also updated the resolved.conf file to resolve to our on-prem, and now the mount apparently works: I'm getting no issues with the actual creation of the ParallelCluster Stack. I am now having separate issues with slurm even if the mount says it worked, but I can raise another issue for that. I'm not even sure if this current issue is entirely resolved, or if the slurm issues I have are a part of this. I'll still raise a separate issue and post logs there.

[demartinofra]

I think we can resolve this specific issue since support for private IPs can be tracked as part of this separate feature request: https://github.com/aws/aws-parallelcluster/issues/1201, and the other mentioned issue is being taken care of here: https://github.com/aws/aws-parallelcluster/issues/1206

Feel free to reopen it if I missed anything

[joeydumont]

@jcpasion Mind sharing your unbound.conf? I'll test setting up something similar as well.

[sean-smith]

@jcpasion If you want to writeup your setup with unbound we can post on the github wiki so everyone can benefit. Thanks!

[jcpasion]

Here is my unbound.conf:

apt-get install -y unbound

/etc/unbound/unbound.conf.d/CUSTOMDNS.conf
server:
        verbosity: 1
        ## Specify the interface address to listen on:
        interface: 127.0.0.1
        do-ip4: yes
        do-ip6: yes
        do-udp: yes
        do-tcp: yes
        do-daemonize: yes
        access-control: 127.0.0.0/8 allow
remote-control:
        control-enable: no
forward-zone:
        name: "."
        ## This is a special IP in AWS for DNS resolution
        # https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html
        # https://docs.aws.amazon.com/vpc/latest/userguide/VPC_DHCP_Options.html
        forward-addr: 169.254.169.253
forward-zone:
        name: "[INSERT CUSTOM DOMAIN]"
        ## Your custom IP addresses, and custom DNS
        forward-addr: [CUSTOM IP]
        forward-addr: [CUSTOM IP]

Here is my resolved.conf:

/etc/systemd/resolved.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See resolved.conf(5) for details

[Resolve]
DNS=127.0.0.1
Domains=ec2.internal

And finally I ran this script, provided by #597, to change /etc/resolv.conf:

#!/bin/bash

RESOLV=/etc/resolv.conf
RESOLV_ORIG=/etc/resolv.conf.orig

/bin/cp $RESOLV $RESOLV_ORIG

/bin/sed 's#search#search ec2.internal#' $RESOLV_ORIG > $RESOLV