Aws-parallelcluster: no cloudwatch log streams in 2.9.1 + slurm

Created on 8 Oct 2020  路  9Comments  路  Source: aws/aws-parallelcluster

If you are reporting an issue with AWS Parallelcluster / CfnCluster please make sure to add the following data in order to facilitate the root cause detection:

Required Info:

  • AWS ParallelCluster version [e.g. 2.9.0]: 2.9.1
  • Full cluster configuration without any credentials or personal data
[aws]
aws_region_name = us-west-2

[aliases]
ssh = ssh {CFN_USER}@{MASTER_IP} {ARGS}

[global]
cluster_template = default
update_check = true
sanity_check = true

[cluster default]
key_name = ec2-admin
base_os = centos7
scheduler = slurm
master_instance_type = m5.large
vpc_settings = default
queue_settings = compute
tags = {"purpose": "development-ondemand"}
custom_ami = ami-0cbdde0662f037f7c
master_root_volume_size = 100
compute_root_volume_size = 100

[vpc default]
vpc_id = vpc-fa52d99e
master_subnet_id = subnet-00bbd054b223b7501
vpc_security_group_id = sg-74e42c12
use_public_ips = false

[queue compute]
enable_efa = false
compute_resource_settings = default

[compute_resource default]
instance_type = m5.xlarge
  • Cluster name: slurm-test

Bug description and how to reproduce:
Maybe I'm missing something obvious, but from the sound of the docs, it seems like by default with no configuration, this should be streaming logs to cloudwatch - however, in my case, the log group for the cluster gets created, but no log streams ever show up. Am I just missing some configuration or permissions or something somewhere?

help wanted
Source
picture keien

Most helpful comment

Okay, thanks for pointing out where the issue is. We do have some of our own setup during AMI creation for CloudWatch agent so it's probably conflicting with that somehow. I'll look into it but it doesn't seem like it's an issue with ParallelCluster so I will close this, thanks.

picture keien on 30 Oct 2020
👍2

All 9 comments

from the sound of the docs, it seems like by default with no configuration, this should be streaming logs to cloudwatch

That's correct. CloudWatch logging is enabled by default. The provided config doesn't disable the feature.

the log group for the cluster gets created, but no log streams ever show up

To help us troubleshoot this further can you provide the following logs (with any sensitive data removed)?

  • /var/log/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.log
  • /var/log/amazon/amazon-cloudwatch-agent/configuration-validation.log
  • /var/log/cfn-init.log
tilne picture tilne on 9 Oct 2020 
[root@ip-172-31-48-40 ~]# cat /var/log/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.log
2020/09/25 22:27:33 I! I! Detected the instance is EC2
2020/09/25 22:27:33 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json does not exist or cannot read. Skipping it.
2020/09/25 22:27:33 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/default ...
Valid Json input schema.
I! Detecting runasuser...
No csm configuration found.
No log configuration found.
Configuration validation first phase succeeded

2020/09/25 22:27:33 I! Config has been translated into TOML /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
2020/09/25 22:27:33 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
2020/09/25 22:27:33 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/default ...
2020/09/25 22:27:33 I! Detected runAsUser: cwagent
2020/09/25 22:27:33 I! Change ownership to cwagent:cwagent
2020/09/25 22:27:33 I! Set HOME: /home/cwagent
2020-09-25T22:27:33Z I! Starting AmazonCloudWatchAgent 1.247345.34
2020-09-25T22:27:33Z I! Loaded inputs: disk mem
2020-09-25T22:27:33Z I! Loaded aggregators:
2020-09-25T22:27:33Z I! Loaded processors: ec2tagger
2020-09-25T22:27:33Z I! Loaded outputs: cloudwatch
2020-09-25T22:27:33Z I! Tags enabled: host=ip-172-31-59-89.us-west-2.compute.internalcerebras.aws     
2020-09-25T22:27:33Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"ip-172-31-59-89.us-west-2.compute.internalcerebras.aws", Flush Interval:1s
2020-09-25T22:27:33Z I! [logagent] starting
2020-09-25T22:27:33Z I! [processors.ec2tagger] ec2tagger: EC2 tagger has started initialization.      
2020-09-25T22:27:33Z I! cloudwatch: get unique roll up list []
2020-09-25T22:27:33Z I! cloudwatch: publish with ForceFlushInterval: 1m0s, Publish Jitter: 37s        
2020-09-25T22:27:33Z W! [processors.ec2tagger] ec2tagger: Unable to describe ec2 tags for initial retrieval: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request
        status code: 404, request id:
caused by: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
                 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
2020-09-25T22:28:33Z I! [processors.ec2tagger] ec2tagger: 1 retry for initial retrieval of tags and volumes
2020-09-25T22:28:33Z W! [processors.ec2tagger] ec2tagger: Unable to describe ec2 tags for initial retrieval: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request
        status code: 404, request id:
caused by: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
                 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
2020-09-25T22:29:33Z I! [processors.ec2tagger] ec2tagger: 2 retry for initial retrieval of tags and volumes
2020-09-25T22:29:34Z W! [processors.ec2tagger] ec2tagger: Unable to describe ec2 tags for initial retrieval: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request
        status code: 404, request id:
caused by: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
                 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
2020-09-25T22:32:34Z I! [processors.ec2tagger] ec2tagger: 3 retry for initial retrieval of tags and volumes
2020-09-25T22:32:34Z W! [processors.ec2tagger] ec2tagger: Unable to describe ec2 tags for initial retrieval: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request
        status code: 404, request id:
caused by: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
                 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
2020-09-25T22:35:34Z I! [processors.ec2tagger] ec2tagger: 4 retry for initial retrieval of tags and volumes
2020-09-25T22:35:34Z W! [processors.ec2tagger] ec2tagger: Unable to describe ec2 tags for initial retrieval: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request
        status code: 404, request id:
caused by: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
                 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
2020-09-25T22:38:34Z I! [processors.ec2tagger] ec2tagger: 5 retry for initial retrieval of tags and volumes
2020-09-25T22:38:34Z W! [processors.ec2tagger] ec2tagger: Unable to describe ec2 tags for initial retrieval: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request
        status code: 404, request id:
caused by: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
                 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
2020-09-25T22:48:34Z I! [processors.ec2tagger] ec2tagger: 6 retry for initial retrieval of tags and volumes
2020-09-25T22:48:34Z W! [processors.ec2tagger] ec2tagger: Unable to describe ec2 tags for initial retrieval: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request
        status code: 404, request id:
caused by: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
                 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
2020-09-25T22:58:34Z I! [processors.ec2tagger] ec2tagger: 7 retry for initial retrieval of tags and volumes
2020-09-25T22:58:34Z W! [processors.ec2tagger] ec2tagger: Unable to describe ec2 tags for initial retrieval: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request
        status code: 404, request id:
caused by: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
                 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
2020-09-25T23:08:34Z I! [processors.ec2tagger] ec2tagger: 8 retry for initial retrieval of tags and volumes
2020-09-25T23:08:34Z W! [processors.ec2tagger] ec2tagger: Unable to describe ec2 tags for initial retrieval: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request
        status code: 404, request id:
caused by: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
                 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
2020-09-25T23:18:34Z I! [processors.ec2tagger] ec2tagger: 9 retry for initial retrieval of tags and volumes
2020-09-25T23:18:35Z W! [processors.ec2tagger] ec2tagger: Unable to describe ec2 tags for initial retrieval: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request
        status code: 404, request id:
caused by: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
                 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
2020-09-25T23:28:35Z I! [processors.ec2tagger] ec2tagger: 10 retry for initial retrieval of tags and volumes
2020-09-25T23:28:35Z W! [processors.ec2tagger] ec2tagger: Unable to describe ec2 tags for initial retrieval: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request
        status code: 404, request id:
caused by: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
                 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
2020-09-25T23:38:35Z I! [processors.ec2tagger] ec2tagger: 11 retry for initial retrieval of tags and volumes
2020-09-25T23:38:35Z W! [processors.ec2tagger] ec2tagger: Unable to describe ec2 tags for initial retrieval: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request
        status code: 404, request id: 
caused by: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
                 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
2020-09-25T23:43:28Z I! Profiler is stopped during shutdown
2020-09-25T23:43:28Z I! [agent] Hang on, flushing any cached metrics before shutdown
2020/09/25 16:46:44 I! I! Detected the instance is EC2
2020/09/25 16:46:44 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json does not exist or cannot read. Skipping it.
2020/09/25 16:46:44 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/default ...
Valid Json input schema.
I! Detecting runasuser...
No csm configuration found.
No log configuration found.
Configuration validation first phase succeeded

2020/09/25 16:46:44 I! Config has been translated into TOML /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
2020/09/25 16:46:44 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
2020/09/25 16:46:44 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/default ...
2020/09/25 16:46:44 I! Detected runAsUser: cwagent
2020/09/25 16:46:44 I! Change ownership to cwagent:cwagent
2020/09/25 16:46:44 I! Set HOME: /home/cwagent
2020-09-25T23:46:44Z I! Starting AmazonCloudWatchAgent 1.247345.34
2020-09-25T23:46:44Z I! Loaded inputs: disk mem
2020-09-25T23:46:44Z I! Loaded aggregators:
2020-09-25T23:46:44Z I! Loaded processors: ec2tagger
2020-09-25T23:46:44Z I! Loaded outputs: cloudwatch
2020-09-25T23:46:44Z I! Tags enabled: host=ip-172-31-59-89
2020-09-25T23:46:44Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"ip-172-31-59-89", Flush 
Interval:1s
2020-09-25T23:46:44Z I! [logagent] starting
2020-09-25T23:46:44Z I! [processors.ec2tagger] ec2tagger: EC2 tagger has started initialization.      
2020-09-25T23:46:44Z I! cloudwatch: get unique roll up list []
2020-09-25T23:46:44Z I! cloudwatch: publish with ForceFlushInterval: 1m0s, Publish Jitter: 37s        
2020-09-25T23:46:44Z W! [processors.ec2tagger] ec2tagger: Unable to describe ec2 tags for initial retrieval: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request
        status code: 404, request id:
caused by: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
                 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
2020-09-25T23:47:44Z I! [processors.ec2tagger] ec2tagger: 1 retry for initial retrieval of tags and volumes
2020-09-25T23:47:44Z W! [processors.ec2tagger] ec2tagger: Unable to describe ec2 tags for initial retrieval: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request
        status code: 404, request id:
caused by: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
                 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
2020-09-25T23:47:48Z I! Profiler is stopped during shutdown
2020-09-25T23:47:48Z I! [agent] Hang on, flushing any cached metrics before shutdown
2020/10/09 14:45:05 I! I! Detected the instance is EC2
2020/10/09 14:45:05 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json does not exist or cannot read. Skipping it.
2020/10/09 14:45:05 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/default ...
Valid Json input schema.
I! Detecting runasuser...
No csm configuration found.
No log configuration found.
Configuration validation first phase succeeded

2020/10/09 14:45:06 I! Config has been translated into TOML /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
2020/10/09 14:45:06 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
2020/10/09 14:45:06 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/default ...
2020/10/09 14:45:06 I! Detected runAsUser: cwagent
2020/10/09 14:45:06 I! Change ownership to cwagent:cwagent
2020/10/09 14:45:06 I! Set HOME: /home/cwagent
2020-10-09T21:45:18Z I! Starting AmazonCloudWatchAgent 1.247345.34
2020-10-09T21:45:18Z I! Loaded inputs: disk mem
2020-10-09T21:45:18Z I! Loaded aggregators:
2020-10-09T21:45:18Z I! Loaded processors: ec2tagger
2020-10-09T21:45:18Z I! Loaded outputs: cloudwatch
2020-10-09T21:45:18Z I! Tags enabled: host=ip-172-31-48-40
2020-10-09T21:45:18Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"ip-172-31-48-40", Flush 
Interval:1s
2020-10-09T21:45:18Z I! [logagent] starting
2020-10-09T21:45:18Z I! [processors.ec2tagger] ec2tagger: EC2 tagger has started initialization.      
2020-10-09T21:45:18Z I! cloudwatch: get unique roll up list []
2020-10-09T21:45:18Z I! cloudwatch: publish with ForceFlushInterval: 1m0s, Publish Jitter: 37s        
2020-10-09T21:45:19Z I! [processors.ec2tagger] ec2tagger: Initial retrieval of tags succeded
2020-10-09T21:45:19Z I! [processors.ec2tagger] ec2tagger: EC2 tagger has started, finished initial retrieval of tags and Volumes

[root@ip-172-31-48-40 ~]# cat /var/log/amazon/amazon-cloudwatch-agent/configuration-validation.log
CWAgent/1.247345.34 (go1.14.7; linux; amd64) No Build Date
The given config: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml is valid

cfn-init.log attached
cfn-init.log

keien picture keien on 10 Oct 2020

From the CloudWatch agent's log, it looks like it's missing some necessary permission(s). In the cluster's main CloudFormation stack there should be a RootRole resource. Can you tell me what policies are attached to that role? Specifically, is the CloudWatchAgentServerPolicy attached?

tilne picture tilne on 10 Oct 2020

yeah I see it attached:
image

there's 3 other inline policies: parallelcluster, parallelcluster-hit, and parallelcluster-hit-route53

acccess advisor says the inline ones have been used but CloudWatchAgentServerPolicy hasn't been accessed at all

keien picture keien on 10 Oct 2020

Hi @keien,

Are you still experiencing the issue? If so, could you try creating cluster with the custom AMI to see if the error in cloudwatch agent logs persists?

Thank you!

rexcsn picture rexcsn on 16 Oct 2020

Yes I am - I'm using a custom AMI built on top of your current 2.9.1 Centos 7 AMI (ami-0bbb41e6db63dee98)

keien picture keien on 17 Oct 2020

Hi Keien,

Networking

I see you're using a single subnet for both head node and computes but you're setting use_public_ips = false.
Is the subnet-00bbd054b223b7501 a public one (connected to an Internet Gateway) or a private one (NAT Gateway)?

If it's a public subnet the use_public_ips setting cannot be set to false, because the internet gateway requires that all instances have a globally unique IP address. See networking documentation for more info.

CloudWatch agent startup

In the cfn-init file I can see:

  * execute[cloudwatch-agent-start] action run  "status": "running",
 (skipped due to not_if)

It means that the "start" of the CloudWatch agent service has been skipped because already running.
You can verify the status of the service by executing:

/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status

AMI customization

You said:

I'm using a custom AMI built on top of your current 2.9.1 Centos 7 AMI (ami-0bbb41e6db63dee98)

Are you following the Modify an AWS ParallelCluster AMI approach to create the AMI?
We suggest to use the pcluster createami one.

In any case I think @rexcsn meant "try creating cluster _without_ the custom AMI" to see if the error is related to your AMI customization. If you're doing only minor customization you could use custom bootstrap actions instead of creating a custom AMI.

Let us know if it helps

Thanks
Enrico

enrico-usai picture enrico-usai on 22 Oct 2020

I tried to replicate your use case by creating a custom AMI starting from Centos7 ParallelCluster base AMI, eu-west-1: ami-0b2c1929139662d8f but in my case I have the log stream and the log files are there.

The difference is that in the cfn-init.log of the created cluster the agent has been started correctly:

  * execute[cloudwatch-agent-start] action run
    [execute] /opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
              Successfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_amazon-cloudwatch-agent.json.tmp
              Start configuration validation...

I think something went wrong on your AMI creation process.

enrico-usai picture enrico-usai on 22 Oct 2020

Okay, thanks for pointing out where the issue is. We do have some of our own setup during AMI creation for CloudWatch agent so it's probably conflicting with that somehow. I'll look into it but it doesn't seem like it's an issue with ParallelCluster so I will close this, thanks.

keien picture keien on 30 Oct 2020
👍2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

mukulagrawal78 picture mukulagrawal78  路  5Comments
huffmabc picture huffmabc  路  3Comments
anders-lgi picture anders-lgi  路  3Comments
rems0 picture rems0  路  4Comments
calebwin picture calebwin  路  4Comments