Aws-parallelcluster: ganglia doesn't plot anything

[adavanisanti]

I don't see any plots on ganglia even after several cluster runs. I am using cfncluster 1.3.2 and SGE for job scheduling. Below is a sample snapshot.

Is there something that I need to do to see the plots?

[mohanasudhan]

what does ganglia logs and messages say? e.g /var/log/messages

[adavanisanti]

sinfo:

ubuntu@ip-172-31-24-108:/shared/data$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
compute*     up   infinite      7   idle ip-172-31-17-239,ip-172-31-20-169,ip-172-31-24-200,ip-172-31-26-240,ip-172-31-28-231,ip-172-31-30-63,ip-172-31-31-233

Below are some error messages and GET requests from ganglia frontend:

/var/log/apache2/error.log:ERROR: opening '/var/lib/ganglia/rrds/cfncluster-sample-cluster-1/__SummaryInfo__/cpu_num.rrd': No such file or directory
/var/log/apache2/error.log:ERROR: opening '/var/lib/ganglia/rrds/cfncluster-sample-cluster-1/__SummaryInfo__/load_one.rrd': No such file or directory
/var/log/apache2/error.log:ERROR: opening '/var/lib/ganglia/rrds/cfncluster-sample-cluster-1/__SummaryInfo__/cpu_num.rrd': No such file or directory
/var/log/apache2/error.log:ERROR: opening '/var/lib/ganglia/rrds/cfncluster-sample-cluster-1/__SummaryInfo__/load_one.rrd': No such file or directory
/var/log/apache2/error.log:ERROR: opening '/var/lib/ganglia/rrds/cfncluster-sample-cluster-1/__SummaryInfo__/cpu_num.rrd': No such file or directory
/var/log/apache2/error.log:ERROR: opening '/var/lib/ganglia/rrds/cfncluster-sample-cluster-1/__SummaryInfo__/load_one.rrd': No such file or directory
/var/log/apache2/error.log:ERROR: opening '/var/lib/ganglia/rrds/cfncluster-sample-cluster-1/__SummaryInfo__/cpu_num.rrd': No such file or directory
/var/log/apache2/error.log:ERROR: opening '/var/lib/ganglia/rrds/cfncluster-sample-cluster-1/__SummaryInfo__/load_one.rrd': No such file or directory
/var/log/apache2/error.log:ERROR: opening '/var/lib/ganglia/rrds/cfncluster-sample-cluster-1/__SummaryInfo__/cpu_num.rrd': No such file or directory
/var/log/apache2/error.log:ERROR: opening '/var/lib/ganglia/rrds/cfncluster-sample-cluster-1/__SummaryInfo__/load_one.rrd': No such file or directory
/var/log/apache2/error.log:ERROR: opening '/var/lib/ganglia/rrds/cfncluster-sample-cluster-1/__SummaryInfo__/cpu_num.rrd': No such file or directory
/var/log/apache2/error.log:ERROR: opening '/var/lib/ganglia/rrds/cfncluster-sample-cluster-1/__SummaryInfo__/load_one.rrd': No such file or directory
/var/log/apache2/error.log:ERROR: opening '/var/lib/ganglia/rrds/cfncluster-sample-cluster-1/__SummaryInfo__/cpu_num.rrd': No such file or directory
/var/log/apache2/error.log:ERROR: opening '/var/lib/ganglia/rrds/cfncluster-sample-cluster-1/__SummaryInfo__/load_one.rrd': No such file or directory
/var/log/apache2/access.log:c-67-189-99-246.hsd1.or.comcast.net - - [25/Sep/2017:20:35:12 +0000] "GET /ganglia/ HTTP/1.1" 200 9076 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"

Syslog messages:

/var/log/syslog:Sep 20 06:35:28 ip-172-31-12-239 systemd[1]: Stopping ganglia-monitor.service...
/var/log/syslog:Sep 21 00:55:16 ip-172-31-12-239 systemd[1]: Starting ganglia-monitor.service...
/var/log/syslog:Sep 21 00:55:16 ip-172-31-12-239 ganglia-monitor[1280]: Starting Ganglia Monitor Daemon: gmond.
/var/log/syslog:Sep 21 00:55:16 ip-172-31-12-239 systemd[1]: Started ganglia-monitor.service.
/var/log/syslog:Sep 21 19:51:28 ip-172-31-12-239 systemd[1]: Starting ganglia-monitor.service...
/var/log/syslog:Sep 21 19:51:28 ip-172-31-12-239 ganglia-monitor[1258]: Starting Ganglia Monitor Daemon: gmond.
/var/log/syslog:Sep 21 19:51:28 ip-172-31-12-239 systemd[1]: Started ganglia-monitor.service.
/var/log/syslog:Sep 21 19:59:54 ip-172-31-12-239 systemd[1]: Stopping ganglia-monitor.service...
/var/log/syslog:Sep 21 19:59:54 ip-172-31-12-239 ganglia-monitor[1913]: Stopping Ganglia Monitor Daemon: gmond.
/var/log/syslog:Sep 25 20:23:23 ip-172-31-12-239 systemd[1]: Starting ganglia-monitor.service...
/var/log/syslog:Sep 25 20:23:25 ip-172-31-12-239 ganglia-monitor[1327]: Starting Ganglia Monitor Daemon: gmond.
/var/log/syslog:Sep 25 20:23:25 ip-172-31-12-239 systemd[1]: Started ganglia-monitor.service.
/var/log/syslog.1:Sep 20 00:09:06 ip-172-31-12-239 systemd[1]: Starting ganglia-monitor.service...
/var/log/syslog.1:Sep 20 00:09:06 ip-172-31-12-239 ganglia-monitor[1303]: Starting Ganglia Monitor Daemon: gmond.
/var/log/syslog.1:Sep 20 00:09:06 ip-172-31-12-239 systemd[1]: Started ganglia-monitor.service.
/var/log/syslog.1:Sep 20 05:05:21 ip-172-31-12-239 systemd[1]: Starting ganglia-monitor.service...
/var/log/syslog.1:Sep 20 05:05:21 ip-172-31-12-239 ganglia-monitor[1250]: Starting Ganglia Monitor Daemon: gmond.
/var/log/syslog.1:Sep 20 05:05:21 ip-172-31-12-239 systemd[1]: Started ganglia-monitor.service.

[rajachan]

Did you make changes to the default /etc/ganglia/gmond.conf or /etc/ganglia/gmetad.conf? Is the gmetad service running (could not tell from your syslog)? I just set up a sample cluster with v1.3.2 and I see metrics on the Ganglia portal. Although, I am using the default AMI. What OS are you running with?

[adavanisanti]

_Did you make changes to the default /etc/ganglia/gmond.conf or /etc/ganglia/gmetad.conf?_
No

_Is the gmetad service running (could not tell from your syslog)?_
Yes

ubuntu@ip-172-31-19-213:~$ service gmetad status
● gmetad.service
   Loaded: loaded (/etc/init.d/gmetad; bad; vendor preset: enabled)
   Active: active (running) since Mon 2017-10-09 18:19:39 UTC; 2min 29s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 6874 ExecStop=/etc/init.d/gmetad stop (code=exited, status=0/SUCCESS)
  Process: 6879 ExecStart=/etc/init.d/gmetad start (code=exited, status=0/SUCCESS)
    Tasks: 9
   Memory: 1.1M
      CPU: 15ms
   CGroup: /system.slice/gmetad.service
           └─6883 /usr/sbin/gmetad --pid-file /var/run/gmetad.pid

Oct 09 18:19:39 ip-172-31-19-213 systemd[1]: Starting gmetad.service...
Oct 09 18:19:39 ip-172-31-19-213 gmetad[6879]: Starting Ganglia Monitor Meta-Daemon: gmetad.
Oct 09 18:19:39 ip-172-31-19-213 systemd[1]: Started gmetad.service.

_Although, I am using the default AMI. What OS are you running with?_
I am using default ubuntu1604 AMI for this test. For my real case, I have a custom AMI built from default ubuntu1604 AMI.

[rajachan]

I am seeing an issue with Ganglia on ubuntu1604 AMIs indeed. In my case, gmond fails to launch on the compute, and is not able to push metrics up to the MasterServer. While this is not the exact behavior you are seeing, there could be a common link here. I will take a closer look.

[rajachan]

It looks like the monitoring is service is unable to find the metric plugin libraries on Ubuntu. Can you send me the output of find /usr/lib64/ganglia -name mod*.so and find /usr/lib/ganglia -name mod*.so from any one of the instances?

[ashuc]

I'm having the same issues. Those directories do not exist on my instance, but /usr/lib/ganglia does.

ubuntu@ip-xxx-xx-xx-xxx:~$ find /usr/lib64/ganglia -name mod*.so
find: ‘/usr/lib64/ganglia’: No such file or directory

ubuntu@ip-xxx-xx-xx-xxx:~$ find /usr/lib/ganglia -name mod*.so
/usr/lib/ganglia/modpython.so
/usr/lib/ganglia/modmem.so
/usr/lib/ganglia/modcpu.so
/usr/lib/ganglia/modproc.so
/usr/lib/ganglia/modload.so
/usr/lib/ganglia/modmulticpu.so
/usr/lib/ganglia/moddisk.so
/usr/lib/ganglia/modsys.so
/usr/lib/ganglia/modnet.so

[adavanisanti]

/usr/lib64 is empty and /usr/lib has some *.so files.

ubuntu@ip-172-31-19-213:~$ find /usr/lib64/ganglia -name mod*.so
find: ‘/usr/lib64/ganglia’: No such file or directory
ubuntu@ip-172-31-19-213:~$ find /usr/lib/ganglia -name mod*.so
/usr/lib/ganglia/modpython.so
/usr/lib/ganglia/modmem.so
/usr/lib/ganglia/modcpu.so
/usr/lib/ganglia/modproc.so
/usr/lib/ganglia/modload.so
/usr/lib/ganglia/modmulticpu.so
/usr/lib/ganglia/moddisk.so
/usr/lib/ganglia/modsys.so
/usr/lib/ganglia/modnet.so

[rajachan]

Thanks for that information, @ashuc and @adavanisanti. I will do some more testing with the ubuntu base image to confirm this is the actual problem. If it is, we should be able to handle this from our Chef cookbook recipe for Ganglia.

[mohanasudhan]

ubuntu@ip-172-31-39-74:~$ gmond -f
Cannot load modcpu.so metric module: modcpu.so: cannot open shared object file: No such file or directory
Cannot load moddisk.so metric module: moddisk.so: cannot open shared object file: No such file or directory
Cannot load modload.so metric module: modload.so: cannot open shared object file: No such file or directory
Cannot load modmem.so metric module: modmem.so: cannot open shared object file: No such file or directory
Cannot load modnet.so metric module: modnet.so: cannot open shared object file: No such file or directory
Cannot load modproc.so metric module: modproc.so: cannot open shared object file: No such file or directory
Cannot load modsys.so metric module: modsys.so: cannot open shared object file: No such file or directory

[mohanasudhan]

Path of mod*.so files in ubuntu '/usr/lib/ganglia/' comparing amazon linux '/usr/lib64/ganglia/'

[dougalb]

Ubuntu intentionally stopped using lib64 on OSes that are 64bit by default.
Probably need a OS specific confit in the cookbook.

On Mon, Oct 23, 2017 at 14:40 Mohan Gandhi [e-mail] wrote:

Path of mod*.so files in ubuntu '/usr/lib/ganglia/' comparing amazon linux
'/usr/lib64/ganglia/'

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/awslabs/cfncluster/issues/228#issuecomment-338804903,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACUViuUD3SZ2Dq5rrwEI95-M89rbRTaQks5svQezgaJpZM4Pf06l
.

[mohanasudhan]

Modifing path in /etc/ganglia/gmond.conf and restarting daemon sudo /etc/init.d/ganglia-monitor restart fixes the issue

modules {
  module {
    name = "core_metrics"
  }
  module {
    name = "cpu_module"
    path = "/usr/lib/ganglia/modcpu.so"
  }
  module {
    name = "disk_module"
    path = "/usr/lib/ganglia/moddisk.so"
  }
  module {
    name = "load_module"
    path = "/usr/lib/ganglia/modload.so"
  }
  module {
    name = "mem_module"
    path = "/usr/lib/ganglia/modmem.so"
  }
  module {
    name = "net_module"
    path = "/usr/lib/ganglia/modnet.so"
  }
  module {
    name = "proc_module"
    path = "/usr/lib/ganglia/modproc.so"
  }
  module {
    name = "sys_module"
    path = "/usr/lib/ganglia/modsys.so"
  }
}

[adavanisanti]

@mohanasudhan Is this change on all nodes or just the master node?

[mohanasudhan]

@adavanisanti master node

[adavanisanti]

@mohanasudhan I see that /etc/ganglia/gmond.conf in my AMI and /etc/ganglia/gmond.conf in the actual master instance are different. How do I make gmond.conf change automatically?

[mohanasudhan]

We are planning for new release of cfncluster in coming weeks which has fix https://github.com/awslabs/cfncluster-cookbook/pull/38 , if you not urgent you can wait for it. Else you can have post_install script to updating new gmond.conf (which can be loaded updated config from your s3) and restart ganglia monitor.

[ashuc]

Actually, it turns out you need to modify /etc/ganglia/gmond.conf and run sudo /etc/init.d/ganglia-monitor restart on every node, not just the master node. I just verified this on a cluster I spun up.

Please let us know when you have the fix in place within the next release. Thanks!