Cacti: Not possible to use single data source for multiple graphs

[gdsotirov]

Describe the bug

As I reported also in the forums (see topic Multiple graphs from a single data source), I'm trying to use single data source for multiple graphs in Cacti 1.2.11, because data could be gathered at once, but could have different presentations (i.e. graphs).

To Reproduce

To summarize what I've done is just the following:

1. Wrote a script that returns field:value pairs (for example for memory on a Linux host - see linux_mem.pl).

2. Defined a Data Input Method for the script with Input Type "Script/Command" and all the output fields.

3. Defined a Data Source Template using the Data Input Method with all the Data Source Items for all the output fields (see cacti_data_template_linux_-_memory.xml).

4. Defined two Graph templates using the Data Source Items (see cacti_graph_template_linux_-_memory_usage_2.xml and cacti_graph_template_linux_-_swap_usage_2.xml).

5. Created two new graphs from "Create Graphs for this Device" one by one.

6. As a result two data sources with the same information were created for the two new graphs.

Templates: templates.zip

Expected behavior

The first data source to be reused, because it already provides all the Data Source Items that the second graph needs.

Desktop

• OS: Slackware Linux 14.2

• Browser: Firefox

• Version: 75.0

Additional context

The problem seems similar to what's reported in issues #948, #2246 and #2364, but as far as I understand it wasn't resolved completely. As it was commented in the forum topic it could be a regression.

[netniV]

There has been a constant battle between those graphs that should share data sources and those that should not share them. I think @TheWitness would have a better idea that me on this one.

[gdsotirov]

Let me share my personal perspective on this.

I believe that information, which could be gathered at once should be gathered and stored at once. I see no point of having multiple calls to the same data input method to get different data values when this data input method could provide all the data at once. Like with the example I've given with memory on a Linux host. All information is retrieved from /proc/meminfo and additional could be calculated from these values with a script like below from my example.

#!/usr/bin/env perl

delete @ENV{qw(PATH)};
$ENV{PATH} = '/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/sbin';

@prints = ('MemTotal', 'MemUsed', 'MemFree', 'Cached', 'Buffers',
           'SwapTotal', 'SwapUsed', 'SwapFree', 'SwapCached');

$val1 = $ARGV[0];
($val1) = $val1 =~ /^([\w.:]+)$/;
my %data;

open(PROCESS, "/proc/meminfo");
while (<PROCESS>)
{
  if ($_ =~ /^([a-zA-Z]+)\:\s*([0-9]+)/) {
    $data{$1} = $2;
  }
}
close(PROCESS);

$data{'MemUsed'} = $data{'MemTotal'} - ($data{'MemFree'} + $data{'Cached'} + $data{'Buffers'});
$data{'SwapUsed'} = $data{'SwapTotal'} - ($data{'SwapCached'} + $data{'SwapFree'});

foreach $item (@prints) {
  printf("%s:%d ", $item, $data{$item})
}

This gathers all the information necessary for two graphs on physical memory and swap, which could be stored in a single RRD archive. Such information would properly represent memory status at single point in time when /proc/meminfo was read. If having more than one data source the gathered values could differ, because it matters when exactly data collection was done. This is even reflected on the graphs as sometimes the calculations doesn't match, so the sum of memory fragments is more than total memory. Take as example the two graphs below.

Note the values going off the total physical memory circled in red.

The second graph for the same time period produced from a single data source filled by the script above doesn't have this problem.

I'm sure there is rationale in having each graph produced from it's own data source, but I also believe there is point in linking more than one graph to a single data source (i.e. same data could have different presentations). I do not ask this to be the default, but let it be an option. I do not remember clearly, but I think such option existed in the past.

[netniV]

I am sure it did too and when duplicating a graph became a problem. At the moment I’m not recalling all the ins and outs but in essence we would agree with your above statements, it’s sometimes how it can work with the code we have.

I still get confusion from the poller at times and I’ve been doing development on this for years :) My colleagues are likely to have a better understanding so I’ll leave them to respond with the ins and outs etc

[gdsotirov]

I still get confusion from the poller at times and I’ve been doing development on this for years :)

I'm just a user, but I've seen in debug log how some values for one graph could be gathered at the start of the pooling process for the host while some other at the end (which may differ by few seconds). I think it depends on when the data source items were added and my instance is 15 years old, so I do not even remember when and why I added or changed some things :-) In this regards, perhaps an audit log would be quite useful.

[frontierliu]

The poller is walking through the poller_cache_table one by one and sample eveay datasource , dispite it is the same or not, and the same datasources belong to different rrd_files. it is squandered.

I think a improvement should be made , when building the poller_cache_table every time, it should find the same record and merge it, Just one record and one file.

[netniV]

It's an American holiday today so maybe next week @TheWitness or @cigamit could offer their view on this. It's been unverified for a few months now so we either need to close it or understand the issue/direction we will take.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[TheWitness]

At some point, we can revisit this one.