Kapacitor: Memory leak

[phemmer]

We are experiencing a memory leak with kapacitor 1.4.0 on CentOS/7. Upon adding a new script to the kapacitor service, memory growth started increasing unbounded, until eventually the Linux OOM killer swoops in and kills it.

The contents of the script triggering the issue are:

var data = stream|from().measurement('jolokia').groupBy('host','application')
var gc_interval = data|difference('gc_ConcurrentMarkSweep_startTime')
var data_join = data|join(gc_interval).as('data','gc_interval')
  |where(lambda: "gc_interval.difference" > 0)
  |eval(lambda: "data.heap_memory_usage_used", lambda: "data.heap_memory_usage_max", lambda: "gc_interval.difference")
    .as('heap_memory_used','heap_memory_max','gc_interval')
  |eval(lambda: float("heap_memory_used") / float("heap_memory_max") * 100.0)
    .as('heap_memory_used_pct')
    .keep()

var data_join_jqFilter = 'del(.details)|.data.series=(.data.series[]|.fields=(.columns as $columns|.values[]|[$columns,.]|transpose|map({"key": .[0], "value": .[1]})|from_entries)|del(.columns)|del(.values))'

var data_join_msg = '{{index .Tags "host"}} jvm_heap_used {{index .Tags "application"}} = {{index .Fields "heap_memory_used_pct" | printf "%.02f"}}% - {{.Level}}'

var data_join_details = '{{.|printf "%+v"}}'

data_join
  |alert()
    .message(data_join_msg)
    .details(data_join_details)
    .stateChangesOnly(15m)
    .warn(lambda: ("heap_memory_used_pct" > 80) AND ! ("heap_memory_used_pct" > 90))
    .exec('bash','-o','pipefail','-c','/usr/bin/jq -c "$1" | /usr/bin/logger -p daemon.notice -t kapacitord/alert/warn','--',data_join_jqFilter)
    .pagerDuty().serviceKey('XXX')

data_join
  |alert()
    .message(data_join_msg)
    .details(data_join_details)
    .stateChangesOnly(15m)
    .crit(lambda: "heap_memory_used_pct" > 90)
    .exec('bash','-o','pipefail','-c','/usr/bin/jq -c "$1" | /usr/bin/logger -p daemon.notice -t kapacitord/alert/crit','--',data_join_jqFilter)
    .pagerDuty().serviceKey('YYY')

This isn't an issue with cardinality and kapacitor creating a large number of groups, as the cardinality is limited:

# influx -database telegraf -execute 'show series' | grep '^jolokia,' | wc -l
38

Here's the top10 on a heap dump:

(pprof) top10
1973.08MB of 1994.16MB total (98.94%)
Dropped 715 nodes (cum <= 9.97MB)
Showing top 10 nodes out of 55 (cum >= 101.90MB)
      flat  flat%   sum%        cum   cum%
 1525.64MB 76.51% 76.51%  1525.64MB 76.51%  github.com/influxdata/kapacitor/vendor/github.com/influxdata/influxdb/models.(*point).unmarshalBinary
  137.53MB  6.90% 83.40%   137.53MB  6.90%  github.com/influxdata/kapacitor/vendor/github.com/influxdata/influxdb/models.Tags.Map
  110.44MB  5.54% 88.94%   110.94MB  5.56%  github.com/influxdata/kapacitor.(*joinGroup).Collect
   98.38MB  4.93% 93.87%    98.38MB  4.93%  github.com/influxdata/kapacitor/alert.newHandler
   54.51MB  2.73% 96.61%    54.51MB  2.73%  github.com/influxdata/kapacitor/edge.(*pointMessage).ShallowCopy
   20.50MB  1.03% 97.63%    20.50MB  1.03%  net/textproto.(*Reader).ReadLine
   14.57MB  0.73% 98.37%    14.57MB  0.73%  github.com/influxdata/kapacitor/tick/stateful.NewFunctions
      11MB  0.55% 98.92%       11MB  0.55%  github.com/influxdata/kapacitor/models.ToGroupID
    0.52MB 0.026% 98.94%    16.58MB  0.83%  github.com/influxdata/kapacitor/edge.(*groupedConsumer).getOrCreateGroup
         0     0% 98.94%   101.90MB  5.11%  github.com/influxdata/kapacitor.(*AlertNode).(github.com/influxdata/kapacitor.runAlert)-fm

(pprof) list unmarshalBinary
Total: 1.95GB
ROUTINE ======================== github.com/influxdata/kapacitor/vendor/github.com/influxdata/influxdb/models.(*point).unmarshalBinary in /root/go/src/github.com/influxdata/kapacitor/vendor/github.com/influxdata/influxdb/models/points.go
    1.49GB     1.49GB (flat, cum) 76.51% of Total
         .          .   1483:       p.time.Round(d).UnixNano())
         .          .   1484:}
         .          .   1485:
         .          .   1486:func (p *point) unmarshalBinary() Fields {
         .          .   1487:   iter := p.FieldIterator()
   11.50MB    11.50MB   1488:   fields := make(Fields, 8)
         .          .   1489:   for iter.Next() {
         .          .   1490:       if len(iter.FieldKey()) == 0 {
         .          .   1491:           continue
         .          .   1492:       }
         .          .   1493:       switch iter.Type() {
         .          .   1494:       case Float:
    1.48GB     1.48GB   1495:           fields[string(iter.FieldKey())] = iter.FloatValue()
         .          .   1496:       case Integer:
  512.01kB   512.01kB   1497:           fields[string(iter.FieldKey())] = iter.IntegerValue()
         .          .   1498:       case String:
         .          .   1499:           fields[string(iter.FieldKey())] = iter.StringValue()
         .          .   1500:       case Boolean:
         .          .   1501:           fields[string(iter.FieldKey())] = iter.BooleanValue()
         .          .   1502:       }
         .          .   1503:   }
         .          .   1504:   return fields
         .          .   1505:}
         .          .   1506:

[bayendor]

We can report similar behavior since upgrading to Kapacitor 1.4.0 three weeks ago.

[nathanielc]

@phemmer I am looking into this. Can you provide the entire heap dump if you are still experiencing this issue? Also is the top10 profile you show in alloc_space or insuse_space?

[bayendor]

Quick update here: After review of supporting logs, this behavior was caused by a single TICK script referencing an non-existent retention policy. When the TICK script was corrected, the behavior resolved.

[nathanielc]

Thanks for the update. Closing for now. If the issue resurfaces we can investigate further.

[shubham7saxena]

@bayendor we are also experiencing this issue. can you please describe in a little more detail what was wrong, how you identified and corrected it?