Kapacitor: JoinNode ignores Delete BarrierNode messages.

[m4ce]

After some testing, I found out that the JoinNode cardinality doesn't decrease when a BarrierMessage is emitted for a group that should expire. This effectively leads the JoinNode's cardinality to increase forever, leading to a memory leak.

[docmerlin]

Are you using the .delete(TRUE) method on the barrier node? If you are not, it will not decrease the cardinality.

[m4ce]

@docmerlin, I am using the .delete(TRUE). However, the join() cardinality keeps increasing nonetheless

[m4ce]

From my testing, it looks like join() is ignoring barrier messages?

[docmerlin]

@m4ce
Can you post here your script?

[m4ce]

var data1 = stream
    |from()
        .database(db)
        .retentionPolicy(rp)
        .measurement('kubernetes_pod_container')
        .where(lambda: isPresent("cpu_usage_nanocores"))
        .groupBy(groupBy)
    |barrier()
        .idle(5m)
        .delete(TRUE)
    |window()
        .period(1m)
        .every(1m)
        .align()
    |mean('cpu_usage_nanocores')
        .as('avgCpuUsageNanoCores')
    // Convert from nanocores to millicores.
    |eval(lambda: float("avgCpuUsageNanoCores") / 1000000.0)
        .as('avgCpuUsageMilliCores')

var data2 = stream
    |from()
        .database(db)
        .retentionPolicy(rp)
        .measurement('kubernetes_pod_container_status')
        .groupBy(groupBy)
    |default()
        // default to one core
        .field('resource_limits_millicpu_units', oneCpu)
    |barrier()
        .idle(5m)
        .delete(TRUE)
    |window()
        .period(1m)
        .every(1m)
        .align()
    |last('resource_limits_millicpu_units')
        .as('milliCores')
    // When no cpus limits are defined, set the default to 1 CPU instead of defaulting to the number of cores available on the box.
    // We want to catch run-away processes that do not have any limits set.
    |eval(lambda: if("milliCores" > 0, float("milliCores"), float(oneCpu)))
        .as('maxUsageMilliCores')

var data = data1
    |join(data2)
        .as('stats', 'limits')
        .tolerance(15s)
    |barrier()
        .idle(5m)
        .delete(TRUE)
    |eval(lambda: "stats.avgCpuUsageMilliCores" / "limits.maxUsageMilliCores" * 100.0)
        .as('usage_percent')
    |stateCount(lambda: "usage_percent" > crit)
        .as('critCount')
    |stateCount(lambda: "usage_percent" <= crit)
        .as('critResetCount')
    |stateCount(lambda: "usage_percent" > warn AND "usage_percent" <= crit)
        .as('warnCount')
    |stateCount(lambda: "usage_percent" <= warn)
        .as('warnResetCount')

[m4ce]

I even had to add a barrier after join() in order to reduce the cardinality on the other nodes. I can see the cardinality decreasing before and after join. But the join itself keeps increasing!

[docmerlin]

Ok, I've been able to confirm that join node is ignoring barrier node delete messages.

[docmerlin]

The NewMultiConsumerWithStats isn't handling the DeleteGroupMessage.
Ordinary Barrier messages are being handled properly but the DeleteGroupMessage are not.

[docmerlin]

A tenative plan is to see if it is easier to add DeleteGroupMessage handling to MultiConsumerWithStats or to create a NewMultiConsumerWithStatsAndDelete that can handle the DeleteGroupMessage.

[docmerlin]

Tentatively adding this to the 1.5.8 milestone, because OOMS.

[m4ce]

@docmerlin, we just had another OOM due to this. Do you think you guys will be able to get it in for this or the next release? Thanks.

[joshdurbin]

I'm also impacted by this. It looks like the 1.5.8 milestone was removed from this, any idea when, if not 1.5.8 this might be resolved?

[vj-gsr]

@docmerlin, we are experiencing frequent OOM due to this. Can you pls let us know the next milestone this will be added to?

[docmerlin]

This definitely will be in 1.5.9. Sorry, but this wasn't quite done and we
needed 1.5.8 out to fix problems for other customers.

On Tue, Feb 2, 2021 at 6:22 AM VJ [e-mail] wrote:

@docmerlin https://github.com/docmerlin, we are experiencing frequent
OOM due to this. Can you pls let us know the next milestone this will be
added to?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/influxdata/kapacitor/issues/2436#issuecomment-771598627,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJN6W7CQGX4DQMUUQBZS33S47VBRANCNFSM4UGBF7GQ
.

[m4ce]

Looks like it's not planned for 1.5.9? Will it manage to make it in 1.6.0..?

[docmerlin]

@m4ce We found a honest to goodness old fashioned memory leak in the join node so we fixed that instead. That will likely fix most people's problems.

[docmerlin]

@m4ce We found a honest to goodness old fashioned memory leak in the join node so we fixed that instead. That will likely fix most people's problems.

[m4ce]

But didn't you say that the join node ignores barrier delete node messages? How is that going to fix this then?

[docmerlin]

@m4ce said:

But didn't you say that the join node ignores barrier delete node messages? How is that going to fix this then?

The old join node kept a copy of every point that passed through it because of a memory leak, even points it was done joining. The fix removes the leak so memory usage should be vastly improved.

[joshdurbin]

Any idea of when this will be merged and released? Is this issue fixed by:

• PR 2474?
• PR 2489?

[m4ce]

@docmerlin - Okay, let's see how the new release performs before closing this issue.