Substrate: Memory leak

[iammelea]

Hello, validating in Poc-4 Alexander and we can see how the use of memory increases over time. here pic.

after stop and start the Polkadot process, the memory is back to a low level and starts increases over time again.

[laboon]

Note that apparently after running out of memory, validator nodes will get slashed as offline, so this is problematic for those running validator nodes.

[iammelea]

I have one node running without validator key and the memory is not increasing like in the validator node.

[kianenigma]

what's the upper bound of this increase? the node just keeps allocating until it crashes or it stops at some point? If the latter, then maybe this is not actually an unexpected behavior.

[iammelea]

1.-what's the upper bound of this increase?
go to check these next days.

2.- the node just keeps allocating until it crashes or it stops at some point?
So I can see in my case I think it reaches a top and stays there in 1.3GB, in my case the node has never crashed. but my server is very light of work. and in telemetrics I can see anothers nodes up to 1.3GB ussage.

[kianenigma]

1.-what's the upper bound of this increase?
go to check these next days.

2.- the node just keeps allocating until it crashes or it stops at some point?
So I can see in my case I think it reaches a top and stays there in 1.3GB, in my case the node has never crashed. but my server is very light of work. and in telemetrics I can see anothers nodes up to 1.3GB ussage.

Okay, I see. I don't suggest that it is a false alarm but it is also not super strange for a software which is _not yet developed with optimization in mind_ to allocate (gradually) around 1Gb of memory.

[iammelea]

just updating the issue
memory is increasing on time, validator or not validator node.

[bkchr]

@iammelea could you try to replicate this with latest master and a small local chain?

[arkpar]

Can't reproduce it on a local chain. The memory grows a bit, but that's mostly database cache.

[iammelea]

@arkpar .Yes grow a bit time to time. Not is a quickly grow.

If need @bkchr

I can try to run one local small testnet this weekend but if @arkpar can still keep runing one too. Can be very helpful

[bkchr]

I recently fixed some memory leak in Polkadot, maybe that solves your problem. But this was no "big" memory leak, so I'm not sure if it was the reason of your problem.

[arkpar]

Identified a leak here: https://github.com/libp2p/rust-libp2p/blob/2fd941122a6c589501ad1c04b0f091f1337f5f49/protocols/kad/src/behaviour.rs#L1399
@romanb

[romanb]

@arkpar Thanks for the heads-up. https://github.com/libp2p/rust-libp2p/pull/1217 should address that.

[iammelea]

v0.6.9 is out lets close this old issue