Lorawan-stack: Data views slow down and freeze after receiving many events

[kschiffer]

Summary

The data views of the console freeze after receiving a lot of events.

Steps to Reproduce

1. Go to the data view for an entity that receives a lot of events
2. Wait for some time while observing incoming events (waiting time is relative to the event frequency)
3. Observe the app becoming slow
4. Observe the app eventually freezing completely

What do you see now?

Freezing data views

What do you want to see instead?

Data views being stable regardless of the amount of events received

Environment

v3.5.8 and older

How do you propose to implement this?

The problem is likely caused by the redux store being filled considerably when the console is subscribed to an event source. Especially since the packet broker update, it will receive a lot of data per event which easily sucks up memory. My ideas to fix this are:

• Introduce a maximum amount of messages that are stored (rest being truncated)
• Discard event data for old events
• Experiment with storing event data in the browsers localStorage instead of in the redux store. We could then only reference the event data in the redux store. This is assumption-based buy maybe the localStorage can handle large amounts of data better.

@bafonins what do you think?

How do you propose to test this?

Manual testing.

Can you do this yourself and submit a Pull Request?

Yes.

[bafonins]

The problem is likely caused by the redux store being filled considerably when the console is subscribed to an event source. Especially since the packet broker update, it will receive a lot of data per event which easily sucks up memory.

Redux store is certainly related here, but the main problem is rendering. After https://github.com/TheThingsNetwork/lorawan-stack/pull/2477#discussion_r421182736 I removed the limit for the number events being rendered in the events view.

Your solution looks fine, but indeed localStorage requires some experimentation

[kschiffer]

So after working on this for a bit, I think the solution here is threefold:

1. Refactor the overall event component structure, which is currently nested in a way that makes it really hard to maintain and optimize performance (we use element cloning and render props, which makes it impossible to use memoization of the list elements)
2. Use a virtualization-module like react-window to only render the list elements that are currently visible
3. Look into other ways to store the event data, as described in the OP (I'll experiment with the performance after introducing 1. and 2. since as @bafonins mentioned, the performance issues are mainly render related)

[bafonins]

1. You can add memoization to the render function ~the top level event component~.
2. This wont work. I tried using virtualized lists when implementing the first version of the events component. Mainly because of dynamic heights.

[kschiffer]

Unfortunately, this issue persists. Albeit, the event views can handle more load now but there's still a certain threshold of event count and frequency that will cause lag and freezes.

I will reintroduce a maximum amount of events as a quick remedy until we can fine-tune this further.

[johanstokking]

Is this in progress?

[kschiffer]

This is not in progress anymore. I had to close two attempts to improve this because it was not verifiable that these solutions helped. The situation improved significantly after virtualizing the lists via #3229 but there's still a point of failure when the event view receives thousands of events. I somehow missed opening a PR that will solve this by limiting the number of events displayed in the console. Will do that now.

[kschiffer]

I somehow missed opening a PR that will solve this by limiting the number of events displayed in the console. Will do that now.

-> #3470

[kschiffer]

Also, these two comments provide some insight into what I tried to do and what issues I had:

https://github.com/TheThingsNetwork/lorawan-stack/pull/3329#issuecomment-712410891

https://github.com/TheThingsNetwork/lorawan-stack/pull/3330#issuecomment-712937573

Hopefully, this will help future attempts to address the issue more thoroughly.

[kschiffer]

Reopening this since the issue still remains when showing events for applications with ~50+ end devices (this depends on how frequent the end devices send or receive data). There are some ideas to solve this in this comment already.

Let's use this thread to discuss possible other ways to solve this issue. Please also see https://github.com/TheThingsNetwork/lorawan-stack/pull/3329#issuecomment-712410891 and https://github.com/TheThingsNetwork/lorawan-stack/pull/3330#issuecomment-712937573

[johanstokking]

@kschiffer what's the status here?

[kschiffer]

This one is tricky. Solving this properly is blocked on #3817. An intermediate fix is described here https://github.com/TheThingsNetwork/lorawan-stack/issues/2231#issuecomment-779659836. 

To add this, we need to be able to filter event streams on event types, which is one of the things that will be added by the Event Server TheThingsIndustries/lorawan-stack#1804.

So either way there does not seem to be a quick remedy. The only other thing I can think of as preliminary fix is to automatically close the event stream and show an explanation message if it reaches a events/second threshold.

[kschiffer]

So this needs to be removed from the current milestone, since it's not actionable right now as we need a decision on this first.