Redux: Can We Store A Large Amount Of Data In State?

[gbmhunter]

As part of an app I am designing, I want to display a large amount of changing scrollable text to the user, possibly up to say, 20,000 characters. New characters are regularly added onto the end of this block of text, and the oldest ones disappear.

As far as my understanding goes, this block of text is part of my 'state'. And because of this, I would have to create a new copy of these 20k characters, and then add new ones to the end/delete old ones of the start, everytime some new characters appear. This doesn't seem very efficient or processor friendly.

Is it that:
1) This copying has actually far less overhead than I think, and it's no big deal?
2) This is a 'special case' where I should be mutating the state directly, and if so, how should I do this?
3) This is a 'special case' where the data should not be part of the state, and if so, where should it go?
4) None of above, there is a better solution?

P.S. Keep up the good work!

Regards, Geo

[tcoopman]

I would say 4. You can take a look at _immutable_ data structures like Immutable. These structures don't copy your data if it hasn't changed. This would mean that if your block of text is part of your state and it doesn't change, it wouldn't be copied.

[arusakov]

@gbmhunter for example, immutable list has "efficient addition and removal from both the end and beginning". This is similar to what you need.

[gaearon]

1. Before drawing any conclusions, profile. It's easy to fall into trap of optimizing something that isn't that costly in the first place. Spend an hour building a stress test and you'll learn a lot about your modeling your domain area efficiently.
2. Utilizing structural sharing in Immutable lists, as suggested above, is a good idea.

[gaearon]

Closing as duplicate of https://github.com/rackt/redux/issues/328.

[Markus-ipse]

As an addendum to what @gaearon said about profiling before optimising, if you have a powerful dev-machine try to find a low-end Android device or something representative of your target audience at least, when profiling. Things that work well on you machine might not run well at all on slower devices, not surprising perhaps, but easy to forget.

[uglycoyote]

I am just learning about Redux, and I'm investigating how far Redux takes this "never mutate the state" dogma. I'm finding it incredible that the idea of cloning 20k characters of text each time a change is made would be considered acceptable for performance. The comments above say there are ways of avoiding copying the text if the text has not changed, or if you are modifying the beginning or the end. What if someone is typing and their cursor is in the middle of the block of text? Does it really make sense to keep copying the entire buffer each time a character is added? That seems really unnecessary.

[johnsoftek]

For a block of text, you might consider not changing the app (Redux) state
on each keypress. Just maintain the text in the containing component state
and dispatch an action when the text entry is finished - however you define
"finished" for the text element - e.g. lost focus, enter key, timeout.

On 14 March 2016 at 07:18, uglycoyote [email protected] wrote:

I am just learning about Redux, and I'm investigating how far Redux takes
this "never mutate the state" dogma. I'm finding it incredible that the
idea of cloning 20k characters of text each time a change is made would be
considered acceptable for performance. The comments above say there are
ways of avoiding copying the text if the text has not changed, or if you
are modifying the beginning or the end. What if someone is typing and their
cursor is in the middle of the block of text? Does it really make sense to
keep copying the entire buffer each time a character is added? That seems
really unnecessary.

—
Reply to this email directly or view it on GitHub
https://github.com/reactjs/redux/issues/768#issuecomment-196055512.

[tracker1]

First off, strings are immutable... when you manipulate a string, add characters, remove characters, etc, new copies are created in memory... the implementation details are up to the JS engine... but updating a new state object with a reference to an updated string, isn't much more than that updated string.

That said, you may want to limit your active state to what is being displayed, and use a separate "client" that will feed the subset of the whole into state as part of an action being dispatched regularly (setInterval), just make sure to cleanup properly...

As to changing state onkeyup/oncut/onpasteend/ondragend, for a _large_ text area, it's probably best to stick to _onchange_ and/or encapsulate anything you need from those other manipulation events (character counter) encapsulated in a localized state... note: this is a break from unidirectional flux/redux style workflows, but then again, so is having a field with that much data in it.

[gaearon]

I'm finding it incredible that the idea of cloning 20k characters of text each time a change is made would be considered acceptable for performance.

I think you might be confused about how memory allocation works on JavaScript.

By the time you read the current text from the input (which is something you have to do anyway to make use of it) the corresponding string is already created. Now you can pass it around, save it in the state, and no further “cloning” will occur. Strings are passed by reference, meaning that on the implementation level what is being passed is a number describing the location in memory, rather than a character array.

Of course, were you to _manipulate_ the string, you would create new copies. But again, this is irrelevant to Redux because strings are immutable in JavaScript, so _any_ string manipulation involves creating new strings. There’s no escaping it. Luckily implementations in modern engines use a data structure called “rope” which makes it possible to keep operations on large strings more efficient than with a naïve approach.

In general, I would strongly suggest you to profile performance problems before being wary of them. Often what might seem like a problem is actually not one, and what might seem to be OK turns out to be the bottleneck. Profile, don’t guess 😉

[uglycoyote]

Thanks for the input. You are correct, that my question only really makes sense if the buffer is being stored in some mutable string buffer or tree, as javascript strings are immutible.

So lets imagine we are using a rope or some other structure that allows manipulation of parts of the string. As I understand, a rope is a tree structure where where each leaf is a small part of the overall string. The efficiency of the rope comes from being able to mutate specific sub-branches of the tree while leaving other branches untouched.

I'm still trying to understand how this would be done efficiently in redux, if the dogma of redux is not to mutate the state tree. From the docs that I have read, the reducer function is supposed to be a pure function that takes a "single immutable state tree" of the entire app and returns a new state tree. If I'm following that literally does it not mean that I have to create a deep copy of the entire rope tree in the reducer? Or is there some way that the reducer can be allowed to operate only on a sub-part of the tree (a specific node)?

Apologies if I'm asking something that has been answered in other places, I could not find the answer in the Redux docs but perhaps I just don't know what I'm looking for.

[gaearon]

So lets imagine we are using a rope or some other structure that allows manipulation of parts of the string. As I understand, a rope is a tree structure where where each leaf is a small part of the overall string. The efficiency of the rope comes from being able to mutate specific sub-branches of the tree while leaving other branches untouched.

Yeah. In this case I was referring to implementation details of strings in some JavaScript engines. This is not really relevant to Redux or state trees so let’s forget about the ropes :smile: .

If I'm following that literally does it not mean that I have to create a deep copy of the entire rope tree in the reducer? Or is there some way that the reducer can be allowed to operate only on a sub-part of the tree (a specific node)?

Again, ropes are not relevant here, let’s forget about them. If your question is “does the state tree need to be cloned deeply on every change?”, the answer is “no”.

This is because most actions only affect small part of the state tree. There is one root reducer but it usually split via functional composition in many small reducers managing small slices of the state tree. Those subreducers will only handle actions they are interested in, and return state (that is, their respective slices of it) unmodified for any actions they don’t want to handle. Effectively this means that only the root state object will be re-created every time, but most of the objects it links to will be reused because actions usually affect only some parts of the state tree. Does this make sense? Redux does not ask you to deeply clone the state on every action. It just asks you to return new objects for the parts that _have_ changed. You can reuse the previous state for any parts of the tree that have not changed.

I encourage you to look more closely at the examples and/or watch my video course where I explain reducer composition in more detail.

[uglycoyote]

Thank you, yes that answers my question in the more general case. Thanks for pointing me towards the information about reducer composition, that sounds like exactly what I'm interested in!