Mui-datatables: v4 State/Props Refactor Roadmap

[gabrielliwerant]

Intro:

This issue is to track progress on and explain the major refactor that is underway to solve numerous issues relating to state/prop updates as well as to make the codebase simpler, easier to reason about, and better able to support additional features moving forward.

Overview:

The table uses state, derived from props to manage the data passed into the table. This means that whenever the props sent into the table are changed, they cannot be properly displayed in the table unless the state is re-initialized. This is making it quite difficult for users of the library to modify the props sent to the table, and have those changes reflected, which users should be able to do easily. In addition, the code is becoming difficult to maintain, reason about, and test. See this blog entry https://reactjs.org/blog/2018/06/07/you-probably-dont-need-derived-state.html or the discussion here for more detail on this issue: https://github.com/gregnb/mui-datatables/pull/650#issuecomment-499262879.

Solution:

The table should operate mainly from the props sent in and only maintain minimal state that is entirely controlled within the table components. Operating off of props will allow users to make any changes they wish to the their data, anywhere, at any time, and have those changes automatically reflected in the table data.

We can use render to capture changes to props and then transform as necessary, as well as filter for searches, deleted rows, filters, etc. without relying on a separate, internal displayed data state. In essence, we will derive the display state directly from the props in the render function, without storing it in state.

Since we will be performing transformations and filtering on each render, there are opportunities for performance enhancements here via memoization, but the initial refactor will focus mainly on feature parity. Some functions may be become obsolete and will be deprecated as a result of the refactor (e.g. the difficult to understand updateValue function may no longer be needed, as it exists mainly to keep state in sync with changes).

Process:

As soon as a working example is ready, it will be released as 3.0.0-alpha and tracked separately from 2.x.y releases. Features or fixes in the v2 releases will need to be ported over, but it will focus mainly on important bug fixes to minimize churn (though some features that are high in demand can still be added during the transition).

Hopefully the community can then help to test and vet the v3 alpha to make it ready as the replacement for v2.

Roadmap:

• [ ] Separate out all data modeling from state-specific functions (ideally in separate functions and/or files)
• [ ] Replace derived state management with direct transformations and filters on props
• [ ] Update API to include onChange functions to match prop-based data (e.g. if a sort option is set as an option on props, an onChange function will be required to be supplied to the table to update that sort, since it cannot be managed by internal state at that point).
• [ ] Include sample onChange functions which can be imported from the main library
• [ ] Make sure all prior tests continue to pass, though many tests may require modification as the API changes
• [ ] Add new tests for direct prop changes and/or remove tests that relied on internal derived state
• [ ] Update documentation
• [ ] Update examples
• [ ] Add documentation for migrating from v2 to v3
• [ ] Release alpha version and request feedback and testing from the community
• [ ] Promote the alpha release to beta and then official release as it becomes more stable

Final Thoughts:

There are some other big changes, such as updating the react and material-ui package versions that should wait until this refactor is completed, as this refactor is more important for the longterm health and stability of the project, and will hopefully make those upgrades easier. Thank you all for your patience while this work is in progress.

UPDATE:

More information to come, but in order to speed along this and support for material ui version 4, major API changes along the lines mentioned here will become available piecemeal throughout version 3 and not switched over entirely until version 4. In the mean time, version 3 will instead focus on more minor API changes and full support for material ui v4.

[VagnerNico]

Soon as this version is ready I will test in my application, actually, I was going to open an issue about closing the filter interface when the data changes using the server-side option true, but after reading this news I going to wait for the release.
Again congratulations on this amazing and beauty project which is helping ReactJS developers to get access to a functional data table.

[DTupalov]

Separating model from view it's very good idea. I've suggested it early and tried to implement, but without breaking currently API can't did that. As I was need it quickly, I've implemented it in my own library. You can see my approach:

1. With recompose
   https://github.com/DTupalov/react-material-ui-datatable/blob/v2.0.0-alpha.8/src/withReactMUIDatatableModel.js

1. With separate state manager effector
   https://github.com/DTupalov/react-material-ui-datatable/tree/v2.0.0-alpha.21/src/model

Right now it in WIP mode, and it has poor functionality, but I'm using it in my production. I'm add functionality as needed.
You can see examples here:
https://dtupalov.github.io/react-material-ui-datatable/

[gabrielliwerant]

@DTupalov Thank you for your input. I think I remember reading about your efforts in an older issue. My first priority is just going to be to split things apart a bit so that state and props don't step on each other. At some point, we can look at more sophisticated modeling solutions, and I will be happy to have your help.

[DTupalov]

@gabrielliwerant, I have any thought about overview:
most problem is to balance between controlled and uncontrolled mode of the table. In fact we have one component Datatable, which should stick to one of this mode. And we can't make intuitive API, which would give to users understanding, what would be happen, if they will change some prop (it will be apply immediately (controlled), or don't be apply (uncontrolled) ). Thinking wider, we should break one Datatable component to the many components that will fold into one big functionality. And those components can be as controlled (for example pass prop sort and onSortChanged) and uncontrolled (defaultSort). Good example is dx-react-grid that separates State Management, Data Processing and View. And each piece responsible for there functionality (sorting, filtering, paginating etc).

So, in fact, if correctly select such functionality, than we can do something similar, but free and open source :)

BTW, if we do it, we can implement not only MUI design :)

[Intelliflex]

Do you have an expected timeframe? Most things I’ve tried so far still work in conjunction with muiv4 so if I ignore ref errors on console that may still be ok if I know the problems are getting fixed. Congrats on this product by the way - it is excellent ... your worth more money ;)

[gabrielliwerant]

@Intelliflex Unfortunately, I don't have a specific timeframe I can give at the moment. Trying to keep up with some smaller changes first, to help people in the mean time. But I definitely plan to keep the roadmap up to date as I work through it.

Thank you so much for the compliment, however, I'm afraid I can only take a small portion of the credit! This library was mostly written by another fellow (I don't want to @ mention on purpose because I don't want to bother him). I've been taking care of things while he deals with some personal issues and he's given me a wide latitude. I'm mostly trying to keep to what I believe the original vision was, as well as help usher it into the future.

[jkeruzec]

Maybe we can link : #105

[patorjk]

@gabrielliwerant Any updates on the status of the refactor? I'm curious what the new API will be like. I find myself in the boat of wanting to externally control certain information, yet I like the idea of the table keeping internal state on things I don't want to manage. I had an idea for one possible API that could allow for the best of both worlds. It would take a cue from Hooks. Properties in the options prop could take in an array with a value and a setter function if the user wanted to control them, otherwise the table would use internal state to manage those options. If the user just wanted to set the initial value, they could set a value on an init prop. Example:

const [rowsSelected, setRowsSelected] = useState([]);

....

options: {
    init: {
        rowsExpanded: [1.2.3],
    },
    rowsSelected: [rowsSelected, setRowsSelected],
}

Since the user provided a setRowsSelected function, it would be called instead of the table's internal function. It'd be up to the user to provide a setter function that would trigger a re-render.

Anyway, you probably have something in mind already, but I figured I'd throw this out there if you were looking for ideas.

[twawszczak]

@gabrielliwerant I also would like to ask about status of changes. ;-) One more thing - is typescript support a part of refactor? Currently typings are handled by independent repository, so these are outdated in many cases. :(

[gabrielliwerant]

@patorjk Having developers pass in their own update functions is exactly the idea I had in mind for solving many of the issues.

Unfortunately, it's been difficult to find a lot of time for a major refactor while keeping up with other things. What I may begin to do, is instead of a single refactor with a lot of breaking changes, I may slowly add the functions and deprecate older APIs as an ongoing process.

@twawszczak I'm not planning on doing anything specifically with typescript for a while, but there is an independent PR underway you might want to have a look at https://github.com/gregnb/mui-datatables/pull/795 (maybe the author would welcome your help). My only sticking point with official support, is that I want it to be possible for myself and others to continue to build and contribute to the table in js. Basically, I'm not looking to switch this over and change it to ts, but I would consider supporting ts if we can keep js. Additionally, feel free to keep the types up to date in definitely typed!

[plourenco]

@gabrielliwerant Based on your suggestions and @patorjk, and after an in-depth analysis of https://reactjs.org/blog/2018/06/07/you-probably-dont-need-derived-state.html, I ended up creating a wrapper component that accepts the same options as the original table and respective update methods. Initially, the aim was to main the table state across re-renders. It works for now, until the refactor is complete.
An example of such case would be setting the rowsPerPage in the tableOptions together with an update method onChangeRowsPerPage. As a result, it would allow the table to become fully controlled. If by any means, the table needs to update rowsPerPage, it would inform the parent calling onChangeRowsPerPage. If none is set, the table manages its own state.

Since the solution seems very close to your idea for the v4 state/props refactor, I would like to know if any contributions are welcome (this time, in the actual table component).
I should be able to write a new PR and discuss some of these changes.

[xStorm111]

@gabrielliwerant Would be really nice if the v4 have some new features like https://material-table.com/ has. This is just a suggestion to implement.
Example: Grouping, swap columns order by dragging them

[patorjk]

Closing this for now as it won't be happening in the near future. For updates on v3 of the table, which will hopefully be releasing soon, please see https://github.com/gregnb/mui-datatables/pull/1300