Treestyletab: Migrate to virtual-DOM based iplementation

[piroor]

Branch: https://github.com/piroor/treestyletab/compare/object-base-tracking

Planned steps to migrate:

• [x] Step 1: Store status and extra information of tabs (collapsed, parent, children, etc.) to tab objects, instead of tab elements.
• [x] Step 2: Isolate each module from needless DOM access.
• [x] Step 3: Switch cahce for the background page from HTML to JSON.
• [x] Step 4: Stop tracking of tab events by sidebar page itself, and manage all tabs by the background page completely.
• [ ] Step 5: Render tab elements from tab objects.

[piroor]

Step 1 is half on the way.

• basic tab states are now stored to tab objects.
• queryings of tabs based on attributes are still there.
• relation of tabs are not yet.

To complete the step 1, I need to merge the object-base-tracking branch to master at first, to delete attributes-based queryings... otherwise the branch will go too far from the master branch and become unmergable.

[piroor]

For more dogfooding, I've merged the branch to the master.

[piroor]

List of modules already isolated from DOM access:

common/

• [x] common/api-tabs.js
• [x] common/api-tabs-listener.js (around I/O)
• [x] common/bookmark.js
• [x] common/cache.js (around I/O)
• [x] common/commands.js
• [x] common/common.js (except dumpTab())
• [x] common/constants.js
• [x] common/contextual-identities.js
• [x] common/diff.js
• [x] common/handle-accel-key.js
• [x] common/metrics-data.js
• [x] common/migration.js
• [x] common/permissions.js
• [x] common/sidebar-status.js
• [x] common/tabs-container.js (deleted)
• [x] common/tabs-group.js
• [x] common/tabs-internal-operation.js
• [x] common/tabs-move.js
• [x] common/tabs-open.js
• [x] common/tabs-update.js
• [x] common/tabs.js
• [x] common/tree.js
• [x] common/tst-api.js
• [x] common/user-operation-blocker.js

background/

• [x] background/background-cache.js
• [x] background/background.js
• [x] background/browser-action-menu.js
• [x] background/context-menu.js
• [x] background/handle-group-tabs.js
• [x] background/handle-misc.js
• [x] background/handle-moved-tabs.js
• [x] background/handle-new-tabs.js
• [x] background/handle-removed-tabs.js
• [x] background/handle-tab-focus.js
• [x] background/handle-tab-multiselect.js
• [x] background/handle-tree-changes.js
• [x] background/index.js
• [x] background/successor-tab.js
• [x] background/tab-context-menu.js
• [x] background/tree-structure.js

sidebar/

• [x] sidebar/collapse-expand.js
• [x] sidebar/color.js
• [x] sidebar/drag-and-drop.js
• [x] sidebar/event-utils.js
• [x] sidebar/indent.js
• [x] sidebar/index.js
• [x] sidebar/mouse-event-listener.js
• [x] sidebar/pinned-tabs.js
• [x] sidebar/restoring-tab-count.js
• [x] sidebar/scroll.js
• [x] sidebar/sidebar-cache.js
• [x] sidebar/sidebar-tabs.js
• [x] sidebar/sidebar.js
• [x] sidebar/size.js
• [x] sidebar/tab-context-menu.js
• [x] sidebar/tab-drag-handle.js

[piroor]

Progress:
https://github.com/piroor/treestyletab/compare/2.7.22...62f5df746232f9453f0a95569b4bc84d8bcfcf80

[piroor]

By recent changes, querying for tabs are logged to TabsStore.queryLogs (recent 100000 queries). If you use very large number of tabs, the logs will help me to detect slow queries. Please run the script TabsStore.queryLogs in the debug console for the background/background.html and send the result to me. If there are quite slow queries, I'll add "indexes" for such queries.

[piroor]

I've added a button to show the performance log from the options page. When you see too slow operations on a development version of TST 2.8.0pre, please send the performance log me or upload it here.

[SXZ1]

So I've just loaded a session with 6700 tabs, enabled "Collect performance logs of tab querying" in settings and tried to expand a bunch of stacks. Here is what I got:
https://pastebin.com/WTjrh0Ai

I tried to expand the stack with 3000 child tabs at the end of the log so telemetry likely went mad at that moment.

[piroor]

@SXZ1 Thanks a lot! Here is an analysis:

Top 10 slowest queries:
{"windowId":"background","toId":5886,"normal":true,"states":["collapsed",false,"expanding",true],"ordered":true,"living":true,"elasped":78}
{"windowId":"background","normal":true,"ordered":true,"living":true,"elasped":37}
{"windowId":"background","living":true,"hidden":false,"!id":[6749],"states":["collapsed",false,"subtree-collapsed",false],"hasChild":true,"ordered":true,"elasped":34}
{"windowId":"background","controllable":true,"ordered":true,"living":true,"elasped":34}
{"windowId":"background","living":true,"states":["dragging",true],"ordered":true,"elasped":34}
{"windowId":"background","controllable":true,"ordered":true,"living":true,"elasped":34}
{"windowId":"background","living":true,"hidden":false,"!id":[6768],"states":["collapsed",false,"subtree-collapsed",false],"hasChild":true,"ordered":true,"elasped":33}
{"windowId":"background","living":true,"hidden":false,"!id":[2938],"states":["collapsed",false,"subtree-collapsed",false],"hasChild":true,"ordered":true,"elasped":33}
{"windowId":"background","controllable":true,"ordered":true,"living":true,"elasped":33}
{"windowId":"background","normal":true,"ordered":true,"living":true,"elasped":33}
Count of query tyepes:
{"count":84,"item":{"living":true,"normal":true,"ordered":true,"states":["collapsed",true,"collapsing",true],"toId":"given","windowId":"background"},"total":1976}
{"count":84,"item":{"living":true,"normal":true,"ordered":true,"windowId":"background"},"total":99}
{"count":56,"item":{"attributes":["data-current-uri",{}],"first":true,"living":true,"windowId":"background"},"total":140}
{"count":56,"item":{"last":true,"living":true,"ordered":true,"windowId":"background"},"total":0}
{"count":42,"item":{"living":true,"ordered":true,"states":["dragging",true],"windowId":"background"},"total":651}
{"count":39,"item":{"fromId":"given","last":true,"living":true,"normal":true,"ordered":true,"windowId":"background"},"total":5}
{"count":30,"item":{"living":true,"states":["throbber-unsynchronized",true],"visible":true,"windowId":"background"},"total":49}
{"count":26,"item":{"living":true,"ordered":true,"states":["selected",true],"windowId":"background"},"total":88}
{"count":23,"item":{"living":true,"ordered":true,"windowId":"background"},"total":855}
{"count":21,"item":{"first":true,"fromId":"given","living":true,"ordered":true,"windowId":"background"},"total":0}
{"count":21,"item":{"fromId":"given","last":true,"living":true,"ordered":true,"windowId":"background"},"total":0}
{"count":16,"item":{"!id":"array","highlighted":true,"windowId":"background"},"total":554}
{"count":5,"item":{"controllable":true,"living":true,"ordered":true,"windowId":"background"},"total":133}
{"count":4,"item":{"first":true,"fromId":"given","living":true,"normal":true,"ordered":true,"windowId":"background"},"total":3}
{"count":4,"item":{"first":true,"living":true,"status":"loading","visible":true,"windowId":"background"},"total":672}
{"count":3,"item":{"living":true,"normal":true,"ordered":true,"states":["collapsed",false,"expanding",true],"toId":"given","windowId":"background"},"total":1966}
{"count":3,"item":{"living":true,"ordered":true,"pinned":true,"windowId":"background"},"total":781}
{"count":3,"item":{"living":true,"ordered":true,"visible":true,"windowId":"background"},"total":682}
{"count":2,"item":{"living":true,"states":["throbber-unsynchronized",true],"visible":true,"windowId":"background"},"total":49}
{"count":1,"item":{"highlighted":false,"id":"array","windowId":"background"},"total":630}
Sorted in total elapsed time:
{"count":84,"item":{"living":true,"normal":true,"ordered":true,"states":["collapsed",true,"collapsing",true],"toId":"given","windowId":"background"},"total":1976}
{"count":3,"item":{"living":true,"normal":true,"ordered":true,"states":["collapsed",false,"expanding",true],"toId":"given","windowId":"background"},"total":1966}
{"count":23,"item":{"living":true,"ordered":true,"windowId":"background"},"total":855}
{"count":3,"item":{"living":true,"ordered":true,"pinned":true,"windowId":"background"},"total":781}
{"count":3,"item":{"living":true,"ordered":true,"visible":true,"windowId":"background"},"total":682}
{"count":4,"item":{"first":true,"living":true,"status":"loading","visible":true,"windowId":"background"},"total":672}
{"count":42,"item":{"living":true,"ordered":true,"states":["dragging",true],"windowId":"background"},"total":651}
{"count":1,"item":{"highlighted":false,"id":"array","windowId":"background"},"total":630}
{"count":16,"item":{"!id":"array","highlighted":true,"windowId":"background"},"total":554}
{"count":56,"item":{"attributes":["data-current-uri",{}],"first":true,"living":true,"windowId":"background"},"total":140}
{"count":5,"item":{"controllable":true,"living":true,"ordered":true,"windowId":"background"},"total":133}
{"count":84,"item":{"living":true,"normal":true,"ordered":true,"windowId":"background"},"total":99}
{"count":26,"item":{"living":true,"ordered":true,"states":["selected",true],"windowId":"background"},"total":88}
{"count":30,"item":{"living":true,"states":["throbber-unsynchronized",true],"visible":true,"windowId":"background"},"total":49}
{"count":2,"item":{"living":true,"states":["throbber-unsynchronized",true],"visible":true,"windowId":"background"},"total":49}
{"count":39,"item":{"fromId":"given","last":true,"living":true,"normal":true,"ordered":true,"windowId":"background"},"total":5}
{"count":4,"item":{"first":true,"fromId":"given","living":true,"normal":true,"ordered":true,"windowId":"background"},"total":3}
{"count":21,"item":{"first":true,"fromId":"given","living":true,"ordered":true,"windowId":"background"},"total":0}
{"count":21,"item":{"fromId":"given","last":true,"living":true,"ordered":true,"windowId":"background"},"total":0}
{"count":56,"item":{"last":true,"living":true,"ordered":true,"windowId":"background"},"total":0}

I've added more codes to output same analysis to the log viewer. It will help furthur investigations.

[piroor]

I've introduced caches for these cases:

{"count":84,"item":{"living":true,"normal":true,"ordered":true,"states":["collapsed",true,"collapsing",true],"toId":"given","windowId":"background"},"total":1976}
{"count":3,"item":{"living":true,"normal":true,"ordered":true,"states":["collapsed",false,"expanding",true],"toId":"given","windowId":"background"},"total":1966}

I hope that collapsing/expanding of tree following to large number of tabs become optimized.

[piroor]

And more indexes were introduced.

[SXZ1]

I've upgraded to the latest dev. build and recorded one more log. Just like the previous log, this one was recorded with caching system disabled. I collapsed/expanded a couple of stacks, the last stack I tried to expand contained 2000 tabs. Unfortunately, github or pastebin don't allow to upload the full log so I'll post the part that seemed the most relevant to me:

https://pastebin.com/gwwnVhc0

Also, I think the lyon's share of CPU time goes to .css styles, especially the ones related to this bug:

https://github.com/piroor/treestyletab/issues/1994

They are responsible for the (fade-out) effect of the tab labels. I'll try to check if that's true by forking TST and removing those commits.

[SXZ1]

Also, subjectively, collapsing/expanding the stack indeed got faster with latest commits (compared to TST with commits related to #1994).

[piroor]

@SXZ1 Thanks again for new query logs. I've seen that there are two similar slow queries:

Sorted in total elapsed time:
{"count":13085,"query":{"living":true,"ordered":true,"source":"sidebar"},"totalElapsed":392334}
{"count":6333,"query":{"living":true,"ordered":true,"source":"background"},"totalElapsed":171579}

They are used to retrieve all tabs, and I've realized that in most cases I don't need to retrieve them - I just need to know the number of tabs and others. So I've introduced more changes to do things without such a slow operation.

[piroor]

After more debugging and optimizing, I'm planning to release TST 2.8.0, it will include just one improvement: a migration of the primary data model of the background page from DOM to JSON. (And as the final step I'll release TST 2.9 or 3.0 after I change the architecture.)

[SXZ1]

I tried to disable the processing of fade-out effect when stack is expanded by removing this line of code:

https://github.com/piroor/treestyletab/compare/master...SXZ1:master

Subjectively, collapsing/expanding stack got faster, I also recorded a new log:
https://pastebin.com/z0bmeg7x

TST2.8.0 seems to be 100% stable so far.

[piroor]

On the step 4, I should use Port.postMessage() instead of runtime.sendMessage(), because it looks fast. With a simple benchmark, Port.postMessage() is faster than runtime.sendMessage() about 4 times.

I think that it is fast because Port.postMessage() sends messages to a specific target, and it doesn't provide ability to receive "response" from the receiver. So, I need to use the feature carefully, for example using it only for one-way messages.

[piroor]

TST 3.0 (3.0.2) is now available. It doesn't include a migration to virtual DOM, but most important migration about its architecture is finished: one-way operation from the background page to sidebars. Chaotic asynchronous operations on both background page and sidebar are simplified. Today all tab operations are tracked only with the background page, and the sidebar is working just like a canvas controlled by the background page completely.

[joehillen]

@piroor TST seems to be working much better now and all the problems with resuming a session seems to be solved. Thank you for all your hard work!

[piroor]

I'm trying to describe the module design of TST 3.0.x with figures, for developers:
https://github.com/piroor/treestyletab/wiki/Module-Design
I hope that it help people who plan to modify TST.

[piroor]

I close this because outdated.