Treestyletab: Quantifying performance issues with very large amount of tabs

[azizLIGHT]

Short description

I wanted to quantify performance for TST when dealing with very large amounts of tabs. I conducted two tests:

1. Open 1000 tabs of about:blank
2. Restoring a session of 1000 tabs of about:blank

I used what I think are performance optimized settings to test and record how long it took for CPU usage to drop near 0% and how long it took Firefox and TST to become responsive again (can mouse-wheel scroll TST sidebar fluidly without stutter, and switch between tabs via TST sidebar immediately without delay).

Both tests were conducted under three scenarios:

1. Fresh Firefox with no TST
2. Fresh Firefox with TST with performance optimized settings (detailed below)
3. Fresh Firefox with TST with performance optimized settings but auto-grouping of tabs enabled

Steps to reproduce

1. Start Firefox with clean profile.
2. Install TST and set options:
3. Appearance -> Theme -> check "no decoration"
4. Appearance -> Theme -> uncheck "apply browser theme"
5. Appearance -> too long label of tabs -> select "crop with .."
6. Appearance -> uncheck "enable animation effects"
7. Appearance -> uncheck "show collapsed descendants in the tooltip on a tab"
8. New Tabs Behavior -> auto-grouping of tabs -> uncheck "auto-group tabs opened within 100 msec"
9. New Tabs Behavior -> auto-grouping of tabs -> uncheck "confirm to group tabs"
10. New Tabs Behavior -> auto-grouping of tabs -> uncheck "auto-group tabs opened from same pinned tab"
    
    
    
    1. Tree Behavior -> uncheck "when a new tree appears, collapse others automatically"
    
    
    2. Tree Behavior -> uncheck "when a tab gets focus, expand its tree and collapse others automatically"
    
    
    3. Open 1000 tabs of about:blank all at once using a bookmarks folder (Open All in Tabs context menu option)
    
    

Results

| Opening 1000 Tabs (mm:ss timeformat) | no TST | TST optimized (without auto-grouping) | TST optimized (with auto-grouping) |
|----------------------------------------------------------------------|--------|---------------------------------------|------------------------------------|
| Tabs launched | 00:00 | 00:00 | 00:00 |
| FF tab strip: Tabs appear | 00:08 | 00:08 | 00:08 |
| FF tab strip: Tabs move position | 00:11 | 00:19 | 00:18 |
| FF tab strip: Group Tab appears with moz extension URL title | N/A | N/A | 00:55 |
| TST sidebar: Tabs appear but not Group Tab | N/A | 01:03 | 01:02 |
| FF tab strip: Group Tab title changes to "New Tabs and more" | N/A | N/A | 02:32 |
| TST sidebar: Group Tab appears with moz extension URL title | N/A | N/A | 02:41 |
| TST sidebar: Tabs get indented under Group Tab | N/A | N/A | 02:49 |
| TST sidebar: Group Tab title changes to "New Tabs and more" | N/A | N/A | 05:50 |
| TST sidebar: Group Tab throbbing animation appears and does not stop | N/A | N/A | 06:01 |
| FF and TST are responsive. CPU usage low | 00:11 | 01:24 | 06:07 |

• Auto-grouping tabs add significant delay vs No auto-grouping tabs (~4x delay)
• Throbbing animation never stops after it starts
• Responsiveness = being able to scroll with mouse-wheel inside TST sidebar in a fluid manner without any stutter as well as being able to click and switch between tabs using the TST sidebar without any delay after all TST duties (showing tabs and groups, renaming titles, and proper indentations) are completed while CPU usage is relatively low.
• What exactly is happening after 2:49 ? Why isnt TST responsive after this point immediately?

| Restoring session of 1000 tabs (mm:ss timeformat) | no TST | TST optimized (without auto-grouping) | TST optimized (with auto-grouping) |
|----------------------------------------------------------------------------|--------|---------------------------------------|------------------------------------|
| FF tab strip: Tabs appear | 00:05 | 00:05 | 00:05 |
| TST sidebar: appears, is gray, brief throbber animation appears/disappears | N/A | 00:06 | 00:06 |
| TST sidebar: turns completely white with no progress animation | N/A | ? | 00:06 |
| TST sidebar: turns gray, with throbber animation | N/A | ? | 01:20 |
| TST sidebar: is gray, progress bar appears and starts filling up | N/A | ? | 01:22 |
| TST sidebar: progress bar fills up and disappears | N/A | ? | 01:26 |
| TST sidebar: tabs appear, unindented | N/A | ? | 01:35 |
| TST sidebar: tabs get indented | N/A | ? | 01:40 |
| FF and TST are responsive, scrolling is fluid, tab switching is instant | 00:05 | 00:23 | 01:42 |

• The steps in the no auto-grouping are too fast for me to measure individually
• Auto-grouping tabs add significant delay vs No auto-grouping tabs (~4.5x delay)
• Restoring tabs is faster than opening 1000 tabs, in absolute terms

Looking at this data, it looks like auto-grouping needs some performance work when opening tabs. I am not sure why it takes so long for TST to handle empty about:blank tabs. There is np pages to render with about:blank and there are no other addons installed. Of course the results depend on my machine, but I have used the extension on faster machines and similar tab counts and still see similar magnitudes of delay. Is there some architectural issue at fault here? As far as restoring a session goes, the cache was already built when the tabs were opened, and it is a matter of parsing the stored data and rendering the groupings and indentation. So in that respect, I expected less difference between auto-grouping enabled and disabled. I was surprised to see a large difference. In the future I will add a test where auto-grouping is disabled, and 1000 tabs are opened, and also put into a group manually, and then the session is restored after Firefox restart

Test Environment

• Platform (OS): Windows 7 x64 on Intel i5-2520M with 16GB RAM
• Version of Firefox: 68.0a1 (2019-05-13)
• Version (or revision) of Tree Style Tab: 3.0.11.9122

[azizLIGHT]

I did the restore session test where auto-grouping is disabled and 1000 tabs of about:blank are opened under a manually created group tab. TST gives very similar timestamps as if auto-grouping is enabled. This tells me that auto-grouping option is not to blame since the group tab was created by me manually with auto-grouping disabled

I have tried this exact test with a similar addon called Tree Tabs (it has no auto-grouping) and it is able to react very fast.

If I were to do another test: select 1000 tabs and put them under a parent tab, that other addon is able to do it very fast, and TST takes in the order of minutes to accomplish it. What is it about the other addon that it can handle such a large task with such speed? Can anything be learnt from how it is implemented? It is also open source.

[piroor]

Does the fix #2275 affect to the performance?

[piroor]

A performance profile https://github.com/piroor/treestyletab/wiki/How-to-inspect-tree-of-tabs#how-to-collect-a-performance-profile will help such optimizations. We need to reduce bottlenecks (too many function calls or too slow functions) to optimize performance, based on such a profile.

On my environment, there were some bottlenecks around mass reflowing triggered with style changes. The fix for #2275 should reduce DOM changes, and less DOM changes should produce less reflowing.

[azizLIGHT]

I tried again on FF 69.0a1 (2019-06-08) with TST 3.0.17.9313. I see some improvement where some seconds are cut.

In open 1000 tabs test, one step has been eliminated: TST sidebar: Group Tab throbbing animation appears and does not stop due to fixes you issued for #2275. Also, responsiveness is already present at the TST sidebar: Group Tab title changes to "New Tabs and more" step. While it is an improvement and cuts overall loading time, the other steps are performed more or less in the same amount of time

In restoring 1000 tabs test, two steps have been eliminated or are very hard to notice now: TST sidebar: tabs appear, unindented and TST sidebar: tabs get indented. Also, responsiveness is already present at the TST sidebar: progress bar fills up and disappears step, when the progress bar fills. This is also an definite improvement on the latter end of user-visible steps with total loading time seconds shortened, while other steps seem to be taking more or less the same amount of time.

To help with the reduction of bottlenecks, I provided some performance profiles as you requested. I tried for 1000 tabs but unfortunately the buffer gets full and I don't think its complete. I have attached it anyway here along with 200, 400, and 600 about:blank tabs performance profiles as I was able to record those completely from start to finish without buffer getting full.
profile.json.zip

[piroor]

Thanks! I've inspected the downloaded performance profile. I've understood that inheritSoundStateFromChildren() was called too much on all patterns. It takes 75-81% of the freezing time, so it is clearly a large bottleneck.

[piroor]

The commit 73e46aae2b1a5b23f0a19560cd72c1ee5012e802 should optimize the function. Could you try profiling with the development build?: https://github.com/piroor/treestyletab/blob/master/README.md#development-builds

[piroor]

...and more optimization with 6b70eb1.

[piroor]

I've found some regressions, so more commits have been introduced.

[azizLIGHT]

@piroor Here's some more performance profiles on the newest dev build (TST 3.1.2.9379 on Nightly 69.0a1 (2019-06-21) (64-bit) with fresh profile but with the Dark Theme this time :P)

2019-06-22.zip.001.zip
2019-06-22.zip.002.zip
2019-06-22.zip.003.zip
So this is a single zip split into 3 files of 10MB size, just remove .zip from the ending of each filename so you can extract them. GitHub won't let me upload files with names ending with .001 .002 .003 so I put .zip at the end of all of them to be able to upload.

Hope this helps

[irvinm]

@piroor does the new performance profile help at all?

[piroor]

Sorry for this delay, I've inspected the new performance profile. I've realized that there are some major bottlenecks and inheritSoundStateFromChildren() is still there. Hmm...

[piroor]

After more inspection, I've realized that the method was called due to very frequently modifications of parent/child tabs. TST modifies tree structure step by step, on a startup and a session restoration. For example if there is a parent tab and four children, the method is called four times for the parent tab.

So, I think I need to optimize operations around session restoration. TST should not update tree structure step by step, instead it should be done as a bulk operation.

[piroor]

I've mentioned that restoration of tree from top to bottom can trigger needless "update" tasks. For example, if there are four tabs A, B, C and D, and they construct a tree, then to-to-bottom restoration will does:

1. Attach B to A. Update ancestors of A => no ancestor.
2. Attach C to B. Update ancestors of B => A will be updated.
3. Attach D to C. Update ancestors of C => A and B will be updated.

On the other hand, bottom-to-top restoration will do:

1. Attach D to C. Update ancestors of C =>no ancestor.
2. Attach C to B. Update ancestors of B => no ancestor.
3. Attach B to A. Update ancestors of A => no ancestor.

So I think that the change may improve performance of session restoration.

Sadly the automated build system is stopped for now, so you need to build an XPI on your environment.

[irvinm]

@azizLIGHT would you be willing to try this?

[azizLIGHT]

@irvinm I'd be willing to test further, yes. But unfortunately I don't know how to build the XPI to start testing again

[piroor]

The automated build system is back, and TST 3.1.6 is already released. Now you don't have to build the XPI by yourself.

[imero]

Tried to get some startup perormance data but could not get profiler to run first at startup.
What I did was in command line set MOZ_PROFILER_STARTUP=1 and set MOZ_PROFILER_STARTUP_FILTERS=tree and then firefox.exe but profiler did not open at all.

Manually opening tst profiler and de-selecting "Optimize tree restoration with cache" behaves similarily as Firefox startup (seems to take about same time). With that, profiler shows first minute all events are garbage collector waits and in task manager one CPU core 100%. Is this expected behaviour? Tried to increase javascript.options.mem.nursery.max_kb but had no effect.

[azizLIGHT]

FYI the automated XPI build is at https://piro.sakura.ne.jp/xul/xpi/nightly/ . Look for treestyletab-we.xpi, which is modified very recently

[azizLIGHT]

Here's an updated performance profiles on the newest dev build (TST 3.1.7.9465 on Nightly 70.0a1 (2019-08-14) (64-bit) with a fresh profile with no other addons and the same optimized settings (with auto-grouping).

2019-08-14.zip

Profiles included:

• loading 1000 about:blank tabs
• close/open TST sidebar and then expand/collapse 1000 tabs
• close 1000 tabs via TST context menu

This time I recorded the time it took to open 1000 tabs (my first test) and for FF and TST to become responsive again and I saw it finished in 1 min 45 seconds, which is a very significant reduction in load time!! This is down from 6 minutes 7 seconds on the same hardware, but updated Firefoxes and TST versions.

I will try the restore test soon enough as well and update about that too

[piroor]

Thanks! I've researched performance profiles you uploaded.

• The largest bottleneck of the first profile is: querying tabs with conditions, especially nearestCompletelyOpenedNormalFollowingTab and nearestCompletelyOpenedNormalPrecedingTab called from snapshotTree and snapshotForActionDetection, for all newly tracked (maybe restored) tabs. They are called too much times and the number of method calls should be reduced.
• The largest bottleneck of the second profile is: reflowing and re-rendering by Gecko. Currently I have no good idea to reduce this cost.
• The largest bottleneck of the third profile is: collecting of children and descendant tabs. TST does special controls for descendants of closed tabs, but they should be optimized more.

[piroor]

I've introduced more optimizations for cases told at https://github.com/piroor/treestyletab/issues/2278#issuecomment-521534290.

[piroor]

@azizLIGHT Could you try a newer development build?

[azizLIGHT]

@piroor I will tonight!

[azizLIGHT]

Just made some more performance profiles on TST 3.1.7.9473 on Nightly 70.0a1 (2019-08-17) (64-bit) using the last test's profile.

I think when I updated/restarted FF and TST, TST settings seem to have gone back to default. I noticed this during the test instead of before. Hopefully it doesn't make too much of a difference

2019-08-18.zip
Profiles included:
loading 1000 about:blank tabs
close/open TST sidebar and then expand/collapse 1000 tabs
close 1000 tabs via TST context menu

[bb010g]

I noticed this issue a few days ago and was encouraged by the recent performance improvements described, as my last attempt to profile Tree Style Tab's high CPU usage ended up freezing Firefox.

Profiles now were successfully recording, so here's #2368 for some work on the issue of high CPU load & UI stuttering due to excessive allocations by TST.

[piroor]

I've applied most optimizations proposed by @bb010g in different way. Thanks a lot!

After I'm researched the given patch I've realized that the important point is: less GC by reducing very-short-life arrays. And I've also realized that JS's Array provides a builtin method Array.prototype.reduce() for such demands. I think still there are more optimize-able codes...

[irvinm]

@azizLIGHT how are things with the current build? If good, can you close this item?

[irvinm]

@piroor, no updates from the original poster since August 18. Maybe consider closing this out?

[azizLIGHT]

@irvinm Sorry. I will take a look at this again this weekend

[irvinm]

@piroor I think we can close this one.

[piroor]

I close this for now. If any progress, please add comments (then I'll reopen this.)

[bb010g]

Sorry for the delay; life's been crazy. TST is generally much better now. On the session I'm typing this from, Firefox is at 1933 tabs with the sidebar open and CPU is hanging around 10%. However, this is after my trees got flattened after a restart after restoring this window from recently closed, and when heavy nesting existed Firefox's CPU usage was floating around 20% or 25% idle.

I think a good benchmark for Tree Style Tab performance under heavy load would be to load a session of 1–2K or so pending (unloaded) tabs of about:blank with various (random?) levels of nesting, some very deep, into TST and see how resource utilization at idle plays out for the next minute. This could then be compared to Firefox with the same tabs open, but TST disabled.

Back in February, I was starting to look into implementing this, but it never went anywhere. Setting up a quality environment for this test would also probably be a pain. Even just reproducing this manually is a pain because TST doesn't have tools to quickly initialize a stress test session of a thousand tabs or more with any sort of nesting.

---

There's still noticeable lag when rearranging tabs in the tree with the mouse, even on this almost entirely flat setup right now. Without TST animations enabled, dragging around a tab normally has ~0.1–0.2s of lag with the target marker (bar or outline showing where the tab will be moved to), with spikes probably up to ~0.4s. I've seen spikes up to almost a second in the past with a large session of complex tab trees.

---

I also still suspect extension add-on communication to be an issue, but I don't have metrics to prove it & really don't know what a solution to this would be beyond trying to communicate entirely using SharedArrayBuffer instances with a standardized layout.

[imero]

Interesting what is causing the CPU load, maybe some tab content or FF config? Currently have 5k tabs and with about 3 levels average and idle CPU load does not even register, 0%. Opening/closing/moving tabs is slow but this is imo expected in this extreme case so do not recommend using that many tabs. Also there is Firefox crash issue with so many tabs - sudden crash without any crash report or trace so no debug data. This I think is not TST related as the crashing is different depending on FF versions - FF 76 was almost stable but 77 and 78 started acting up again.