Terminal: Test Terminal in low-memory conditions, measure/reduce working set, minimize number of used shared libraries

[vadimkantorov]

In top bar we have + and v buttons.

After not using Terminal for some time (the app was opened, but not used, e.g. after recovery from hibernation), sometimes there're up to 2-3 seconds passing between clicking on v and context menu drop down appearing. Maybe it's delay for reading "settings.json" or loading memory pages into physical mem and O.S. being slow, but it'd be nice to have some guarantees on this drop down appearing (it'd better say loading if this can take a lot of time).

Same for + button. It takes me up to 5 seconds hanging before the tab appears (I'm not even talking of the title being shown and WSLv1 bash started).

After I've used these buttons couple times, they become faster, but still lag noticeably:
+ button maybe takes up to 100-200ms (judging by eye)
v has some 50-100ms delay

I wonder if these can be optimized to appear instantly and then do some loading, then one has a feeling that terminal does not hang and is doing some work.

It'd also be good if new tab instantly prints some temporary title, before the shell loads. Currently it's empty for up to 1 second (time of hanging + loading the shell inside the tab).

This is on version: 0.11.1333.0

It would also be nice to have these delays printed somewhere in the logs (maybe if some debugging mode is enabled), so we can easily pull them out, understanding that time-to-rendering may be hard to extract from the UI framework. But it'd be good to have for diagnostics, since terminal must be super-fast. E.g. I stopped using calc.exe and Windows Photo Viewer, since they take up to 5-10 seconds to load and render (probably because of using slow UI framework and requiring a lot of libraries loaded from disk), so I want my terminal + python loading fast for calculation needs :)

cc @DHowett-MSFT

[zadjii-msft]

So for the record, those buttons are built statically when the Terminal is first launched or the settings are reloaded. If there's any delay in them appearing for you, it's not coming from our code, it's probably lower down in the XAML stack.

On my machine, that dropdown appears instantly, and tabs appear (reasonably) fast, though the animation for the tab could use some improvements. The tab content certainly appears instantaneously.

What kind of hardware are you working with on your machine?

[vadimkantorov]

I'll record a gif next time I recover from Hibernate. I understand that it's probably XAML that's not fast enough (and . A more satisfactory response would be if Terminal could push the XAML framework team to improve performance or at least do logging/tracing performance in order to have some numbers to record/inverstigate/profile perf as an end-user (including enabling this for an already running app). Maybe terminal could somehow ask Windows to not unload Terminal pages to disk or to not do memory compaction to them.

It's definitely hardware to blame: I have too many Chrome tabs (with a Great Suspender, but still it eats a lot), only 8Gb and HDD. It's an old-ish Lenovo L470.

But still, even on little RAM resource (most likely that's the problem) Terminal would better be fast and use very little resources on its own.

[zadjii-msft]

if Terminal could push the XAML framework team to improve performance

Honestly, these kind of requests are usually louder when they're made by members of the community. They've got their own repro over at https://github.com/microsoft/microsoft-ui-xaml/, and they'll be far more helpful at diagnosing your perf issues than we ever could be.

[vadimkantorov]

It's Terminal team that made the choice of UI framework, I think it's expectable for users to report perf problems to Terminal team and at least get help with instrumentation / proper measurement.

If you choose to refer regular terminal users to directly communicate with Microsoft UI Xaml team, that's like hoping that regular users are all like Bruce Dawson and can independently do a complete investigation and serve as liaison between Terminal and XAML => a lot of burden and very little chance for success. I'm a regular user, not a perf engineer.

There are long-standing similar requests in Windows Calculator: https://github.com/microsoft/calculator/issues/209

[zadjii-msft]

Okay, but I don't know how to collect XAML perf traces, and the XAML team does. I don't think anyone's expecting that level of investigation from random users. I think if you said "something in the XAML stack is performing slowly, could you help me collect traces to help identify the issue", they'd be more than happy to help. Especially since you've got such a nice compact repro case to analyze. They're regular engineers just like you and me, heck they even sit in the offices in the hallway next to ours ^{when people used to work in offices of course}

[vadimkantorov]

I'll ask them, sure. Just I have zero context about how Terminal uses XAML framework. Unless they have a ready recipe for instrumenting Terminal, this will likely not go anywhere.

The fact that you do not have done UI perf instrumentation before is troubling, especially given the known problems with XAML perf on not giant machines. Especially for such a core app like Terminal (arguably even more core than calc.exe or Photo Viewer).

[zadjii-msft]

I mean, I'd love to use something more performant and consistent with Windows UI guidelines, if it existed. I'm not sure which of the following would have been a better choice though:

• WPF, which isn't actively maintained and updated any longer, and would have required writing the application in C# (which would require booting up the CLR on launch, not just XAML)
• Win32 & GDI, which is a technology that's older than I am, and certainly looks that way. This would have been the most painful possible development experience.
• HTML/JS, which would have required standing up a WebView or electron instance, which is 40MB of RAM just to display an empty window
• Doing it all directly with DComp, which would require us to built all of our own custom UI elements, inevitably leading to a _less_ consistent and performant solution.

If there are performance problems with using WinUI XAML in the Terminal, I'd much rather use those performance issues to help drive improvements within the framework to benefit _all_ Windows applications.

[vadimkantorov]

Well, I understood that you don't have generic UI perf tracing at the moment :) It's quite unfortunate, but I'll create an issue in XAML. Let's see what they say.

I'd much rather use those performance issues to help drive improvements within the framework to benefit _all_ Windows applications.

Me too, it's just that specific app-context is important. And tracing may require cross-linking with app-specific events. E.g. in the linked calc.exe issue one of linked PR was delaying loading some libraries and reducing working set size. Look at advice an author of a performant XAML app gave there - it's quite app-specific. If the problem is with too-large working set and loading too many UI libraries at button click, a solution could be preemptive library loading to memory when Terminal gets focus or maybe load some settings from disk preemptively in the background and not on UI thread.

But anyway, I'll create an issue and discussion can also continue there.

[vadimkantorov]

So far my current/inconclusive summary (discussion continues in that issue) is that WT has a quite large WorkingSet: 65Mb wtih PrivateBytes: 200Mb (one WSLv1 tab open) according to Procexp, and uses a lot of shared libraries and also some fonts. In low memory conditions they only start loading from disk synchronously when a user interacts with the UI.

So one goal could be to have a WT app with a smaller working set and include used memory resources in key metrics

Another could be starting loading shared libraries/unswapping in background once the user focuses the app. An opposite direction could be somehow making sure that no unnecessary / very few libraries are loaded while basic UI interaction.

One way for prevention could be somehow showing a some message UI libraries are loading (but putting the message may itself be tricky if UI libraries are not loaded), but at least Windows has a way to check if memory pages are not in physical memory yet: https://stackoverflow.com/questions/51342150/how-to-determine-if-a-logical-address-is-in-physical-memory, or at least putting some message after they are loaded indicating that WT lacks memory for smooth operation.

Maybe also one way could be asking to never page WT on disk (or have such option). I'm not sure if this is possible. Right now Google Chrome always eats all memory :)

E.g. WT has Windows.UI.Xaml.Phone.dll, Windows.UI.Xaml.InkControls.dll, Windows.UI.Xaml.Maps.dll in modules list while I use WT on a regular laptop and no intention of using maps :)

Related tricks about hinting Windows to not swap out or to swap in the process in one go:
https://stackoverflow.com/questions/1039017/can-i-tell-windows-not-to-swap-out-a-particular-processes-memory
https://superuser.com/questions/600698/is-it-possible-to-hint-windows-to-bring-a-process-completely-out-of-swap

[zadjii-msft]

From reading that thread, this doesn't sound like the Terminal is doing anything particularly wrong here. Nothing above sounds like anything we're doing specifically, it sounds like these might be things that's universal to UWP XAML applications. I'd love to see a comparison with a dummy UWP application with just a MenuFlyout, something to isolate WT effects on memory pressure compared to a baseline UWP application.

I'd love to reduce our own memory usage, but this sounds like an issue that would be better served by driving the solution upstream. If there's a way to reduce the number of loaded libraries for the Terminal, it'll probably be useful to reduce that commitment from _all_ UWP applications 😉

[vadimkantorov]

Terminal doesn't do anything particularly wrong.

On the other hand, Terminal could do / at least discuss potential perf improvements for Terminal even without waiting for some general Windows / XAML / WinUI level improvements.

E.g. was (IL?) linker considered? (for Terminal assemblies and for WinUI assemblies to maybe eliminate unused methods in controls) Why is XAML.Phone and XAML.Maps being loaded at all? Can Terminal reduce number of used libraries? (it would probably lead to faster startup times as well) What are libraries that Terminal loads to draw the context menu? Could it forbid Windows from paging to disk a few core libraries? What is minimal free memory requirement for Terminal for comfortable use? Can we have private bytes tracked as part of release cycle and have a goal that Terminal app should work in low-memory scenarios?

This issue may be better solved upstream, but pointing out perf problems in concrete scenarios (along with numbers, measurements, some telemetry etc) should come from downstream, isn't it?

[zadjii-msft]

I'm pretty sure IL Linker isn't really useful for a C++ project 😉

We're just using bog-standard XAML, we're really not doing anything special. I couldn't tell you why the Phone and Maps DLLs are getting loaded, they sure don't _seem_ useful, but I honestly couldn't tell you. Hence why I'm thinking that this is a problem that should be investigated to the root of the issue. The Terminal finding a workaround is fine, but that just hides a potentially more fundamental issue with the platform.

IMO, having smaller, more atomic repros makes root-causing issues easier. It's easier for the platform to fix "I have an app that just does a MenuFlyout, and it loads all these DLLs and has this horrible perf" compared to "the MenuFlyout in the Terminal is slow". The first is easy* to debug and analyze - there's not that many moving pieces in that scenario. The second is much harder - it's easier for someone to say "well it might not be XAML's fault, it might not be the Terminal's fault", etc. Hence why I'd love to have a baseline comparison. To isolate whether this is truly the Terminal or XAML's fault.

[vadimkantorov]

About IL linker - you are right :) Is there some other native linker that could strip off unused parts of controls libraries and Link all Terminal / dependencies in one executable. This may or may not help. For that a testing lab setup (with Windows running in low-memory mode) should be done.

About more atomic repos - I guess you are right. Unfortunately, I at the moment do not have resources to contribute in Terminal perf testing at this level.

At the very least I'm proposing to explicitly track and measure Terminal perf / working set sizes / strartup time in low-memory mode and collect some telemetry and number of setups that have these problems (regardless of XAML / UI. this can be done by tracking major page faults and their completion time). Then the ask in XAML / WinRT will gain more priority and have more context.

[DHowett]

With this new title, and the input in the thread, I'm yanking triage and marking this "help wanted."

[vadimkantorov]

@DHowett i hope help is wanted from performance engineers / Windows perf experts (from MS?)...

[DHowett]

Is it necessary for you to be snarky in most of your responses to us?

[vadimkantorov]

It is not. I am sorry for snarky comments.

@zadjii-msft suggested that performance investigation requests to XAML team should be upon users which is very strange to me (and expectedly there is no substantial response from them). In my opinion, requesting performance investigation from regular users asks for at least some explanation or comment.

It is my genuine question if help is required of users or if help is being requested from real experts (within MS included). My wording may not be right, but I genuinely do not understand why a basic performance question (such as mere measuring working set size) is not a considered a core one and requires help from users.

[Rob-Wal]

I didn't want to create a brand new issue for this.

Are there any user steps for troubleshooting or debugging wt slow startup? On one of my system if I do not run terminal for a while it takes 10-20 seconds to startup. I have gone as far as uninstalling, deleting settings, deleting anything in the install directory, etc and performing a clean install. I have tried changing the default profile to cmd. It does not matter.

The terminal app is just a blank grey box during the slow startup then the tab loads. CPU, RAM and storage seems find when this delay is happening. After a random number of so slow startups it will load instantly until its not used for a while or after system starts.

cmd, powershell and wsl startup instantly when runindependently

Win10 1909 (Version 10.0.18363.959
Terminal 1.1.2021.0 (even running wt --version takes a while for the box to show)
I am usually connected to work vpn on that pc. But issue exist when not connected.

[DHowett]

@Rob-Wal When it's taking a while to start up, would you mind trying to run wsl -l (specifically) from another shell and seeing whether that's unusually slow? Terminal will wait a small handful of seconds for wsl to respond with a list of distributions before moving on during startup.

[Rob-Wal]

@Rob-Wal When it's taking a while to start up, would you mind trying to run wsl -l (specifically) from another shell and seeing whether that's unusually slow? Terminal will wait a small handful of seconds for wsl to respond with a list of distributions before moving on during startup.

Yes, if I run wsl -l from powershell or cmd during slow startup its slower than normal. Its about as half as slow as terminal startup. Its instant when Terminal is already open or closed.

Update: I tried commenting out my wsl entry from settings, but terminal recreated it in the config file.

[Rob-Wal]

@DHowett I was able to run some ProcMon trace today. I see the below LocalState hanging for 15seconds. The settings.json is the only file in that folder.

Is this the wait for wsl or something else?
Any ideas?

Time of Day | Process Name | PID | Operation | Path | Result | Detail | Duration | Completion Time | Command Line
-- | -- | -- | -- | -- | -- | -- | -- | -- | --
**44:17.9** | WindowsTerminal.exe | 14548 | NotifyChangeDirectory | C:\Users\USERNAME\AppData\Local\Packages\Microsoft.WindowsTerminal_8wekyb3d8bbwe\LocalState | CANCELLED | Filter: FILE_NOTIFY_CHANGE_FILE_NAME, FILE_NOTIFY_CHANGE_DIR_NAME,   FILE_NOTIFY_CHANGE_ATTRIBUTES, FILE_NOTIFY_CHANGE_SIZE,   FILE_NOTIFY_CHANGE_LAST_WRITE, FILE_NOTIFY_CHANGE_SECURITY | **15.30458** | **44:33.2** | "C:\Program   Files\WindowsApps\Microsoft.WindowsTerminal_1.1.2021.0_x64__8wekyb3d8bbwe\WindowsTerminal.exe"

[zadjii-msft]

That's probably just the Terminal listening for changes to its LocalState directory so that it can be notified when setting.json changes, and hot-reload the settings.

If WSL is acting slowly for you, I'd recommend adding

"disabledProfileSources": ["Windows.Terminal.Wsl"]

to your settings, which will prevent the WSL profile generator from running at all.

[Rob-Wal]

@zadjii-msft Thank you but this did not help. I also did an uninstall/re-install before adding the setting. My WSL instance is not available so I know the setting worked. The native console when running cmd, bash or powershell launch in 1sec. 🤷🏾‍♂️

[Rob-Wal]

@zadjii-msft Thank you but this did not help. I also did an uninstall/re-install before adding the setting. My WSL instance is not available so I know the setting worked. The native console when running cmd, bash or powershell launch in 1sec. 🤷🏾‍♂️

After a lot of time troubleshooting, I have finally found the root cause of the slow startup of WT on my system.

So I spun up windows sandbox on the same system and wt had no issues in the sandbox. I unregistered wsl distro and completely removed wsl (I backuped up first), no change. I started to compare processes on my other pc with no issues and AV was a difference.

It was the F&(3&*ing AV!

I was able to reproduce and solve the issue by unloading and starting the AV. I added WindowsTerminal.exe to the AV Trusted Applications list and WT has been consistently starting fast. Thats the solution I have been running with. Nothing in the AV logs at all. Its TrenMicro OCSE 11.x