Microsoft-ui-xaml: WinUI incremental building not working causing a disastrous full build experience

[Felix-Dev]

There are a couple of unfortunate things I have noticed over the past weeks and months working on WinUI when it comes to the incremental build process of WinUI.

• While incremental builds work when on the same local branch, unfortunately, as soon as I switch branches and build the project again, Visual Studio is always doing a full build of the project. In fact, further builds will also always be full builds until I do a clean. Only then will incremental builds work again. What that means for my current workflow? Switch a branch -> do a clean -> enjoy a full build to get incremental building working again,

• As soon as I modify an .idl file (like NavigationView.idl) and start a build process, VS always does a full build instead of an incremental build.

WinUI Full build experience is disastrous

Why are full builds a problem here? First, full builds take up to 10 minutes on my Core i7-8750H maxing out my CPU during most of the time. Secondly, and most importantly, the VS build process consistently runs out of memory at some point during the build process. I am particularly dreading the moment the compiler reaches the following files during the build process:

ScrollInputHelper.cpp
ScrollPresenter.properties.cpp
ScrollControllerInteractionRequestedEventArgs.cpp
ScrollControllerScrollByRequestedEventArgs.cpp
ScrollControllerScrollFromRequestedEventArgs.cpp
ScrollControllerScrollToRequestedEventArgs.cpp
ScrollPresenter.cpp

When the VS compiler reaches these files, the RAM consumed by it jumps significantly, so much that VS often runs out of memory here and outright cancels the build task. This happens with me making sure that VS has at least 8 GB if not 9 GB of RAM available before starting a build process. And to be clear, these are not the only files in the build process causing the out of memory issue but that issue does occur more often than not at this stage in the build process.

I think I don't have to tell you that this experience is really frustrating. Especially as VS was already building for a few minutes here until it runs out of memory. In other words: I just completely wasted my time.

Case in point, yesterday, two simple commits took 45 minutes of my time because I just couldn't get VS to finish building the project here (so I could quickly verify their correctness in the MUXControlsTestApp). This is unacceptable and absolutely detrimental to get anything done in WinUI.

While the root cause of the build failure always seems to be VS running out of memory, the reported error messages vary a bit. Here are the three different types of errors I have seen (writing them out of my memory here so those will lack details):

• PCH - The compiler ran out of heap space
• System.OutOfMemoryException - VS just cancels the task build (that's the one I often see occuring around the Scroll*.cpp files above)
• Internal C++ compiler error - Please contact the VS team (seems to happen rather rarely)

Nearly 50% of my full builds end in such failures and when such a failure happens, all bets are off. VS needs to be restarted, and I even had to restart Windows (!) at one point. More time being wasted here getting nothing done...

Note that the specific System.OutOfMemoryException error I mentioned above only started occuring since a few days ago. I am not aware of any VS or Windows update I did over the past week which could have contributed to that. I use VS and Windows in the following versions:

• Windows 10 19041
• Visual Studio 16.6.2

How to improve the WinUI build story

So, ideally, these issues could be solved and the WinUI development experience significantly improved by achieving the following:

• Switching (local) branches won't require a full build of the project. Instead, incremental building will keep working.
• Modifying IDL files also won't cause a full build of the project and use incremental building instead.

When incremental building works, building WinUI does not take away the fun of working on it. When it doesn't - especially in common cases like the ones mentioned above - the WinUI development experience is disastrous and takes away all the fun.

I'm not sure if this is something the WinUI team can fix or if we need some VS folks to help out here. If it's the latter, I'd hope that the WinUI team can put some emphasis on these issues internally at MS so that the WinUI build process in VS can be improved in a timely manner. I'm hardly one to press for immediate action on this repo but I have to say these past days have just been incredibly frustrating.

[StephenLPeters]

@llongley FYI

[StephenLPeters]

and @kmahone FYI

[robloo]

With 16GB of RAM and 8 GB of free HDD space I wasn't able to build. Ram was used up and then the disk space. How do you guys manage this internally? Is this going to get a lot worse with WinUI 3.0?

[ranjeshj]

I agree that having long build times is quite painful for dev loop. @chingucoding and others helped build the inner loop solution which can be used to filter what exactly builds which should drastically reduce build time if you are working on a single feature. More info here. You can essentially build a much smaller subset and associated tests and let the lab do the full build and test run.

[yoshiask]

This issue is one that made me drop trying to make a new control for WinUI, because 8GB of RAM and Ryzen 5 2500U in a 2-in-1 with a puny cooling solution just isn't enough for it.

[robloo]

This is definitely a significant barrier to contributions. This and being written in the strangest mix of C++/IDL/Generated Code I've seen in a long time -- I miss 'pure' architectures and C#.

That said, we really should add minimum and recommended system requirements to the documentation somewhere, I'm still not sure what Microsoft is using internally to make building more reasonable but they must be some pretty powerful systems.

Long-term, a lot of optimizations and changes are needed to improve the build experience across the board.

[ranjeshj]

Regarding the out of memory issues, it might be related to the number of cores on your machine and the multi proc builds - https://docs.microsoft.com/en-us/cpp/build/reference/mp-build-with-multiple-processes?view=vs-2019. By default msbuild is using the number of processors on the machine to figure out the parallelism.

Turning off multi proc builds should avoid the out of memory, but that is going to increase build time unfortunately. There are msbuild options to limit the parallelism but I don't see an option that scales well depending on the hardware and avoids the out of memory. @llongley and @alwu-msft as FYI.

[Felix-Dev]

I have been using the MUXControlsInnerLoop solution consistently in the last 1-2 weeks and that definitely delivers a significant build improvement. In fact, during the past two weeks I only once had to deal with VS breaking down during a build due to no more memory available using the inner loop. And I did quite a few builds during that time period. No more breakdown every second full build!

@ranjeshj That said, I noticed that apparantly API tests are not available in the inner loop solution:

Assuming you see that as well on your machine, is this something which could be improved? For now, the missing API tests require me to load the full MUXControls project and start a full build - at which point the API tests can be run again.

There also seem to be some issues with the interaction tests in the InnerLoop solution. For example, while I can start interaction tests, the NunberBox interaction tests constantly fail to find the test numberbox instance and thus the tests fail with a NullReference exception. Is that something you are seeing too and if so, could this be improved as well?

[ranjeshj]

@Felix-Dev can you see if it works for you in the debug-test configuration instead of debug ? I think debug-test ensures that the test app packages are built to enable running tests.

[Felix-Dev]

@ranjeshj Nice! That solves the missing API tests:

While the specific NumberBox interaction tests still fail to find the NumberBox instance I have tested other interaction tests just now, like NavView interaction tests, and they work fine. Perhaps I made a recent change to the NumberBox test page which I do not recall right now which is causing the NumberBox issues.