Stride: skip asset build if there are no asset changes

[jeske]

Is your feature request related to a problem? Please describe.

I'm frustrated when working in C# code / Visual Studio, and every time I build/run I have to wait for a fairly slow AssetCompiler build even though I have not touched any assets and don't even have Xenko Studio running.

Describe the solution you'd like

If I don't touch any assets, then don't rebuild the asset bundle. Here is a conservative speedy way to do this...

(a) Have the asset compiler dump an "asset-build-dependency" snapshot when it builds an assetbundle, which would include the path (relative? absolute?) and modify timestamp of every file that was used to build the asset bundle. When starting a new asset build, it would first load these dependencies, and check all files and timestamps to see if anything changed since the last build, and if not, it would skip the asset build.

The dependencies could be stored right in the asset bundle, or they could be stored in a separate dependency build file.

Describe alternatives you've considered

(b) The asset bundle could just check the modify timestamp of the studio-project-file. This would be much faster, but then if someone edited an asset (such as model or image) outside studio, the build wouldn't re-run the asset build, and this is probably not a good thing.

(c) Alternatively, a running copy of the studio could be watching for changes to any file in the project, and if anything changed, it could touch the modify timestamp of the project file, and prompt the user to reload the asset. This would allow using both the above super-fast one-file dependency checking, and allow picking up changes to any asset file (by keeping studio running in the background). However, it would require educating users that they have to keep studio running in the background, which could be non-intuitive.

(d) Perhaps the best option would be to combine (a) and (c). Dump the dependency list when building as a "fallback". When the asset compiler runs, it could check to see if studio has this project open, and if so, it could assume studio will touch the project file if anything changes, and thus use the "fast" single file check. If it's not running, it would fallback to checking the entire dependency list. Because implementing this solution requires implementing (a) anyhow, it seems better to just implement (a) now and not worry about (d) unless (a) is actually slow for some practical project.

[xen2]

Here's a quick summary about the current state of things:

Asset build is done in several steps:

1. Read assets and generate "build steps". This is the job of AssetCompiler classes.
2. Compile "build steps" to the git-like storage (this is usually were most of the time is spent in initial build). This is the job of BuildEngine.Command classes.
   We cache things by parameter-hashing, and use timestamps on file inputs as you described, so this step should be almost instant on the 2nd build.
3. Pack the git-like storage in a single file (asset bundle).
   We have an incremental mode where it would basically do nothing on the 2nd run.

So assuming an incremental build (step 2 and 3 should be very fast), what still takes time is:

• Step 1 (read assets and generate build steps).
  We could improve on that by doing what you said (input/output and timestamps).
• Process startup and initialization; this is quite slow. We do various initialization in module, and there's the JIT. Some room to make it faster, but it can only go so far.
  We used to have a server/client process (a little bit like MSBuild and Roslyn compiler) so that process could stay in memory, but this was quite complex to maintain and sometimes behaving strangely. It is currently disabled.
  Once we switch to .NET Core, there might be various performance improvements to do, and AOT could help there as well.

To go even further, once step 1 has input/output + timestamp asset-level caching, one approach to skip build almost instantly would be to interoperate more closely with MSBuild and be able to generate list of input/outputs and make MSBuild not even start the process anymore (which takes some incompressible time).

[xen2]

Here's the doc for MSBuild: https://github.com/dotnet/project-system/blob/master/docs/up-to-date-check.md
I now remember looking into it in the past, but the "Set" feature was missing (or at least not documented) so skipping asset compiler was somehow possible manually but it was impossible to achieve the fast path. Seems like it's now possible :)

[jeske]

My main use case is the full shortcut of the asset build when there are no changes in studio.

This not only happens when i change only code.. but also during a typical build fail, build fail, build succeed, run cycle. Even if assets are changing, that final run always needlessly rebuilds them again.

The msbuild looks very promising way to speed this up, as it would mean avoiding the asset compiler launch entirely.

[xen2]

I have made a prototype of MSBuild bypass, it seems to work quite well!

[jeske]

I'm not 100% sure, but this documentation suggests that Include= specs can include * and ** wildcards.

https://docs.microsoft.com/en-us/visualstudio/msbuild/how-to-select-the-files-to-build?view=vs-2019

To include all .jpg files in the Images directory and subdirectories
Use the following Include attribute:

Include="Images\*\.jpg"

To include all .jpg files starting with img
Use the following Include attribute:

Include="Images\*\img.jpg"

I guess an option question is whether it supports patterns that end in wildcards, such as:

Include="Assets\\**

[jeske]

Okay, this change is the bomb.... when working in Visual Studio, it speeds up my workflow 10x, because I can really really quickly build the game now when there are no asset changes. Thanks so much for this.