Sdk: Could we use -Oz flag to reduce the dart VM size？

[wangying3426]

@mkustermann @mraleph It almost reduce 1M when use -Oz to replace -O3 flag to compile dart vm runtime. So,could we use -Oz or -Os to reduce the dart VM size in a release mode? And is there any risk?

Thank you!

[mraleph]

I have sent this patch to our performance evaluation infrastructure. I am seeing 10% to 50% performance drops on various benchmarks. GC performance is affected by around 20% - so I would say that Dart runtime becomes on average 20-30% slower. Performance of benchmarks that is not spending considerable time in Dart runtime is unaffected - but I don't think Flutter applications fall into this category.

[wangying3426]

I have sent this patch to our performance evaluation infrastructure. I am seeing 10% to 50% performance drops on various benchmarks. GC performance is affected by around 20% - so I would say that Dart runtime becomes on average 20-30% slower. Performance of benchmarks that is not spending considerable time in Dart runtime is unaffected - but I don't think Flutter applications fall into this category.

Thank you very mach, we are also evaluating the impact on performance to see if we can balance performance and package size.

[mkustermann]

Our runtime code is quite large and I would assume only parts of it are really performance critical. We can try to identify those hot spots and mark those functions with __attribute__((always_inline)) and use -Os.

At least it's worthwhile doing an experiment.

[mraleph]

@wangying3426 let us know what you find.

I think @mkustermann is right - we should evaluate those benchmarks that regressed and see if we can get performance back by turning on optimizations more selectively, e.g. we could compile files which contain GC implementation with O3 while compiling the rest with Oz or be even more targeted.

[wangying3426]

@mraleph We use Flutter benchmark to evaluate the performance after change O3 to Oz on iOS，only to find that the timeline is almost no difference, just like the picture shown. Maybe GC has less impact on UI thread due to the mechanism of using idle time to GC, or is our benchmark not cover the Dart's cases?

And, according to previous experience, we found that FPS of Flutter applications depends mostly on GPU, especially for those low-end Android devices whose GPU is low, therefore Dart influence on FPS would be reduced?

The first column is base, and second column is Oz's.

And also, I think "turning on optimizations more selectively" is really a wonderful idea ! :)

[dnfield]

@wangying3426 - it would be helpful to know what hardware you ran those benchmarks on. Our devicelab setup intentionally runs in slower/older hardware to make things consistent, and if you used a later model Android or iPhone that could explain things.

That said, it's entirely possible that Dart is still "fast enough" for whatever else Flutter engine/framework have to do in a given frame. We should also be evaluating whether the build flag is suitable for Flutter engine builds.

If it turns out that compiling with this flag has no major adverse affect on the benchmarks Flutter has, but it does actually slow things down, we may want to try to capture such benchmarks in our devicelab. Alternatively, if it slows down what Flutter would consider "microbenchmarks".

I'm curious if @goderbauer ever tried this flag for the Dart VM, or if @chinmaygarde has. I can also try it locally to just see. Also /cc @liyuqian who probably has good ideas here.

[liyuqian]

Wow, I'm happily surprised that our developers are so dedicated to performance: even our quite-flaky-and-hacky and not-well-documented device lab performance tests and cocoon dashboards are reproduced locally!

With that dedication, my first thought is that @wangying3426 could probably use the open-source nature of Flutter and Dart to compile and deploy whatever version that best fits their performance metrics and goals.

Flutter would of course be very interested in learning how -Oz flag works in the wild if -Oz is deployed in production by some Flutter apps. Flutter has opened https://github.com/flutter/flutter/issues/37437, and we're currently working on a precursor issue https://github.com/flutter/flutter/issues/37434. If @wangying3426 can share some metrics about -Oz in production, I think that would allow Flutter and Dart to make a more informed decision.

Finally, we do have some very early and experimental work on how to automatically tune inline to optimize binary size and probably some other performance metrics (e.g., speed of Flutter). I'll forward this issue there.

[wangying3426]

@wangying3426 - it would be helpful to know what hardware you ran those benchmarks on. Our devicelab setup intentionally runs in slower/older hardware to make things consistent, and if you used a later model Android or iPhone that could explain things.

That said, it's entirely possible that Dart is still "fast enough" for whatever else Flutter engine/framework have to do in a given frame. We should also be evaluating whether the build flag is suitable for Flutter engine builds.

If it turns out that compiling with this flag has no major adverse affect on the benchmarks Flutter has, but it does actually slow things down, we may want to try to capture such benchmarks in our devicelab. Alternatively, if it slows down what Flutter would consider "microbenchmarks".

I'm curious if @goderbauer ever tried this flag for the Dart VM, or if @chinmaygarde has. I can also try it locally to just see. Also /cc @liyuqian who probably has good ideas here.

Our devices are iPhone 6 (12.4.1) and iPhone 5s (12.4).

We have use -Oz in production on iOS，but we have no more online comparative FPS data yet due to some reasons. If have more metrics, I will synchronize it for you.

[dnfield]

@a-siva those links are 404s

[dnfield]

Discussed this over chat - the benchmarks look like they're google-internal, but there are only a few significant regressions that could probably be optimized specifically while still allowing for most of the runtime to be compiled with Os or Oz.

[dnfield]

Been looking into this again. @wangying3426 - I'm a bit curious about where you saw 1m reduction in size. I'm seeing more like 100kb to 150kb depending on how much of the VM I allow to be optimized. I'm looking at the compressed size (i.e. the size in the JAR file) primarily for Android release arm64.

The size is bigger uncompressed, but still not 1m.

[wangying3426]

Maybe due to the different of arm modes and flutter version? Most of our apps use arm64 on iOS and arm-v7a on Android. After our testing, it reduce 1M on iOS release arm64, and 340 K on Android release arm-v7a.

The size is Flutter.framework‘s disk occupation.
 
In fact，the package size problem on iOS is more important than that on Android.

flutter doctor -v :
```
[✓] Flutter (Channel xxxxxx, v1.5.4-hotfix.2-pre.161, bd0.0.0-unknown, xxx on Mac OS X
10.14.6 18G103, locale en-CN)
• Flutter version 1.5.4-hotfix.2-pre.161 bd0.0.0-unknown at xxx
• Framework revision 628d129241 (2 weeks ago), 2020-01-21 14:51:55 +0800
• Engine revision 224a4d2b64
• Dart version 2.3.0 (build 2.3.0-bdflutter.8a385f2e81)

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.2)
• Android SDK at /Users/wangying/Library/Android/sdk
• Android NDK location not configured (optional; useful for native profiling support)
• Platform android-29, build-tools 29.0.2
• ANDROID_HOME = /Users/wangying/Library/Android/sdk
• Java binary at: /Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/bin/java
• Java version Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
• All Android licenses accepted.

[!] iOS toolchain - develop for iOS devices (Xcode 11.2)
• Xcode at /Applications/Xcode.app/Contents/Developer
• Xcode 11.2, Build version 11B52
✗ Verify that all connected devices have been paired with this computer in Xcode.
If all devices have been paired, libimobiledevice and ideviceinstaller may require updating.
To update with Brew, run:
brew update
brew uninstall --ignore-dependencies libimobiledevice
brew uninstall --ignore-dependencies usbmuxd
brew install --HEAD usbmuxd
brew unlink usbmuxd
brew link usbmuxd
brew install --HEAD libimobiledevice
brew install ideviceinstaller
• ios-deploy 1.9.4
• CocoaPods version 1.7.5

[✓] Android Studio (version 3.4)
• Android Studio at /Applications/Android Studio.app/Contents
• Flutter plugin version 39.0.1
• Dart plugin version 183.6270
• Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1343-b01)

[!] Android Studio
• Android Studio at /Applications/Android/Contents
✗ Flutter plugin not installed; this adds Flutter specific functionality.
✗ Dart plugin not installed; this adds Dart specific functionality.
✗ Android Studio not found at /Applications/Android/Contents
• Try updating or re-installing Android Studio.

[✓] VS Code (version 1.41.0)
• VS Code at /Applications/Visual Studio Code.app/Contents
• Flutter extension version 3.7.1

[✓] Connected device (1 available)
• BKL AL00 • RKKDU17B29000948 • android-arm64 • Android 9 (API 28)
!```

[dnfield]

I've opened https://dart-review.googlesource.com/c/sdk/+/135791 to address this.

Unfortunately, going directly to -Oz introduces a number of performance regressions we probably dont' want to accept. However, going to -Os with some parts as -O3 seems like a good tradeoff of space/speed (smaller number of regressions in Dart benchmarks, and some improvements).

-Oz only saves a few more KB on top of -Os in this case. We might be able to squeeze a little more, but the win become less obvious and it may not be worth the additional maintenance burden.

[wangying3426]

I've opened https://dart-review.googlesource.com/c/sdk/+/135791 to address this.

Unfortunately, going directly to -Oz introduces a number of performance regressions we probably dont' want to accept. However, going to -Os with some parts as -O3 seems like a good tradeoff of space/speed (smaller number of regressions in Dart benchmarks, and some improvements).

-Oz only saves a few more KB on top of -Os in this case. We might be able to squeeze a little more, but the win become less obvious and it may not be worth the additional maintenance burden.

ok, good work.

[dnfield]

@wangying3426 I'm not sure about the iOS numbers you're seeing. With -Os my Flutter binary for arm64 on iOS comes out at 11mb - that's without bitcode. With bitcode it's much larger because of all the bitcode IR - about 163mb.

I'm curious about your build flags and revision info. I'm also curious if the binaries you have work :D

[dnfield]

Also, to be clear, Flutter would want to avoid a situation where it was building Dart with different optimization flags than Dart builds itself. If we do that, we will definitely run into issues where Dart refactors something and our flags no longer are capturing the right critical paths with -O3 etc.

[wangying3426]

@wangying3426 I'm not sure about the iOS numbers you're seeing. With -Os my Flutter binary for arm64 on iOS comes out at 11mb - that's without bitcode. With bitcode it's much larger because of all the bitcode IR - about 163mb.

I'm curious about your build flags and revision info. I'm also curious if the binaries you have work :D

It works fine, and has been used in the online environment for a long time. we have some other optimizations included which metioned at #40220 😄, just need to be care about of the decreasing number.

[dnfield]

@wangying3426 -

It would be helpful if you could share a repository with the build configuration/build scripts necessary to achieve this result.

It would also be helpful if we could see the full size of other shared libraries that the Flutter binary depends on - for example, I strongly suspect you're building Flutter here with a dynamic Skia library, and perhaps other shared libraries as well. In my experiments with doing this, those libraries actually grew significantly in size when loaded dynamically - there are also performance considerations, particularly around getting to first frame quickly.

For example, if the Flutter dylib is normally around 10MB, and you've gotten it down to 5MB without changing compiler flags on the Dart VM, what else was stripped out and how big is it now? Is the combination of Skia + Image Decoders + whatever is doing SSL work + libcxx + whatever else was stirpped out also still under the original size of the Flutter dylib?

[liyuqian]

@dnfield : Is there an easy way to count how many modules are compiled with Oz, Os, and O3 while we're doing this transitioning? I'd like to also apply such counting to Flutter engine.

[dnfield]

The only way I know how to do that is to grep the generated .ninja files and see what arguments are actually being applied to clang.

For the most part, we use -Oz in the engine, but some modules (like dart) override that.

[chinmaygarde]

The only way I know how to do that is to grep the generated .ninja files and see what arguments are actually being applied to clang.

An more reliable indicator would be parsing out/compile_commands.json.

[liyuqian]

The problem with out/compile_commands.json is that it's only available right after gn. If I first gn with release mode, and then gn with debug mode, all release mode info will be replaced by debug mode info in out/compile_commands.json. Is there something that we can count from out/android_release?

[dnfield]

I'm assuming you only care about gn in release mode, and compile_commands.json will give you a better idea - but if that's not working well enough you can just go through the .ninja files in out/android_release.