Nixpkgs: firefox is slower than firefox-bin

[ivan]

Describe the bug

Repeated test runs of https://browserbench.org/Speedometer2.0/ in a clean Firefox profile show that nixpkgs firefox scores about 68.5, while Mozilla's firefox-bin scores 82 (both tested on Intel 4790K).

Mozilla builds Firefox with Clang and LTO, maybe should do that too?

To Reproduce
Steps to reproduce the behavior:

1. Install firefox and firefox-bin on NixOS master.
2. Run https://browserbench.org/Speedometer2.0/ a few times using both
3. Observe firefox result is ~83% of the firefox-bin result

Metadata

• system: "x86_64-linux"
• host os: Linux 5.4.5-hardened, NixOS, 20.03.git.14e375c (Markhor)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3.1

cc maintainers @edolstra @andir

[andir]

IIRC they are also doing some profile based optimizations and not just LTO. We can probably dig that out of their source trees or the build logs..

e.g. here is a log of an x86_64-linux optimized build: https://firefox-ci-tc.services.mozilla.com/tasks/X7r7t9OyTBWrJZ31Kg3itg/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FX7r7t9OyTBWrJZ31Kg3itg%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log

[lovesegfault]

I recently tried enabling LTO for Thunderbird and ran into a myriad of issues before giving up. The Gentoo ebuilds support both LTO and PGO, but porting the support is not trivial, I think.

[lovesegfault]

c.f. https://gitweb.gentoo.org/repo/gentoo.git/tree/www-client/firefox/firefox-71.0-r1.ebuild

Grep for pgo and lto

[wamserma]

c.f. https://gitweb.gentoo.org/repo/gentoo.git/tree/www-client/firefox/firefox-71.0-r1.ebuild

Grep for pgo and lto

here's a stable link https://gitweb.gentoo.org/repo/gentoo.git/tree/www-client/firefox/firefox-74.0-r2.ebuild?id=da126193051d51f5bf90d079b16883f553476cad

[S-NA]

I have successfully built Firefox v79 with LTO using lld and clang from llvmPackages_10 (i.e., stdenv = super.overrideCC super.stdenv super.llvmPackages_10.lldClang;) with some hardships of course.

The change in performance was a ~15% percent increase compared to the default firefox build under the Speedometer 2.0 benchmark. Note this is not a comparison to firefox-bin. As well as compared to the firefox build I patched in sndio support and had some extra configureFlags set such as --enable-rust-simd, etc., so your mileage may vary and this is a far from scientific result.

Some peculiar things likely due to using lld as the linker was the binaries produced by firefox-unwrapped had missing libraries, namely libunwind.so.1. I was able to fix this by using patchelf --replace-needed as well as setting doInstallCheck = false. The reason why doInstallCheck had to be set to false was because of the following error:

XPCOMGlueLoad error for file /nix/store/h7m3nb7i0l2a2qr954dcnwydjyxvfl9n-firefox-unwrapped-79.0/lib/firefox/libxul.so:
/nix/store/h7m3nb7i0l2a2qr954dcnwydjyxvfl9n-firefox-unwrapped-79.0/lib/firefox/libxul.so: undefined symbol: _ZN7mozilla11SandboxInfo10sSingletonE
Couldn't load XPCOM.

This was fixed by in the firefox wrapper setting libs = "${self.firefox.passthru.unwrapped}/lib/firefox:" + oldAttrs.libs; so the produced shell script exec's firefox works. Though directly launching the firefox-unwrapped variant does not. You can of course set LD_LIBRARY_PATH manually to $out/lib/firefox make it find the libraries.

I should mention this was all done in an overlay.

[lovesegfault]

Could you share your work @S-NA? I've gone down this road before a few times and I've never succeeded past the "Couldn't load XPCOM" point in the installCheck, I'm glad you got it working!

If you put this up I can start playing with trying out PGO as well as LTO.

[S-NA]

I have not had the chance to clean up the expression so it is a bit messy but sure here you go:
https://gist.github.com/S-NA/b8ed8fe9b1cb2b56d92f7201a11c4ad7

Enjoy.

[S-NA]

I have a draft of LTO support (and some other stuff) into my firefox's common.nix for awhile now (since Firefox 79). The main change is Firefox is now built using clang and lld which allows easy LTO support. Going from Firefox 79 to 80 (and possibly 81 if it builds) required no significant changes which seems good.

I find it cleaner than what I originally had cobbled together to get LTO in https://github.com/NixOS/nixpkgs/issues/76484#issuecomment-673231152. I recently [hopefully] modified it to support Firefox 81. (It currently is building.) If someone is interested in testing it, make sure to override ltoSupport = true;. The main reason I have not tried to upstream it is the non-LTO Firefox configuration needs to be tested. In the event LTO breaks rather than delay an update, falling back to non-LTO should be easy. However, I am starting to wonder if that is really needed. Nevertheless, it needs more testing which is hard do because of long build times (which is purely from my own hardware limitations).

[wamserma]

How about upstreaming it as firefox-lto? This would allow for painless testing and a smooth switchover once it has matured.

[colemickens]

This gets a +1 from me. I've been wanting this. Are we actually worried about LTO causing regressions?

[colemickens]

cc: @worldofpeace since they were interested in and merged my last firefox PR

[S-NA]

This gets a +1 from me. I've been wanting this. Are we actually worried about LTO causing regressions?

My apologies, I should have mentioned I created PR #99922 for this. As for LTO causing regressions it is a possibility but not one I think is worth being concerned about anymore. LTO support which implies clang and lld brings us closer to what upstream tooling is like for their Linux builds which is good.

Darwin is currently a mystery but it probably doesn't build. lld is unsupported there so it needs to be handled. I pushed a commit to place --enable-linker=lld behind a conditional to fix this but then realized overrideCC selection probably needs some more logic for Darwin. While investigating what needed to be done I learned LTO is broken across Darwin so for now I've asserted LTO doesn't work there. It should build but not with LTO unfortunately. I would appreciate if someone could verify that.

[worldofpeace]

@colemickens I approved on @S-NA PR :sparkles: I will leave the merge up to andir.