Nixpkgs: Firefox 82 and firefox-bin 82 do not run under Wayland when LTO is enabled

[artemist]

Describe the bug
A clear and concise description of what the bug is.
When I attempt to run firefox-wayland within my Sway session it runs under X11. This happens even if I set XDG_SESSION_TYPE=wayland and MOZ_ENABLE_WAYLAND=1. If I set GDK_BACKEND=wayland then Firefox prints to stderr:

(firefox:4408): Gtk-WARNING **: 23:20:43.048: cannot open display: :0
mozilla::detail::MutexImpl::~MutexImpl: pthread_mutex_destroy failed: Device or resource busy

and then crashes with a segfault. I get the same result with firefox-bin

To Reproduce
Steps to reproduce the behavior:
1.nix-shell -p firefox-wayland --run "env GDK_BACKEND=wayland firefox"

Expected behavior
A clear and concise description of what you expected to happen.
Firefox runs using Wayland

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Notify maintainers

@edolstra @andir

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

• system: "x86_64-linux"
• host os: Linux 5.8.14, NixOS, 21.03.git.a920bf43082 (Okapi)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3.7
• channels(root): "nixos-21.03pre246543.24c9b05ac53"
• channels(artemis): ""
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: [firefox,firefox-bin,firefox-wayland]
# a list of nixos modules affected by the problem
module:

[andir]

Yeah I can confirm this happening. For some reason my firefox-wayland now defaults to X11 unless I unset DISPLAY (which comes from yet to figure out locations).

[artemist]

If I unset DISPLAY then I get the same error for GDK_BACKEND=wayland, just with an empty display. I have no idea where this is coming from. This is probably a NixOS thing since Firefox runs perfectly fine in Wayland under arch

[andir]

I am currently rebuilding firefox without LTO as that is also something that recently changed on unstable.

[andir]

Yeah, disabling LTO did the trick. I'll open a PR disabling it again and then we can go ahead and figure out why it is causing this regression.

[S-NA]

Sorry for the delay, I will investigate this.
I got around to digging into this. The issue seems to be that libwayland-client.so's symbols are not being found. libmozwayland.so is supposed to be a sort of wrapper which provides them by dlopen or some other run-time magic and for some reason it is not. One can however set LD_PRELOAD with libwayland-client.so to work around this. At this time I have not come up with a proper solution but I am still looking into this.

[andir]

We can probably patch the rpath in the Mozilla wrapper as part of the Firefox build. I am still a bit irritated by the fact that this only occurs on LTO.