Rust: Use lld by default on x64 Ubuntu 20.04 LTS

[Gankra]

This is a metabug, constraining the unbound scope of #39915.

What is lld

A linker that's part of the llvm project, which is desirable for two reasons:

• it's very friendly to cross compilation (hence its emphasis for embedded targets)
• it's very fast (often runs in half the time as Gold -- linking can take several minutes for big projects (rustc, servo, etc.) and linking can be a huge % of the compile with incremental builds, so halving this runtime is a Big Deal.)

Rust currently ships its own copy of lld which it calls rust-lld. This is used by default to link bare-metal targets and wasm.

Goal

The goal of this metabug is to use rust-lld by default on a major x64 linux distro. I have arbitrarily chosen Ubuntu 20.04 LTS as our target. With all likelihood, this will incidentally get it working/enabled on many other linux distros, BSDs, and other similar platforms, but I am purposefully constraining the scope to one distro for the sake of focusing the effort.

My understanding is that the ELF backend is quite well maintained. Quoting lld's own landing page:

We are currently working closely with the FreeBSD project to make LLD default system linker in future versions of the operating system, so we are serious about addressing compatibility issues. As of February 2017, LLD is able to link the entire FreeBSD/amd64 base system including the kernel. With a few work-in-progress patches it can link approximately 95% of the ports collection on AMD64. For the details, see FreeBSD quarterly status report.

Hopefully this has continued to improve, and using lld will be a slam dunk. Although I specify that we should use rust-lld, it may be necessary/desirable to detect and use system copies of lld.

Blocking Issues

• [ ] #71519 - support making gcc use rust-lld
• [ ] #71233 - coverage information not generated when using lld

[cuviper]

As a distro maintainer (Fedora and RHEL), I would definitely want this to use the system lld, at least when we're talking about the system rustc that I compile.

[Hello71]

why ubuntu 12.04? it is EOL for 3 years already.

[Gankra]

Oh whoops, I meant to write 20.04, the latest one.

[est31]

There is no such thing as a Ubuntu 20.04 LTS target platform (unlike wasm or the bare embedded targets you mention), but restricting it to x86_64-unknown-linux-gnu would be a good idea.

[Gankra]

yes, doing a good job with Ubuntu will likely get a lot of x86_64-unknown-linux-gnu working. but i'd rather get one specific (major) distro working first, since we're talking about integrating with system toolchains here.

[est31]

@Gankra I assumed you meant a target based check because the two other cases you mention (wasm and bare metal) are target based. Now I'm even more confused. What do you want to do? A host based check? As in: cargo or rustc trying to invoke lsb_release -a or something at its startup and enable LLD if it's Ubuntu 20.04? What if you cross compile to Windows or Mac? Or do you just want to talk to Ubuntu's rustc packagers so that they make it use LLD in the Ubuntu package?

[Gankra]

I don't intend to explicitly check for the distro/release, but rather am willing to have toolchain checks which succeed on vanilla Ubuntu 20.04 but e.g. don't succeed on the latest Arch or older Ubuntus.

So we don't need to worry about supporting using rust-lld with older versions of gcc/clang or whatever other tools, as long as our checks are solid enough to detect those things and fall back to using the native ld.

The starting point of this would be to not have this detection set up. We would instead blindly try to use rust-lld on x64 linux gnu targets if it's explicitly requested with the -C linker or -C linker-flavor options. This means using those options could lead to catastrophic failures in many configurations, but we will do our best to ensure they work on typical Ubuntu 20.04 installs (e.g. no older/weird toolchains installed).

In all likelihood this will incidentally get most major modern distro releases working well, but I would regard that as a Happy Accident, and not a priority. It would however provide a tool for distro maintainers to check if rust-lld mode works for them, which is good and useful.

After that we would add the detection or expand support until we could have the compiler try to automatically enable rust-lld without an explicit request on x64 linux gnu targets (possibly this step would be done by Cargo and not rustc). Once this is in good enough shape (everything working, not necessarily using rust-lld), we would enable this behaviour in production.

From there we will have a solid foundation for folks to push forward rust-lld support on their platforms by either making our toolchain detection more sophisticated, or by updating their distros to meet our requirements.

[est31]

@Gankra thanks for the explainer. Have there been any "please test LLD" threads? Those were done e.g. for pipelined compilation and might be helpful in this instance as well.

Also maybe one should think about adding an LLD label for issues so that they can be tracked.

[Gankra]

we should have an mvp of #71519 before we ask folks to try out linux lld support

[Gankra]

(CC to @rui314 in case they're interested in the fact that we're investigating this -- see also #71520)