Julia: Option for portable build and document recommended JULIA_CPU_TARGET value

[nalimilan]

We currently use JULIA_CPU_TARGET=native by default. That's a reasonable choice assuming people want to use the build on their machine. But when this default was chosen it was not possible to build multiple sysimages optimized for multiple targets. Now that we can do this, it would make sense to replace native with a list of two or three targets, knowing that the target with the most recent instruction set would be as efficient as native even on recent CPUs. That way binaries built from source would automatically be usable on other machines too.

I suggest this change because I've realized distribution packages often build a single target, by setting MARCH (this is the case of Debian/Ubuntu and Arch at least). This means people using Julia in distributions will silently get slower code than they could, which would be bad publicity. This also requires distribution packagers to update the list of targets every few years, which some of them will probably forget to do. Using a default value would allow us to have upstream changes reflected automatically in distribution packages.

For reference, the list of targets used when building official binaries is in this file: https://github.com/JuliaCI/julia-buildbot/blob/dd1708ef6a591d46b358754a3505921d3bc846e7/master/inventory.py

Cc: @ginggs @petercolberg @eli-schwartz

[staticfloat]

Note that while JULIA_CPU_TARGET defaults to the same as MARCH, they are two distinct things. To create a redistributable binary, you need to set both MARCH and JULIA_CPU_TARGET. The benefit we get with a JIT is that we can choose at runtime which one to choose; but for general C code we don't have that luxury, and so we sacrifice compiler performance (from what I remember, not that much) for compatibility by setting e.g. MARCH=x86-64.

So if we're going to default JULIA_CPU_TARGET to something intelligent, we should also think about defaulting MARCH to something intelligent, as right now I believe it defaults to native, meaning that the libjulia.so you build may include instructions that cannot be executed on an older machine.

[cdluminate]

@nalimilan Thank you for the hint. Currently as a workaround I added two flags in Debian's build script, CUSTOM_MKL and CUSTOM_NATIVE. And I use a locally rebuilt Julia package (MKL+native).
https://salsa.debian.org/julia-team/julia/blob/master/debian/rules#L99-107

It would be great if building sysimage for multiple targets was done on the upstream side.

[yuyichao]

That way binaries built from source would automatically be usable on other machines too.

How often is that needed? I don't think it's good to make the compile time and binary size for EVERYONE longer only because the few people that compiles or write the script that compiles julia don't want to set a proper target.

I suggest this change because I've realized distribution packages often build a single target, by setting MARCH (this is the case of Debian/Ubuntu and Arch at least).

And having an document entry or even a recommended Make.users for distros is fine. I just don't think distro package maintainers settings should be the default.

as right now I believe it defaults to native

I don't think so.

[yuyichao]

Currently as a workaround

What are you working around? What's preventing you from setting multiple target in the build rules?

[nalimilan]

Or maybe we could add a JULIA_PORTABLE_BUILD variable defaulting to 0. My point is that if we require package maintainers to master so many different settings, they will necessarily miss some of them or get them wrong (and even more likely miss changes we make across releases). Currently the list of targets used by the official binaries isn't even in the Julia source, but in an external repo.

[yuyichao]

Sure, document, readme, another build option are all fine. As long as it's not the default.

[staticfloat]

as right now I believe it defaults to native

I don't think so.

You are right.

[staticfloat]

It seems to me the easiest thing to do would be to have a JULIA_PORTABLE_BUILD variable that, if set, automatically sets MARCH and JULIA_CPU_TARGET. The main reason, IMO is not so that packagers can set just one option instead of two; it's so that if we decide to change the JULIA_CPU_TARGET string defaults, packages get that transparently, without them having to worry about it.