Runtime: Allow SSE/SSE2 intrinsics in R2R and crossgen scenarios

[tannergooding]

CoreCLR currently requires SSE/SSE2 support as a "min-bar" on x86/x64. However, we have disabled all HWIntrinsic code-paths for the ready-to-run and crossgen scenarios.

Given that these ISAs are a minimum requirement and the JIT will already emit SSE/SSE2 instructions for non-HWIntrinsic code paths, we should look into enabling these two ISAs for the above scenarios.

[tannergooding]

CC. @CarolEidt, @jkotas, @fiigii

Does this sound like a reasonable thing to do for 3.0? It looks simple enough to enable and I was going to try and get a PR up.

[jkotas]

Sounds reasonable to me.

Note that we are about to switch CoreLib to R2R as well: https://github.com/dotnet/coreclr/pull/21497

[CarolEidt]

I don't think it is reasonable to attempt this for 3.0.

First, we need to focus on stabilization. I'm currently working on a desktop regression due to dotnet/coreclr#21499.

Second, if we enable SSE/SSE2 code in R2R, there could be perf impact due to additional VEX transitions.

We may want to consider whether we should support guarded use of HW intrinsics for R2R, but I don't think we should consider this for 3.0 without strong justification.

[jkotas]

cc @fadimounir

[fiigii]

Second, if we enable SSE/SSE2 code in R2R, there could be perf impact due to additional VEX transitions.

Agree.
Only enabling SSE/SSE2 in R2R is a reasonable workaround. Eventually, we need a solution for all the HW intrinsic in AOT compilation, especially for AVX-SSE transition penalty https://github.com/dotnet/coreclr/issues/21062.

In JVM world, there are two solutions as I know.

1. Graal-based AOT compiler (JAOTC and SubstrateVM) can select target ISA by configuration (-XX:UseAVX) . That makes post-install/upgrade compilation better to fit the target hardware.
2. Run the core lib code "normally" (interpreter + JIT).

@CarolEidt @jkotas Is it possible to have a similar solution in the .NET world?

[tannergooding]

I don't think it is reasonable to attempt this for 3.0.

Thanks for the feedback @CarolEidt. I'll put this in the future milestone instead.

[CarolEidt]

Graal-based AOT compiler (JAOTC and SubstrateVM) can select target ISA by configuration (-XX:UseAVX) . That makes post-install/upgrade compilation better to fit the target hardware.

IMO a configuration-based approach is not ideal, though I'll let @jkotas comment further on that.

Another option, used by the static C++ libraries I believe, is to dual-version the code at some granularity.

[jkotas]

we need a solution for all the HW intrinsic in AOT compilation

Note that the primary purpose of R2R images is to make the startup faster and be resilient to changes in dependency versions and other configuration changes.

The solution for all the HW intrinsic for AOT compilation would be only needed if:

• We choose to support full AOT configurations that cannot fallback to JIT
• Or we find that the JITing caused by non-precompiled HW intrinsics is the top bottleneck

[fiigii]

@jkotas I agree, but a more urgent problem is the AVX-SSE transition penalty from JITed code calling AOTed code, please see https://github.com/dotnet/coreclr/issues/21062#issuecomment-439578185

[jkotas]

We plan to depend on tiered JIT for best throughput and optimize the system for it. Does tiered JIT make that penalty go away?

[CarolEidt]

Does tiered JIT make that penalty go away?

I believe that it should, assuming that there are no constraints that prevent us from re-jitting all the R2R methods on the hot paths.

[fiigii]

So the plan is to JIT compile hot system methods in Tiered1? And other system methods are still AOT compiled?

If yes, I am not pretty sure. The vzeroupper insertion strategy described at https://github.com/dotnet/coreclr/issues/21062#issue-381772049 is a _convention_ for minimum insertion. So, if some code does not follow the convention, the penalty (dirty upper state) may infect to other places...

When you enable tiered JIT on mscorlib, please let me know. We need to collect PMU data to verify.

[mjsabby]

For self contained apps that may specify their entire app (include S.P.C) as a single version bubble, it makes sense to allow the tool to specify a target ISA. I was about to open the same issue that as SPC goes to R2R we should allow this capability for apps that want to build everything as a single universe.

[mjsabby]

@tannergooding does this issue capture adding a command line argument to crossgen to capture a target ISA and propagate that to RyuJIT such that is generates code utilizing that or is it specific to the Hardware Intrinsic API work?

[tannergooding]

@mjsabby, this issue does not capture that and I do not believe it is specific to the HWIntrinsic work (although, outside of Avx/Avx2, HWIntrinsics may be the only thing to take advantage of such a feature).

[mjsabby]

Ok, so maybe I do need to open an issue to allow crossgen to specify target ISA so that it thinks it is generating code for processors with certain features. My impetus is to see that R2R images have the best codegen possible.

[CarolEidt]

I think there's a bit to sort out before we determine our long-term approach. Possibilities include:

• As @tannergooding indicates here: https://github.com/dotnet/coreclr/pull/24689#issuecomment-497357028, we could compile all methods assuming only SSE2 is supported. This would mean that all methods would have to include SSE2-only paths, as well as the appropriate IsSupported checks - but that's already the case for SPC.dll
• Support for all platform intrinsics during R2R, as long as they are protected by an IsSupported call (that would be deferred to evaluation at runtime if it is beyond SSE2. This would enable multi-path implementations, but would always incur the cost of the call.
• Orthogonal to the above, options could be added to crossgen that would set the target platform. Ideally this would only be done with additional support for validating the target platform when the assembly is loaded.
• Recognition of all SIMD types during R2R. This will be necessary once we address dotnet/runtime#9578, and is captured in dotnet/runtime#10152.

[tannergooding]

Is there any possibility of allowing R2R generation for methods which contain an SSE/SSE2 or Software Fallback path and otherwise requiring they be JIT'd. Additionally, the hardware intrinsics themselves should never be R2R generated.

Ultimately this would look to solve:

• For common scenarios (methods which have a baseline IsSupported check or which only contain baseline intrinsics), the method can be AOT compiled. Any non-baseline codepaths can be allowed later via tiered compilation if the method is considered "hot"
• For methods which are only meant to be called from an IsSupported scenario (such as MyMethod_Avx2()), they will not be R2R generated and jitting will be required (which will only impact startup if invoked during startup).
• For hardware intrinsics themselves (which is impactful for reflection scenarios and jitting IsSupported checks/etc) the JIT replacing them in importation is the best case scenario so we don't want them to be R2R (only methods consuming them, where the intrinsics are inlined should be R2R).

[mjsabby]

Thanks @CarolEidt.

Your third option is most appealing to me, because it unblocks those want to compile their own SPC with AVX2. Of course, someone could compile their own crossgen if they're going to make their own SPC, but the bar becomes a bit lower if we've done some level of testing.

To that end, I'm inquiring if the team at large (you and others) know if turning on AVX2 is as simple as setting the ISA or there is more work involved. And of course the answer could be it is unknown.

[CarolEidt]

To that end, I'm inquiring if the team at large (you and others) know if turning on AVX2 is as simple as setting the ISA or there is more work involved. And of course the answer could be it is unknown.

I'm not sure. If you just wanted to support this in your own build, I would look at what @MichalStrehovsky did in dotnet/coreclr#24689, and extending it to allow AVX2. However, I don't see us doing this in master without some non-trivial support for checking platform compatibility at runtime.

[MichalStrehovsky]

@CarolEidt the second option with generating all intrinsics, but making IsSupported always do a call is pretty intriguing. We can probably only do this for SPC (because code that doesn't guard intrinsic use with IsSupported would lead to invalid instruction traps instead of a managed exception - which would be a user experience regression that is probably acceptable only in SPC), but it would let us crossgen the methods that are on the critical startup path in a safe manner.

I think I found a pretty good spot to do this and measurements seem to indicate the cost of not inlining is negligible in practice.

See dotnet/coreclr#24917.