Runtime: x64 app with references to x86 dlls

[ghost]

I send this in Roslyn repo, but comments say it maybe belong here:
https://github.com/dotnet/roslyn/issues/33294

x64 apps that references x86 dlls
VS.NET 2019 Preview 2.2
Win10 x64

Why can't Roslyn run x634 apps that have some x86 components?
To demonstrate that, I created a simple console app (compiled as X64), and added two trivial .net standard library projects to it, one compiled as x64, and the other as x86. When I run the app, it breaks with this message:

Your app has entered a break state, but no code is currently executing that is supported by the selected debug engine (e.g. only native runtime code is executing).

and the exception window shows:

System.BadImageFormatException: 'Could not load file or assembly 'ClassLibrary2, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. An attempt was made to load a program with an incorrect format.'

I don't understand why does this happen? Window 64bit it self can run 32bit apps! Why don't Roslyn have some bridge to run 32 bit dlls on 64bit aps? Even more: why can't it run 64 bit dlls on 32bit aps? these dlls are not true assembly code, and they are compiled by JIT compiler, so, is it possible to do something in this stage to bridge the gab?

Otherwise, It will be better to give the programmer a hint from the begging that mixing different platforms in the same app will not work.

If I am the owner of the dll source code, sure I can fix such problems, but there are legacy .net dlls which are 32 bits, and they will force us to compile our new apps as x86 if we want to use them! This becomes worst if you have an x64 dll and an x86 dll that you don't have thier source codes and want to use them in same project!
So, I think Roslyn should find a way to solve such problems. JIT can compile x86 to x64 native code if the main app is x64, and vice versa. If there is an unmanaged component, then VS.NET must force all app components to target the same platephorm as that unmanaged component.
I don't have enough details about the differnce between x86 and x64 IL codes, but if the difference is just some header format or attribute value, then I think solving this incompatabily should be easy.

x64 process cannot run x86 code or dll

Can't Roslyn create a temp process to run the x86 dll and make it communicat with the x64 process? In other words: Wrap the dll in a siutable process?
I don't know if this can be done or be practical. I have no enough knowlage about this, but I faced such a prblem in practice once, and I had to compile to x86 to solve it. This can be impossible if I deal with both x86 and x64 dlls in same project and have not theie source codes.

[MeikTranel]

You would have to share heap and stack memory with another process according to your plans. Even if they were able to pump out glue code to marshal everything right between process, they would still have to break every memory safety model ever. I don't think this will ever change. If you need to do both, you can do this yourself by extracting the component into a separate process, but orchestrating this from runtime standpoint isn't gonna happen anytime soon.

[ghost]

@MeikTranel
Thanks. But there is another suggestion:

JIT can compile x86 to x64 native code if the main app is x64, and vice versa.
I don't have enough details about the differnce between x86 and x64 IL codes, but if the difference is just some header format or attribute value, then I think solving this incompatabily should be easy at the evel of JIT.

[jkotas]

the difference is just some header format or attribute value

If the legacy .dll is a portable IL and it was compiled for x86-only by mistake, you can use corflags tool from .NET Framework SDK to clear the bit that says the .dll is x86-only:

corflags /32BITREQ- <filename>

If the incompatibility goes beyond this, there is no good way to solve this as @MeikTranel explained.

[ghost]

@jkotas
Thanks for the info. It is helpful.
So if i got it correctly, the difference between x64 and x86 managed assemblies is just some flag?
Then why can't JIT auto fix this if it can?

[jkotas]

the difference between x64 and x86 managed assemblies is just some flag

The flag says that the assembly is meant to run on x86 only, for example the code in it can have dependencies . The JIT cannot tell whether the flag was set by mistake and the code will run on non-x86 just fine. If we started ignoring this flag, we would get hard to diagnose crashes instead for these cases where the code does not work at all on non-x86.

[MeikTranel]

@MohammadHamdyGhanem As @jkotas said, if it really is just a result of sloppy build configs, you could just recompile for AnyCPU or just modify the corflags, which honestly sounds more complicated than it is.

But the key to your request is another. You mentioned unmanaged components... this is where it gets hairy. Many x86 or x64 compiled .NET assemblies are like this because of a very specific reason - often times it involves dependencies to other components down the line. Sometimes a DLL uses imports of native (or "unmanaged") components to act as managed wrapper orchestrating PInvoke-Calls to native APIs like SQLite or LibGit... sometimes it goes as far as COM+ communication. You could compile these wrappers easily to AnyCPU (or MSIL (or now called CIL)) with some warnings. But that doesn't help really... it call comes down to your native dependency. If it's an amd64-DLL, the runtime or the compiler ( @jkotas i don't actually know where this happens.. help 😄 ) has to make sure that the PInvoke is marshalled in a way that it fits what the native dependency expects.

 [DllImport ("gdi32.dll", CharSet=CharSet.Auto, CallingConvention=CallingConvention.StdCall)]
 private static extern bool TextOut (IntPtr hdc,int nXStart,int nYStart,string lpString,int cbString);

GDI32.dll is compiled for x86, x64 and most recently also arm32 (not sure about the specifics here), which means that the IntPtr for the window handle in the first argument has to be spot on for everything to work. You can't push a 64-bit wide pointer into PInvoke entrypoint that expects a 32-bit wide pointer. So at a certain point in the lifecycle of the DLLImport someone has to make sure that this thing fits. How does it know what fits and what doesn't.
Maybe there are ways to query the endpoint of the imported library, but that goes beyond my knowledge. But its entirely besides the point. In x64 your entire program is tuned to produce 64-bit pointers everywhere. You can't just cast a pointer down... that wouldnt make sense. So tweaking the DLL to orchestrate exit points and entrypoints to have the same bitness at runtime will never work. Either your process has the right bitness or it doesn't. Either it hits a Managed DLL with a incompatible bitness in the corflags or it tries to PInvoke into a native DLL that doesn't provide the same bitness as the processor archictecture that the native library was compiled against.
JIT will not help your here. JIT doesn't decide whether to compile something against amd64, itanium, arm32 or something else. The process does.

@jkotas Don't shoot me if i presented you the oversimplification of the year 😄

[jkotas]

@MeikTranel Your more detailed explanation of the problem is spot on. Thank you for writing it!

[ghost]

@MeikTranel
Thanks. You nailed it :)
I have a question and hope it is not out of the scope.
What is the benefit of using x64 in .NET app?
The only reason I know, is that x64 app can use large memory (with the disadvantage of references and pointers consuming twice the memory as consumed by x84). I think apps that needs that amount or RAM is rare, maybe except for services running on servers.
My question is: Is windows 64 bit is a true 64 bit operating system on low level, or it is just a way for addressing more RAM?
Do commands run faster on x64, making use of wide registers and saving half cycles? I did some experiments on win 64 and I think the answer is no. Am I right?
This can affect how we choose to compile our apps, because x68 can consume less ram and run faster on win 64, or at least doesn't run slower than x64!
So, do you advice to compile to x64 by default? Or is it better to stick with x86 unless needing more RAM?
Thanks in advance.

[jkotas]

You are right that small applications tend to run faster and consume less memory when run as 32-bit. If you care enough about that delta, you can still run your apps as 32-bit.

The world as a whole is on the path to give up on 32-bit because it comes with extra complexity that offsets the performance improvements. E.g. Apple is dropping 32-bit support, Linux distros are dropping x86 support, ... . The 32-bit is going to have its place for low end IoT devices, but I expect that everything new running on more powerful machines (e.g. cloud, higher end phones) is going to be 64-bit in not so distant future.

[ghost]

@jkotas
I agree with you. Also SIMD enabled types need x64 system and app.
But ending support for 32 bit will kill millions of legacy apps that still used today, and will force users to stick with old OSs at least running on VMs. This is the case with COM components and old managed DLLs that we use in our apps, and will keep using them for a while, and they the reason I opened this issue in first place.
I'm still asking about Win 10 (x64): Is it a true 64 bit operating system that can run apps twice faster(theoretically), sending data to registers in 64 bit packets instead of 32 bit ones? I lived the period when 32 bit windows emerged and each new version MS said that "This is the true 32 bit windows ever!". Is this the case here, or did MS re-wrote Win 64 from scratch to make the most of the the 64 hardware?
I'm sorry if I ask you about something that maybe out of your area, but this concerns us as programmers.
Thanks.

[jkotas]

Win 10 (x64)

Yes, Win10 x64 is a true 64 bit operating system. The 32-bit support is emulated by https://en.wikipedia.org/wiki/Windows_on_Windows on top of it. I expect 32-bit support is going to be present in the classic desktop Windows editions forever (ie for the next 20 years at least). It is not going anywhere due to the sheer number of Windows 32-bit apps that exist today as you have explained.