Cxbx-reloaded: Port to Linux using Wine

[PatrickvL]

Cxbx might be ported over to Linux, by using either a dependancy on Wine or by linking to Wine libraries.
This way Cxbx doesn't need to implement many API's again for Linux, but could "just" rely on Wine to handle most of the Xbox kernel API's.

As long as not-available and/or problematic Windows API's are avoided, it might even be possible to launch the windows-targeted Cxbx executable as-is using Wine on Linux.

[Sunderland93]

I preffer native port

[escape209]

@Sunderland93 I'm sure everyone prefers a native port. Unfortunately, it's just not nearly as easy.

[PatrickvL]

@Sunderland93 I'd say, submit a pull request offering native implementations for all Xbox Kernel API's ;)

[LukeUsher]

Has anyone tried running the current POC release on Wine? If not I may have to get my old laptop out this weekend

[Djhg2000]

@LukeUsher I tried running it with WINE 1.9.21 but I just got "wine: Invalid address." and then it returns me to the shell. It seems to be the same issue mentioned in WINEHQ bug #38432: https://www.winehq.org/pipermail/wine-bugs/2015-April/411936.html

Some "genius" prelinked the executable to 0x10000 and made the image
non-relocatable (code in 'newcode' text section assumes a fixed address space
layout/mapping).

A bit harsh but maybe a non-relocatable executable is required to get the emulation working at all? I'm not familiar enough with the inner workings of Cxbx to tell whether that's actually the case though.

Edit:
At least I found where this is set in the project files: https://github.com/LukeUsher/Cxbx-Reloaded/blob/master/build/win32/Cxbx.vcxproj#L98-L99
Does the emulator still run on Windows if it's built with <FixedBaseAddress>false</FixedBaseAddress>?

Edit2:
I may have spoken too soon, line 145 and 146 in that same file looks identical to lines 98 and 99.

[LukeUsher]

Fixed base address is definitely required, Xbox executables are not relocatable and have to be loaded at the 0x10000 base address in order to function. As we execute the Xbox code directly on the host CPU, we need to reserve that region of memory and load the XBE file there.

On Windows, the only way to allocate memory at that region is to create an executable with that base address, as attempting to use VirtualAlloc() to allocate memory in that region always fails.

[rcaridade145]

There was back on the day effort to port Cxbx to Linux . Here you can find a thread on emuforums on it emuforums .

edit : https://github.com/Echelon9/cxbx-shogun/commits/wine The link on that page was broken.

All the best.

[Djhg2000]

@LukeUsher That makes sense.

Reading up on it it seems the problem is WINE reserves the 0x10000 to 0x110000 region for DOS stuff ( http://source.winehq.org/git/wine.git/blob/34b2d920b47122007b65d435e064d018fb37b21f:/dlls/ntdll/virtual.c#l117 ), probably because almost all software is relocatable and so there was originally no reason to change it. The problem is that #ifdef on line 118; 64 bit WINE (which I'm using) should not have that address space reserved in the first place.

[some time elapses]

Here's some debug output from WINE after digging some more:

0028:trace:module:load_builtin_callback loaded ntdll.dll 0x8060158 0x7bc10000
0028:trace:relay:load_list L"RelayExclude" = L"ntdll.RtlEnterCriticalSection;ntdll.RtlLeaveCriticalSection;kernel32.48;kernel32.49;kernel32.94;kernel32.95;kernel32.96;kernel32.97;kernel32.98;kernel32.TlsGetValue;kernel32.TlsSetValue;kernel32.FlsGetValue;kernel32.FlsSetValue;kernel32.SetLastError"
0028:trace:relay:load_list L"RelayFromExclude" = L"winex11.drv;winemac.drv;user32;gdi32;advapi32;kernel32"
0028:trace:module:load_builtin_dll Trying built-in L"kernel32.dll"
0028:trace:module:load_dll looking for L"ntdll.dll" in L""
0028:trace:module:load_dll Found L"ntdll.dll" for L"ntdll.dll" at 0x7bc10000, count=2
0028:trace:module:load_builtin_callback loaded KERNEL32.dll 0x8062b30 0x7b410000
0028:Call KERNEL32.__wine_kernel_init() ret=7bc5b9c3
0028:trace:module:LdrGetDllHandle L"kernel32" -> 0x7b410000 (load path (null))
0028:trace:module:load_dll looking for L"E:\djhg2000\install\win\Cxbx-Reloaded v0.0.1-POC\Cxbx.exe" in L".;C:\windows\system32;C:\windows\system;C:\windows;C:\windows\system32;C:\windows;C:\windows\system32\wbem"
0028:trace:module:get_redirect looking up redirection for L"E:\djhg2000\install\win\Cxbx-Reloaded v0.0.1-POC\Cxbx.exe"
0028:trace:module:get_redirect no redirection found for L"E:\djhg2000\install\win\Cxbx-Reloaded v0.0.1-POC\Cxbx.exe"
0028:trace:module:get_load_order looking up loadorder for L"E:\djhg2000\install\win\Cxbx-Reloaded v0.0.1-POC\Cxbx.exe"
0028:trace:module:get_load_order got main exe default n,b for L"E:\djhg2000\install\win\Cxbx-Reloaded v0.0.1-POC\Cxbx.exe"
0028:trace:module:load_native_dll Trying native dll L"E:\djhg2000\install\win\Cxbx-Reloaded v0.0.1-POC\Cxbx.exe"
0028:trace:module:map_image mapped PE file at 0x8180000-0x101c9000
0028:trace:module:map_image mapping section .text at 0x8181000 off 400 size 18e00 virt 18d6a flags 60000020
0028:trace:module:map_image clearing 0x8199e00 - 0x819a000
0028:trace:module:map_image mapping section .cxbxplg at 0x819a000 off 19200 size 200 virt 35 flags c0000020
0028:trace:module:map_image clearing 0x819a200 - 0x819b000
0028:trace:module:map_image mapping section .rdata at 0x819b000 off 19400 size 9600 virt 946e flags 40000040
0028:trace:module:map_image clearing 0x81a4600 - 0x81a5000
0028:trace:module:map_image mapping section .data at 0x81a5000 off 22a00 size 600 virt 8002484 flags c0000040
0028:trace:module:map_image clearing 0x81a5600 - 0x81a6000
0028:trace:module:map_image mapping section .gfids at 0x101a8000 off 23000 size 400 virt 31c flags 40000040
0028:trace:module:map_image clearing 0x101a8400 - 0x101a9000
0028:trace:module:map_image mapping section .rsrc at 0x101a9000 off 23400 size 1f200 virt 1f1c8 flags 40000040
0028:trace:module:map_image clearing 0x101c8200 - 0x101c9000
0028:warn:module:perform_relocations Need to relocate module from 0x10000 to 0x8180000, but there are no relocation records
0028:warn:module:load_dll Failed to load module L"E:\djhg2000\install\win\Cxbx-Reloaded v0.0.1-POC\Cxbx.exe"; status=c0000018
wine: Invalid address.
0028:trace:module:LdrShutdownProcess ()

It looks to me like WINE is trying to load the executable into 0x8180000-0x101c9000 but only after doing so does it realize the binary isn't relocatable and gives up. It's as if the logic to handle non-relocatable executables is completely unaware that my amd64 machine _probably_ has MMU support (lol). I'll do some more digging and see what I can find.

[PatrickvL]

Here's an idea :

What if we assume the OS is 64 bit, somehow run Cxbx in 32 mode, and offset all allocated memory with 4GB (0x0000:1000 would become 0x1:0000:1000, etc.) - would that work ? ?

[LukeUsher]

Cxbx-Reloaded is already broken on 32-bit systems (unless PAE is enabled) due to allocating memory for a fake kernel header at 0x80010000, so that part isn't a problem for us.

That could work, it's definitely something to try at least.

[PatrickvL]

Blueshogun investigated the possiblity of a port to Mac OS - read part one here : http://shogun3d-cxbx.blogspot.com/2017/01/the-macos-experiment-part-1.html

[slugdude]

I took the WINE 2.0 source and replaced all references to the size of the DOS reserved area to 0xffff instead, compiled, and we get further!
I mean, uh, we don't get it working but this is better than instant rejection. Here's what I got on Ubuntu 16.04 64bit: http://imgur.com/XmDoe9I.jpg

After talking on #winehacking on freenode, it seems that the DOS reserved area is used for compatibility with Win16 apps so Cxbx-reloaded shouldn't be affected

[PatrickvL]

Okay, that's at least something. Just add any further discoveries to this issue, to keep things centralized.

BTW : Because of Cxbx using native code execution, we really must be able to load to 0x00010000, so a port to Linux support will have to deal with that, and judging by you're discoveries, probably requires a modified version of Wine.

[ghost]

I did most of the work here back in 2012, with forked blueshogun's branch:
https://github.com/Echelon9/cxbx-shogun/commits/wine

I didn't get much further than that, due to the FS register.

loader.c is where you want to look for getting a native port to Linux:
https://github.com/Echelon9/cxbx-shogun/blob/wine/build/wine/loader.c

The loader is always compiled with gcc -m32. You cannot use 64-bit addressing, which is unnecessary anyway. All Xbox code is 32-bit.

@PatrickvL
@Sunderland93 I'd say, submit a pull request offering native implementations for all Xbox Kernel API's ;)

This is a much better solution than having Wine as a dependency to Cxbx. All kernel thunk ordinal addresses can actually go to a native POSIX implementation in Linux, and loader.c can be used on its own. blueshogun's MacOS port is actually the same step in the right direction.

@slugdude
I took the WINE 2.0 source and replaced all references to the size of the DOS reserved area to 0xffff instead, compiled, and we get further!
...
After talking on #winehacking on freenode, it seems that the DOS reserved area is used for compatibility with Win16 apps so Cxbx-reloaded shouldn't be affected

Yep. I did run into this issue with Wine. You'll need to undo both maps that Wine does below:

/* 0x010000-0x110000 was dos area by reserve_dos_area() */
/* 0x232000-0x330000 was wine thread/signal stack by virtual_alloc_thread_stack() */

[PatrickvL]

I wasn't aware of that branch - thanks for mentioning it, it seems you've solved the biggest issues. If that would be combined with the FS patching approach Luke applied in Cxbx Reloaded, it might actually work!

[CakeLancelot]

Update on this: the new Wine development 2.18 update includes a fix for Cxbx(Reloaded). It now starts and runs homebrew! I haven't had much success with retail games but my driver is probably limited in 3d functionality (intel gma) so it would be better to test this on a more powerful computer perhaps.

[Djhg2000]

@CakeLancelot Fantastic news, I'll give it a try as soon as I have the time!

[RadWolfie]

Now we don't have to port it over to Linux? 😉

Still, I do would like to support Linux & Mac OSes if possible using cross-platform of openGL and openAL(?). With xbox's Kernel API base on Windows API usage, I don't think it's possible and take more time to adapt it in order to use Linux and Mac's Kernel API? Food for a thought 🥘

[slugdude]

Someone actually went in to the WINE code and changed this for us: if you
read the current ANNOUNCE file in the WINE mirror on Github (
https://github.com/wine-mirror/wine/blob/master/ANNOUNCE ) you can find
this under "Bugs fixed in 2.18":

42715 Cxbx (Xbox emulator) doesn`t start (wine: Invalid address)

Then later you can find:

ntdll: Support loading binaries that start inside the DOS area.

Honestly I wasn't expecting this, but it's great news. Will try messing
around with the new version soon.

On 6 Oct 2017 17:38, "RadWolfie" [e-mail] wrote:

Now we don't have to port it over to Linux? 😉

Still, I do would like to support Linux & Mac OSes if possible using
cross-platform of openGL and openAL(?). With xbox's Kernel API base on
Windows API usage, I don't think it's possible and take more time to adapt
it in order to use Linux and Mac's Kernel API? Food for a thought 🥘

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Cxbx-Reloaded/Cxbx-Reloaded/issues/11#issuecomment-334807204,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJupANvuOtAEdDcdC94FN_ux-hnYRV24ks5spleTgaJpZM4KX-ec
.

[LukeUsher]

It wasn't just anybody, according to the log it was Alexandre Julliard: The project leader for Wine.
Pretty cool stuff!

[Sarkie]

Amazing news!

https://bugs.winehq.org/show_bug.cgi?id=42715

[PatrickvL]

Amazing indeed.

This issue can be closed now, although it would be good to document in our readme that we can now also run on Linux using Wine.

[mrdeathjr28]

Is a good news however in some games shows dinput cant initialize case sega gt 2002, forza motorsports, dead or alive xtreme beach volleyball, ninja gaiden

[LukeUsher]

It's definitely not going to be perfect under wine, we still have major issues even under Windows. That said, this is pretty good for us. I'm sure we can improve functionality under wine in the future. I may even dual boot so I can test it with every change

[Serentty]

It might be useful to allow building using Winelib, since by producing an ELF instead of an EXE (although still depending on Wine being installed), it makes it easier to have it installed normally by a package manager instead of in a Wine virtual drive. This also means that the package can list Wine as a dependency so that it is automatically installed when trying to install Cxbx-Reloaded.

[PatrickvL]

@Serentty since this issue is closed, please create a new issue with your suggestions, otherwise they'd only get lost

[LukeUsher]

@Serentty we've looked into it, the problem is that Cxbx-Reloaded can not be compiled with any compiler that isn't Microsoft Visual C++.

There's effort being taken to allow compilation with GNU based compilers, but doing so requires a large rewrite of significant portions of the codebase.

Much of the calling convention handling and assembly language parts need completely replacing, since inline assembly using GNU and Microsoft compilers has a different syntax.