Cxbx-reloaded: Work towards portable codebase

[PatrickvL]

Cxbx-Reloaded is currently using Windows API's.
It's is build with Visual Studio.
It uses many code constructs of MSVC.
It's written in a (bad) mix of c and c++.
It targets x86 architecture (due to the direct code execution method).
It runs as a 32 bit process under a 64 bit OS.

In order to become portable, all these aspects need to be addressed.

Some possible solutions:

Platform: c++11 or 14 standards (?)
Build system: cmake
Compiler : gcc or LLVM clang
Architecture: Linux x86, Android x86
CPU emulating: unicorn, KVM, HAXM

It's probably wise to create separate issues for all aspects that can be done independently. Let's use this issue as a central collection of options and ideas.

[LukeUsher]

For C++ standards, we already target C++17 (We make use some functionality that is not available in earlier standards, so we have that covered already :P)

For a build system, I highly recommend CMake, not only because it is easy to use, but because it can export build scripts/project files for many different development environments and compilers. The same CMakefile can output build scripts for Linux Makefiles, Windows Makefiles (for mingw32) or even full MSVC Solutions/Project files, it's a great option.

For compiler, we should try not to use any compiler specific functionality, any commonly used compiler should be compatible (So gcc, clang and MSVC compilers should all work), this isn't too difficult, just make sure not to use compiler extensions, and if you must, #ifdef them out based on the compiler being used.

CPU emulation i wouldn't worry about just yet, Cross platform across Windows and Linux on X86 is a much easier target (and still requires a huge rework of the codebase to stop using the Windows API functionality directly), but is much less of a headache than porting to other architectures at this point.

We should keep it simple and take it in steps.

That said, the best approach for the future would likely to start a new, blank source code project, migrating Cxbx-Reloaded code into it and being cross platform from the get go, but such a thing is a mammoth task that I doubt everybody would be onboard with :P (Although this is what Cxbx-Reloaded was originally planned to be, it just got a little out of control!)

[Fisherman166]

Very rarely does the 'throw out the code base and restart' go well. A lot of time has to be spent fixing all of the new bugs introduced with the rewrite before the project returns to the same level of functionality. A more realistic goal is to make sure that whenever someone touches a piece of code, they leave it better than it started. Over time the code base will naturally improve through many small incremental updates instead of a massive throw it all out and restart methodology.

[LukeUsher]

The reason for the start fresh idea is because there are many fundamental design flaws with Cxbx-Reloaded that are extremely difficult to fix without almost a complete rewrite anyway: the core design needs to change in order to actually be an accurate emulator, It's not a code problem, it's design flaws in the original Cxbx architecture. I'd say more on the topic but once you work with the codebase for some time and become familiar with how the real Xbox does things, you realise how wrong the design is.

I'm not saying it cannot be improved / cleaned up over time, but the process of moving cross platform will be just as much effort as starting fresh. We have little to no portable/abstracted code, we call undocumented and unsupported windows APIs constantly, forward Xbox functions directly to Windows NT function, depend entirely on an ancient version of DirectX (8), annoying, not just the rendering code depends on this, but so does the GUI and many other aspects of functionality.

It's not something we can just do casually either: once we move some kernel functions away from calling the Windows API directly (like NtCreateFile) everything will break, because things like WaitForSingleObject get called on handles returned by that, so the functions we need to rewrite just keep cascading.

I've suggested this before and been hit with some resistance, but if we would end up with a long period of instability, and having to rewrite the majority of the code base, why not take the opportunity to look at the core design too?

[LukeUsher]

I apologise for being so negative, it's just rather frustrating when most of my time is spent working around design flaws, so that things can improve/develop further, rather than being able to work on improving the emulation as a priority.

Things that are known to still be very wrong include Timing, Threading, TLS, Interrupts, huge chunks of the kernel.

Some of these cannot be solved easily either. The issues with thread switching and timing cannot easily be solved while Direct code execution is used. Others have also had issues with timing using virtualization (lack of control over certain aspects that the Xbox relies on), while interrupts cannot be done reliably with this approach either: we actually run interrupts in a thread because we cannot actually interrupt running code without causing major instability, because of direct code execution.

Things are getting better thanks to recent developments like the amazing memory manager, but these still work around fundamental issues that should have never been a problem.

[ObiKKa]

I agree with much of Luke's last comment body. But in taking away the Direct Code Execution method wouldn't that make it harder to launch a .xbe file for a game or Dashboard? Eg. the fact that you'd need to pair a compatible BIOS file with a future CXBX-Spinoff emulator project just to launch the software?

Also, from the further above comments, I have had bad experiences with trying to manually compile the XQEMU emulator using the CMake files on my Windows 7. I don't want to do that again. I hope it'd all be automated, just download the ready-made files/archives and use it.

[LukeUsher]

XQEMU doesn't use CMake, just plain make, I believe. That said, CMake on Windows just spits our normal Visual Studio projects: You'd compile the same as already,except you'd run Cmake first before launching Visual Studio.

As for the kernel/bios: HLE is still possible without direct code execution, we can still implement our own HLE kernel, and we could still implement Direct3D/DirectSound/XAPI patching, there is no reason things like the EmuDisk and direct Xbe loading need to disappear, and no reason why we MUST require a bios.

[x1nixmzeng]

I have to agree with @Fisherman166 and argue that a fresh project would take a lot of upfront effort.

In retrospect there are always going to be better system designs and drip-feeding code quality improvements over time can help modernise the codebase.

Our ties to the Windows APIs are all temporary IMO - this whole project is an extended "proof of concept". In future we can always encapsulate functionality - for example, supporting XISO images would require something like a virtual filesystem which would need to provide IO calls and fill out Nt-like structures.

I'd love to see how CPU emulation would compare against direct code execution.

[Fisherman166]

Another wrench to throw into the mix: assuming a rewrite was chosen as the way forward, would the project want to stick with C++ or move to Rust? I haven't done a lot with my Rust myself but I know memory/thread safety is a lot better in Rust. If the code base is going to be rewritten anyway, it may be worth it to switch to Rust to help prevent the memory leaks that have been floating around in the issue tracker.

[ghost]

Cxbx-Reloaded is currently using Windows API's.
It uses many code constructs of MSVC.
It's written in a (bad) mix of c and c++.

I had the idea to implement all Windows API calls into standard POSIX calls. The codebase would be C.

It targets x86 architecture (due to the direct code execution method).

It can be made portable by emulating x86. The exception handler makes partial use of this, for MMIO.

It runs as a 32 bit process under a 64 bit OS.

This is for direct code execution. Running this as a 32-bit process will require 32-bit dependencies on a 64-bit host OS. When emulating the CPU, this wouldn't be an issue.

The x32 ABI would require its own dependencies too: https://en.wikipedia.org/wiki/X32_ABI

[x1nixmzeng]

Bit of a bump.

As the project stands - is it feasible to investigate an alternative to direct code execution? Specifically, would it be worth looking at CPU emulation (when a large part of the kernel is forwarded onto the Windows host)?

[ghost]

As the project stands - is it feasible to investigate an alternative to direct code execution?

Direct code execution is about as fast as the hardware.

A project that stands for CPU emulation is Xenia. It involves PowerPC and endian byte-swapping.