Cxbx-reloaded: Redesign Xbe loading

[PatrickvL]

Cxbx currently contains a class that loads an Xbe file into memory (but not in the address range where it should be loaded during emulation).

This class was build to be able to read and write Xbe files, and to write Windows executable (.exe) files.

Nowadays, we don't generate .exe files anymore, and saving modified Xbe files is strongly discouraged.
Most of the use of this Xbe class, is in the GUI environment.

When emulating an Xbe, it's loaded into memory only partially via the Xbe class, partially via specific code.

Instead of all the above, loading an Xbe should be done via the kernel API meant for this task : XeLoadSecton and XeUnloadSection.

Another reason to redesign Xbe loading, is because some Xbe actually crash Cxbx.
One such case is Quantum Redshift\XDemos\NFLFever2003\NFLFeverTrial.xbe which crashes Cxbx because of uninitialized sections.

So for this issue:

• Build an overview of where Xbe files are accessed
• Define which parts of the Xbe are actually needed per use case
• Implement the above mentioned kernel API's
• Use the kernel to load and unload xbe's properly
• Update all GUI code to keep functioning properly
• Remove the Xbe write-back code

[PatrickvL]

Update: Xbe sections are now loaded via XeLoadSection, but still not under control of the kernel. Also, the Xbe class is still used to load the Xbe image header, the certificate and other parts.
Instead, the kernel should load an Xbe without the Xbe class, and retrieve the Xbe path via the LaunchDataPage.

[Fisherman166]

I was doing some digging around on this issue yesterday and beginning to write up a text document on a possible solution. If I understand correctly, the Xbe class would still exist, but would only be used for the GUI?Which would require the kernel functions to parse the Xbe in its own way to provide Xbe loading functionality to the kernel.

[LukeUsher]

All the real Xbox kernel does is store a file handle to the xbe, as well as a copy of the header and certificate in memory, then whenever a section is loaded, it simply reads from the file handle into memory and increments the reference counts, it's very simple.

The reason we store a full copy of the xbe in memory is patching: applying HLE patching at section load time could cause rather large delays/hiccups in execution, and could cause issues if xrefs cross section boundaries. Additionally, the intercepted functions that EmuFS and HLEIntercept jump to can change per build of Cxbx-Reloaded (even if the code didn't change), so caching the patched binary data isn't really viable either, unless we come up with our own type of relocation table.

Even so, the relocation step takes time. This is something better left for when the HLE side of things is no longer required for hardware.

[x1nixmzeng]

The Dxbx solution is called out in the CxbxKrnl source - and all that does is read the file and copy the raw file data into memory - see:

https://github.com/PatrickvL/Dxbx/blob/bcafc5b5e022c8a884241731607f99731ef3e7ee/Source/Delphi/src/uEmuExe.pas#L251-L257

    // Copy the complete XBEHeader over to the ImageBase :
    _ReadXbeBlock(
      {RawOffset=}0,
      {RawSize=}XBE_HEADER_SIZE, // =$1000
      {VirtualAddr=}XBE_IMAGE_BASE,
      {VirtualSize=}XBE_HEADER_SIZE,
      {NewProtect}PAGE_READWRITE);

My questions:

1. What is the use of XeLoadSection if it is always in memory?
2. When an Xbe is switched - does this address get replaced with the new XBE?
3. Does a soft-reset keep the XBE handle in memory?

[ghost]

What is the use of XeLoadSection if it is always in memory?

The implementation of XBE is in XAPI. The microkernel loads these sections via this interface.

These routines are equivalent to mmap() and munmap(). You could stub these routines if you load all sections at once on XBE load.

When an Xbe is switched - does this address get replaced with the new XBE?

The microkernel keeping track of the current XBE header will have new values replaced from the new loaded XBE, including entry point address, and kernel thunk table address.

Does a soft-reset keep the XBE handle in memory?

A soft-reset boots to dashboard. It's also In Game Reset.

[LukeUsher]

Ah section loading: a few points.

The Xbox is a single process system, only one xbe can be loaded in a session, the system reboots when loading a different xbe, and only memory marked with MmPersistContiguousMemory remains, everything else is discarded. This is usually used to pass the launch data page.

Now for Xeload/unload section: on real hardware, the xbe is not loaded to RAM in full, we only do that because we need to apply patches and do symbol scanning across an entire xbe.

On real hardware, the kernel initially loads only the xbe headers, the certificate and the section headers, after that, it calls XeloadSection to load only the sections that have the preload bit set in the header, everything else is ignored.

The running application can than load/unload sections on demand. This is useful if you have data within the Xbe that wastes ram if allocated all the time, but is needed in certain scenes.

It's also possible to have xbes at almost unlimited size, you could have a 1gb xbe and everything would work fine as long as you never have more data loaded at once than can fit into system RAM.

It's also possible to have multiple sections have the same virtual address in memory, as long as you only have them loaded one at a time. This could be useful in an RPG for example: the battle system code could replace the map code when a battle starts, and code that isn't needed right now wouldn't have to be in memory anymore. Tricks like that were common in the PS1 days but are still possible on the Xbox.

And that is why XeloadSection and XeUnloadSection exist/how they are used

[LukeUsher]

Closing this as we now use XeLoadSection/XeUnloadSection to load executables as a real Xbox would.