Cxbx-reloaded: XISO (XDVDFS) support

[PatrickvL]

Dxbx contains a virtual file system, that's able to load Xbe's from either local file system or from an Xbox DVD file-system ISO. Port that back to Cxbx.

See https://github.com/PatrickvL/Dxbx/blob/master/Source/Delphi/src/uXDVDFS.pas
and https://github.com/PatrickvL/Dxbx/blob/master/Source/Delphi/src/uFileSystem.pas .

Alas, Dxbx doesn't use this virtual filesystem for it's file-related kernel functions, so apart from the loading of an Xbe, all other file accesses are still entirely done through the local file system.

So, once we've back-ported the virtual file system code of Dxbx, start using it in our file-related kernel functions too, in order to gain complete XISO support.

[x1nixmzeng]

This is a great idea, we can easily support loading XBEs now we're executing them in-memory.

Aside from porting the Dxbx virtual filesystem work, how feasible is it to redirect file io calls yet? Surely we need to wait for some of our own kernel implementations.

[PatrickvL]

Our kernel currently forwards all file related API calls to NtDll, but that is just a temporary solution. Eventually we must implement our own file system handling code anyway, so why not use a virtual filesystem right from the start?!

As an alternative to porting the Dxbx vfs code, we could import a library, ttvfs (here on GitHub https://github.com/fgenesis/ttvfs) for example could be one candidate, although despite it's name is not really all that tiny. The Dxbx vfs is much smaller.

Regardless which vfs we choose, the XDVDFS code has to be back ported anyway, in order to support reading from xbox iso files, as one type of vfs.

Btw: the XDVDFS implementation as present in Dxbx, is a port of an unspecified version of XDVDFS Tools, written in c, dating back to 2006. We don't know if this is the most recent version. There seems to exist a v2.1, from someone called CloneXB, but it's nowhere to be found.

[BiatuAutMiahn]

What about this?

[PatrickvL]

Hey, good finds! In hindsight I had that repository starred already. Let's compare that version with the Dxbx code - perhaps Dxbx contains some fixes that might need porting.

[PatrickvL]

I diff'ed them - the Dxbx version indeed fixes two corner cases and uses named constants instead of literal integer values. Otherwise they're identical, which means we can start out with that version, and just have to port back the few Dxbx fixes.

[LukeUsher]

Xbox discs consist of two areas, the first being a standard ISO filesystem, the second being a 'secure area' which contains the XDVD file system.

Most XISO tools/dumps contain only the XDVD file system, and work with all extraction tools, however any 1:1 dump of an original Xbox disc (and also the Redump set of ISOs) contain both partitions and most extraction tools fail. I've had some success using Xbox 360 tools such as XDVDMulleter to extract these types.

We need to make sure both types are supported, the first because it is most common, and full 1:1 dumps because they are what you get when you 1:1 dump a full Xbox disc using a compatible drive.

[BiatuAutMiahn]

If I'm not mistaken, the index of the XDVDFS is on the outer rim of the disc, no?

[LukeUsher]

Another resource: A FUSE driver for accessing XDVD filesystems.
https://github.com/multimediamike/xbfuse

Looks like this supports the raw 1:1 image dumps and works for both Xbox and Xbox 360 ISOs.

[PatrickvL]

Reading xbfuse isn't all that different from XDVDFS.

The main difference between xbfuse and XDVDFS is, that XDVDFS expects an ISO to directly start with a MICROSOFT*XBOX*MEDIA signature, and xbfuse scans for it.
Compare https://github.com/multimediamike/xbfuse/blob/master/src/xdvdfs.c#L244 , with https://github.com/Rocky5/Xbox-Softmodding-Tool/blob/master/Tools/XDVDFS%20Tools/src/xdvdfs/xdvdfs.c#L56

[LukeUsher]

Looks like if we scan for the header rather than looking for it at the start of the image, we automatically support both types. That's good to know!

[PatrickvL]

Indeed. Step 1 is now available as PR #210 - this introduces fundamental sector reading and parsing.
Following this, we could port the Dxbx file system code.

[PatrickvL]

Reading https://www.reddit.com/r/emulation/comments/7o4h58/quick_compression_method_comparison/?utm_source=reddit-android , CHD files seem like a potentially useable format for this. https://github.com/rtissera/libchdr

[PatrickvL]

@JayFoxRox downvoted the CHD format, and had this to say about it :

I think MAME CHDs are also a bad choice
having worked with CHDs before, they come in various flavours and they add a rather big dependency. They were originally made to hold rom collections but they also contain disc images now. while this might lead to Xbox CHDs in the future, right now, only the Chihiro FATX images are CHDs, but even those might be broken
CHDs are also a pita to pack / create because you need MAME [or have to add a packer to your emu]
and last but not least: xbox games do not compress properly - there is no point in doing this unless you write a xbox specific compressor which strips metadata or random data (which is hard to compress, but "easy" to generate)
I think a lot of time is wasted on this "compressed games" discussion when it's a super niche feature which is also very impractical. the people proposing it put no thought into it. the discussion about it should have ended already imo (albeit nicer than how espes ended it)

[PatrickvL]

For most, useability will probably be their main criteria
This requires having easy to use and fast tools to dump their media
Preferably without the need for any further conversion, getting a single output file that can readily be used anywhere, and is not too big

[PatrickvL]

@LukeUsher wrote :

On softmodded consoles you just extract the iso to raw filesystem dumps (that we already run in Cxbx-Reloaded) and then the xbes get patched to remove anti piracy checks and region checks, that's pretty much all their is to it

[PatrickvL]

As for CHD, the recent libchdr project on github offers an independent implementation for all five CHD versions currently in existence, with only limited dependencies
@LukeUsher Ah, okay thanks. (Like I said, I never owned an Xbox)
@JayFoxRox I share your concern about suboptimal compression. (Do you have any resources on how to re-generate the random data?) The downside to any Xbox-specific solution, is that by definition it becomes a niche format, which is a liability we'd rather avoid.

[PatrickvL]

@JayFoxRox replied :

I don't think I've uploaded my stuff anywhere yet. I'll probably continue with xbox disc preservation this month
but I've documented the algorithms on the wiki iirc
http://xboxdevwiki.net/XDVDFS#Random_blocks

[Papermanzero]

Is there any update on this one? I would like to use original dumps but without the extraction process. CHD v5 support would be great. But an alternative format (like rvz for dolphin) would be also appreciated