Retroarch: (Bounty) Real CD/DVD read support

[anothername99]

This bounty is about adding real CD/DVD read support to libretro/RetroArch. This would allow a user to for example play a game disc that is in the user's real CD/DVD drive. To complete the bounty:

• Add support to RetroArch and the libretro API for reading the contents of a real CD or DVD
• Add CD/DVD read support to a core
• Design your implementation so that it will be relatively easy for future core developers to add the feature to new cores

Bountysource link: https://www.bountysource.com/issues/63527883-bounty-real-cd-dvd-read-support

[markwkidd]

Thank you for getting the bounty rolling with a nice starting pot of $105!

[orbea]

Short of music CDs and video DVDs what consoles would this be practical for? I know for some consoles there are not any readily available drives that can both read the discs and do so at a playable speed.

[twinaphex]

@orbea It would be to make commercial products that are based entirely around being able to read the original game discs moot (without naming names now since I don't want to give them attention, but I'm sure you know what i mean), when every RetroArch device can potentially do this. This is why I consider this so important - by us not covering this, this right now leaves a gap in the market for some entrepreneur to fill, and with it would likely come some patents as well. It's important to nip that stuff in the bud as soon as possible. The best thing is to make this kind of stuff commonplace and as easy to access as drinking water from the tap, so that there is nothing exclusive to build some business model around anymore.

[orbea]

Good points all around, I'm just wondering what technical issues need to be tackled. :)

[anothername99]

It would also be useful for PC emulation. The ability to read and burn real discs (burn support is for a future bounty), in an emulated environment, would allow us to reuse a lot of existing software without having to install Windows on the host machine. This is also a motivation for the MIDI bounties, as well as future bounties for things like audio capture and drawing tablet support.

[twinaphex]

@anothername99 All good points. Let's just start with the low-hanging fruit first, reading of CD/DVD discs. We can expand the scope to writing or whatever in a future stage and in future bounties. First we need that foundation in place.

[nayslayer]

This issue seems to align perfectly with my personal backlog, so I'd like to work on it. As there are enough reference cross-platform disc reader implementations scattered around various emulators, I don't expect it to take too long either - I'll likely finish a working version sometime second half of December. Here I'll keep track of my thoughts and progress, as an invitation for discussion. I'll try to keep my posts scarce and meaningful, with finer progress reports once I make a PR.

Let's get started, shall we?

A disclaimer: for now I'm just throwing around ideas, some of those I won't implement anytime soon. The minimal valuable implementation for this issue will be much more grounded: a standalone C library implementing read-only raw disc access in a cross-platform way, with support for CD and DVD discs and Linux/OSX/Windows platforms, architected to also support disc images in the future; a libretro API to negotiate supported disc types, with Retroarch implementation; API support patches for 3 or more emulator cores. Do tell me if this isn't what you want.

Some background.

1. There are about two dozens of disc-based consoles (an incomplete list can be found on Wikipedia), and some consoles have two or more - oh, hello, PS1 - emulators. While most of them are compatible with generic CD's, some platforms have their own quirks like the very same PS1 requiring additional subchannel data for libcrypt-protected games to work properly, with the failure to provide it resulting in random freezes mid-playthrough. _A corrupt disc image or an unsupported CD drive can silently mess this up, so this data needs to be validated beforehand, maybe using a content database._ There are also some discs like UMD or Blu ray I don't know anything about, those will most likely require their own APIs.
2. There are quite a few CD image formats _found_ in romsets, both open and proprietary: iso and bin/cue, Alcohol 120% mdf/mds, Nero nrg, CloneCD bin/ccd, MAME chd, etc. The lack of image format compatibility is a very real problem, and it's not feasible to implement those individually for each core. _As there's no definitive disc image format, and some images originate from torrent-backed collections, you can't expect users to cater for individual emulator authors' whims. Therefore all of those formats need to be supported consistently._
3. There needs to be an app to make disc images that is designed with emulators in mind, otherwise there will be problems with non-standard data tracks being neglected, resulting in non-playable images. _Such an app will differ from a generic imager by detecting disc contents and picking settings automatically, and maybe validating image correctness by matching its hash against a content database. There is an emulator-minded tool already, thanks mirh for pointing this out. It needs to be repackaged into a GUI app with mentioned extras so it can be operated by non technical minded users, though._
4. Having both the VFS API and the CD/DVD API will allow experiments on creating a generic game archival format. _The CHD is only a part of the solution, as it isn't designed with ROMs, metadata, or compression speed/size tradeoffs in mind. This is something to look out for in the future anyway._

Some thoughts on implementation.

1. Some method to negotiate the support for this API will be required. Maybe an entry in core info files, or a special token in supported extensions list, or something like that, I'll look into how it's done later.
2. Once the API is estabilished to be supported, the core will be able to request a disc type identifier and a pointer to a method table specific to that type while providing a list of wanted disc types and required quirks, like DVD or CD|CDDA|LIBCRYPT. The latter will contain methods like ReadTOC/ReadSector/PlayCDDA, etc. Each disc type will have its own distinct method table so that new, possibly incompatible disc types may be added in the future.
3. Once the image file support is implemented, and this API's presence is determined, the cores will use it in favor of their native _image reader implementations that already exist, trusting the frontend to have better implementations of its own image reading quirks and caching/readahead strategies_. The former will remain maintained for compatibility with legacy frontends, if there are any.
4. Having the API implemented as a library will allow it to be baked into cores instead of their native implementations so that the compatibility will remain but a legacy CD access method won't need to be maintained.

This is all very preliminary, and I'll refine my ideas as I collect more info and delve deeper into the issue. Please feel free to voice your objections/suggestions/concerns.

Edited: Thanks mirh for pointing my mistakes. Changes are marked in italics.

[Colossus90]

Maybe you can contact some people at redump.
They know a lot about physical media on disc.
http://redump.org/

[mirh]

some platforms have their own quirks like the very same PS1 requiring additional subchannel data for libcrypt-protected games to work properly

Reading of subchannel shouldn't really be rocket science.
Putting aside buggy drives and unholy consoles, the reader should already do most of work (unlike, say, if you were to implement mounting)

The lack of image format compatibility is a very real problem, and it's not feasible to implement those individually for each core.

That's pretty secondary tbh. There'll always be time for a converter later.

There needs to be an app to make disc images that is designed with emulators in mind

I'm not sure what'd be special about emulators, but for all intent and purposes imo: https://github.com/saramibreak/DiscImageCreator
(and uh, yes, you should totally get in touch with redump devs, which might know quite some things)

Having both the VFS API and the CD/DVD API will allow experiments on creating a generic game archival format

Wasn't retroarch already settled with CHD?

the cores will always use it in favor of their native implementations. The latter will remain maintained for compatibility with legacy frontends, if there are any.

Mhh, why so?
Supporting reading original media seems pretty of a feature if you ask me.
In fact, I could imagine, say, some ps1-clone replicating it faithfully these days.

[nayslayer]

Thanks for your reply, @mirh. This is exactly the feedback I'm asking for :)

Reading of subchannel shouldn't really be rocket science.

It isn't, but you can't expect users to get it right without assistance, and I'd rather have the tedious tasks automated instead. I've had my share of corrupt dumps already, thankyouverymuch.

That's pretty secondary tbh.

Secondary, yes. Converting my images? Hell no, not unless there's a definite format that can be fed to any emulator. Image formats mostly differ in metadata, so it's not like they're that hard to support anyway. I only mentioned disc images so that they won't be blocked out by an ossified API.

I'm not sure what'd be special about emulators [...]

This isn't relevant for now, but basically, the idea is to make a disc imager that will detect disc types by their content and pick imaging settings accordingly, preventing users from making corrupt dumps. The tool you've mentioned is good news, though it'll need to be packaged into a user-friendly, foolproof GUI to make it useful to average users.

Wasn't retroarch already settled with CHD?

This is even less relevant for now, but the CHD isn't designed as a universal game format, it's primarily a HDD image format. For a format to become useful to collectors it'll need much more than that: content ids, metadata, reproducibility, error correction codes, compression size-speed tradeoffs, multiple partition support, etc. It's good as prior art, but it's unwieldy and easy to mess up, e.g. by specifying incorrect sector sizes.

Supporting reading original media seems pretty of a feature if you ask me.

Sorry, I meant the emulators' support for image file formats. I edited the original post to clarify that. Basically, I want cores to be oblivious of whether the data is backed by a read device or an image file, it's as simple as that.

[mirh]

foolproof GUI to make it useful to average users.

https://github.com/reignstumble/DICUI 🙃

but the CHD isn't designed as a universal game format, it's primarily a HDD image format. For a format to become useful to collectors it'll need much more than that

I'm not sure why a "hdd format" would support CDDA.. And I'm not sure if MAME guys would like to be told they are not true collectors /s

To the best of my knowledge bin/cue (what redump uses atm) and CHD are completely interchangeable.
And, I mean, CHD is already used successfully in PS1 cores, isn't it?

... Then I'm not claiming any particular expertise myself in this field, but maybe it's just you are left in what the state of affairs was years ago?

[anothername99]

Very good to see someone taking on this, and you bring up some good points. Here are some things I would like to add:

1. As the original purpose was to use this feature with PC emulation (but this goal is paused because we don't have a good and up-to-date PC emulator core yet), I would very much like it if your implementation allows for detecting disc type and how to read it even when the core is already running. Imagine you want to, say, rip a disc with Imgburn, in Windows XP, in a PC emulator core, in RetroArch. The user will expect Imgburn and the DVD drive to "just work". Discs will be put in and out during runtime and Imgburn will control how the disc is read.

2. While the goal for this bounty is read support, I intend to also follow this bounty up with a bounty for write support. It would be a very good thing if your design is open to be extended by someone who wants to implement write support in the future, so that they don't have to rewrite everything from scratch.

3. I agree that it would be a very good thing if the disc image support can be unified somehow. I noticed this the other day when I was testing Kronos. My whole Saturn set is CHD because I use that with Beetle Saturn, and suddenly I had to spend time converting games back to cue/bin.

4. Most consoles have games that require disc swapping. At the same time, some consoles don't respond very well if you arbitrarily open the disc tray. The Dreamcast for example will outright take the player back to the BIOS menu when it happens. It would be a good thing if the core can detect when the game actually wants a disc swap, and have RetroArch respond appropriately. So by default, opening the tray while playing a Dreamcast game, emulation should be paused and the emulated system should ideally not be informed at all of what happened. But, if the game wants the player to switch discs, then the emulated system should be informed of the tray status just as if it's a real drive. With PC emulation on the other hand, opening the disc tray should never trigger an emulation pause, because PC software is designed to be able to handle this.

5. As for bad CD/DVD drives not reading PS1 discs properly, this isn't a big priority for me and I think that validating the disc against a game database would be a very complicated approach. You would need a complete database of every single game and every single translation or hack that is out there. This would be a never-ending project just like the scanner that RetroArch has.

6. Another idea though would be to have a database of CD/DVD drives that are known to work well with this type of disc, and if the user has a bad or unknown CD/DVD drive, display a warning. This would effectively do the same thing with a dramatically smaller scope. If there are any holes in the database, the user can just choose to ignore the warning.

7. I do also think that it would be very useful to have a way of automatically detecting which system the disc is for. Every proper PS1 disc for example has a game ID starting with SLUS, SCUS, SLES, SCES, SLPS or SLPM. You don't need to know which exact game is in the drive to know that it's a PS1 disc.

[markwkidd]

CHD format for optical discs does seem to be gaining ground as a de facto standard over the last year or two now that it uses a lossless audio codec. Certainly I have seen CHD support spread quite a bit throughout the libretro cores over that time period.

[Colossus90]

“As for bad CD/DVD drives not reading PS1 discs properly, this isn't a big priority for me and I think that validating the disc against a game database would be a very complicated approach. You would need a complete database of every single game and every single translation or hack that is out there. This would be a never-ending project just like the scanner that RetroArch has.”

The redump site is creating such a database, all the hashes to confirm a correct dump. Which is why I think it would be interesting to work with the people of redump. They also have a lot of tools available. I have some contacts there.

[barbudreadmon]

While this idea is nice, keep in mind that some systems had non-standard optical media, or peculiar ways to read them.

[anothername99]

Another thought that just came to my mind is, what are the specific problems with reading PS1 discs? If the main issue is subchannels, then Redump is actually collecting those: http://redump.org/downloads/

So an idea is to put all the SBI files in a database. Then, when a PS1 game is in the CD/DVD drive, look for the game ID and then use a matching SBI file to "patch" the disc in memory.

[markwkidd]

My understanding is that this PR just added the ability for the PS2 port of RetroArch to read CDs: https://github.com/libretro/RetroArch/pull/8482

The "libcdvd" library seems to be PS2 specific but I wanted to make sure that the author of that patch @fjtrujy is aware of this bounty. Could it be that the PS2 implementation is a useful step towards a more generalized API for reading CDs and DVDs in RetroArch?

[markwkidd]

This bounty stands at $155 as of today.

[fjtrujy]

Thanks @markwkidd for notifying me about this bounty.

[chuchuw]

@anothername99 maybe the platform should be specified, like at least linux 64bits or windows 64bits.

[markwkidd]

Madnafen evidently uses libcdio: http://www.gnu.org/software/libcdio/

[anothername99]

Either Windows or Linux is fine. But if you can only do one OS, it is preferable that there is room for someone to pick up the code and add support for another OS in the future.

[markwkidd]

Bounty is up to $265 thanks to a $100 contribution from libretro organization bounty funds!

[twinaphex]

While there is still work to be done, I believe we should reward @bparker06 already under the condition that we will continue the work and further improve compatibility with existing drives.
We do need more eyeballs in this endeavor though. But I'll close it with the full faith and confidence that this will be delivered on.

[anothername99]

I will test this today. Was a possible expansion (for burning discs) taken into account with this implementation? Either way, thank you @bparker06 and I don't mind rewarding already.

[SpaceAgeHero]

DVD is not yet supported, right? Would be useful for PS2 / Play!