Retroarch: Integration of ROM Hacks into Online Updater

[SpaceAgeHero]

Dear devs,

I would like to ask for an integration of Rom Hacks (preferably romhacking.net contents) into the online updater of RetroArch.

Here are some additional thoughts:

• maybe somehow like cheats work?
• download complete hack-sets system-wise
• RomHacking.net propably doesn't provide an API so contents must be mirrored / hosted somewhere else?
• ability to enable a specific Rom Hack on demand for soft-patching

Benefit: easier access and use of hacks, like translations, bug fixes, awesome color hacks or other community improvements.

[orbea]

With all due honesty this sounds like a huge maintenance burden and bug farm with very little benefit for most users.

[hizzlekizzle]

I think it's a great idea, but yes, without an API from RHDN, it would be a maintenance burden, for sure.

As for being of little benefit to most users, I think that's what OP is getting at: it would make romhacks more accessible to people who know nothing about them.

[orbea]

As for being of little benefit to most users, I think that's what OP is getting at: it would make romhacks more accessible to people who know nothing about them.

I see what you mean, but do most people even want to use hacks? My impression is that its a niche use case, but maybe I'm mistaken?

I think the biggest problems excluding the lack of an API is making sure the content and patch matches, romhacking is full of patches that only work with very specific roms, some which depend on other hacks or changing the rom from headered and back and forth... (i.e the first patch needs a headered rom and the second patch needs the first patch and a unheadered rom)

I would think this is out of scope, but then I'm not sure we have a clearly defined scope in the first place...

[i30817]

This will probably never happen, but just in case, a aside:

The ips rom hacks for pc-98 and snes that 'require' a header can be made to apply on roms without a header with https://github.com/heuripedes/ipsbehead

bps romhacks of course, never need to care about headers.

btw, this would get significantly more complicated for iso romhacks. Some of them need conversion to iso first and subsequent cue edits, some of them need cdmage image rebuilding etc.

My romhhacks PRs for libretro database list the romhacking.net urls of the patches applied on the romhack entries on the dats in order (multiple patches may be applied) with the idea that all those patches that 'need' a header had ipsbehead applied, so they work with no intro.

I use this project to keep them up to date: https://github.com/i30817/rhdndat

I think the biggest obstacle to this happening is that romhacking.net never bothered to mandate a standard 'container' format. A patch zip might have 2 patch files, ips and bps, 2 ips files, for headered or unheadered, or divide the hacks into multiple options you're 'supposed' to apply at will etc.

[i30817]

I also had the idea of using the urls on the romhack dats on liibretro-database to allow a person to see what 'romhacks' entries are applicable to a original game and opening the urls from retroarch; which is a much lesser version than 'automatizing everything'.

I believe that i gave up on that when i figured out that retroarch s currently not even using the CRC32 of roms to recognize romhacks, so there is really no point to make it easier to apply them when RA won't even recognize them.

[hizzlekizzle]

@i30817 good point on the lack of standards for patch containers. Certainly complicates things.

@orbea I would think the fact that we do CRC scanning would help with the header thing, but it would really require the hacks be pre-sorted to know what goes with what.

[RobLoach]

Softpatching in RetroArch works. Download the patch, and name it the same as your rom file, in the same directory, launch the game. RetroArch will apply the patch in memory.

• Having a UI around ROMHacking.net to download patches would be neat, but as orbea mentioned, it would be a lot of work to maintain.
• i30817 has submitted a few of the most popular hacks to libretro-database/metadat/hacks.
• The Menu UI is difficult to work with, but I could imagine providing users a way to run roms with a selected patch. This would allow users to run a patch on the rom without having to rename it manually.
  

[i30817]

Unfortunately softpatching is not suitable for roms larger than 32mb or so. Retroarch doesn't even support it for the n64 (or NDS... or even genesis plus DX) much less the ps1 and larger.

It's a elegant idea, don't get me wrong (to calculate the crc32 of ips/bps files instead of the final result to match it to a romhack entry). I wouldn't mind creating a new entry that lists the patch crc32 of the patch file for rhdndat output.

However, people use different patch formats than the original for valid reasons all the time (ipsbehead is a example, but not the only one, for example, it's possible to adjust a N64 rom byte order to create a romhack patch that applies to another byte order by applying the patch on the right byte order, switching the byte order and creating a new patch to apply to no-intro for instance).

I also do not keep patch files for cds/roms larger than N64 roms (because they can't softpatch) and instead keep 'revert' patches xdeltas to get back the original after a hardpatch, so i couldn't even calculate those without redownloading all of the cd patches... which i probably won't, for something that is useless to me (and everyone because RA can't softpatch them).

[orbea]

I would think the fact that we do CRC scanning would help with the header thing, but it would really require the hacks be pre-sorted to know what goes with what.

Exactly, also consider that new hacks and translations are added frequently, often not in English. :)

[i30817]

That already occurs. Having a incomplete database is not enough reason to give up having a database, and if people are especially attached to, say, spanish translations, they can PR the database using rhdndat to keep the patches up to date, like I do for many english ones (not all ofc).

I'm really disappointed in RA having the information and not actually using it (or allowing more complete PRs for snes and the nes). Especially when a translation (hardpatched or not) is 'recognized' as the original game because there is no way to turn off the serial scanner for the crc32 scanner.

[orbea]

if people are especially attached to, say, spanish translations, they can PR the database using rhdndat to keep the patches up to date, like I do for many english ones (not all ofc).

Are you volunteering?

[i30817]

No, because i don't know spanish and therefore have no interest in spanish translations (in spite of living right next to spain). I'm just saying that the idea that 'we don't want to recognize translations and hacks because some aren't popular enough to attract a PRs' is bizarre.

[orbea]

You're not understanding my point, the point is that this is massive ongoing project with no willing maintainers.

[i30817]

This - to be clear, just recognizing romhacks, not downloading them - is a optional effort that needs no ongoing maintainers in the main team and there is the possibility to make tools to make it as easy as possible to update them too (my own being a example, though that one needs the roms to calculate the final hashsums). If that's your attitude, you might as well gut everything not from dumper groups from libretro-database.

That there is no way for a user to request to use the CRC32 method again means that those entries that are there, are useless for users.

[ghost]

whats difference of just browsing to romhacking sites like romhacking, download patch and rename as needed? also, how to know if the patch requires a patched rom or not? who is gonna maintain database for this (ugh.. more database files....)

[Lreaper]

It would be really neat if there was a menu option to load content with a specific ips patch, I'm aware softpatching already works if you have the patch in the same folder as the rom file but what if you have more than one patch for one specific game? Having a menu option to load content+patch would allow the user to have several romhacks available for any given game without having to constantly rename/move patches around.

[i30817]

That would require ips/bps softpatching to be externalized from cores, which is something that RA-libretro has steadfastly refused to do for many years.

Though i'm sympathetic about the fact that if they did it would be done badly (by using a tmp file hardpatch) because Windows filesystem / ramdisk apis utterly suck and RA is not willing to make a feature only work on unix or whatever.

About 'more than one patch', i suggest looking into hardlinks in linux, even if they sabotage a trick i'm hoping RA picks up (xattr records of the patched file crc32 for softpatches rom targets).
edit: symbolic links have the same problem... linux should fix this shit...

[Mte90]

I agree that a system to download is not easy but can be a step 2, better focusing on reconizing rom hacks more easily.
Right now softpatching seems cool but what to do in case that the patch are more then one?
Or in the case that the format is not supported by the softpatching?
Also right now renaming the patch doesn't let us to remember what was the patch doing.

An user case like mine:

• Get the rom
• Download different patch
• Try them one by one
• See if it is possible to merge more then one

Right now retroarch use crc that is an issue on detecting some roms so maybe we can find another way to add roms already patched.
Maybe adding in the rom file "Amazing-game.nocrc.zip" and if the filename include nocrc doesn't use the rom scan but add to the playlist of the console (reconizing the file format content).

Another solution can be simplify the adding of patches to https://github.com/libretro/libretro-database/tree/master/dat from the community so the patch are availabl right there so the scanning is more simple.

[i30817]

Right now softpatching seems cool but what to do in case that the patch are more then one?

Modern filesystems can deduplicate roms. When i need this I just create a hardlink, but i bet that some modern filesystems are thinking of doing this automatically. It's more annoying in the case of cd hacks, because those have to be hardpatched and it's in those cases that a hardlink would be great (because huge waste of space). RA doesn't support softpatching after a certain size (enforced by simply not implementing it for the N64 up), so you can't really do anything about it. I simply tend not to have more than 1 patch for cds at a time...

Right now retroarch use crc that is an issue on detecting some roms so maybe we can find another way to add roms already patched.

Merged patches willl probably never be 'automatically' detected. It's simply impossible with any checksum approach. I do put in some merged patches into libretro-database but that's because i merged them manually as good combinations, but even then i don't bother with edgy 'patchers with 10.000 options' that some hackers are doing and just add the 'maximal' option of 'all hacks'. Combinatorial explosions are unfeasible.

Right now retroarch [doesn't] use crc [except on netplay, only that crc is of the game executable so mostly useless for the current database of hacks] that is an issue on detecting some roms so maybe we can find another way to add roms already patched.

FTFY. This regression was added because people were complaining that their self-dumped games in random formats were not being recognized by the scanner and were slow to scan so now your 'reward' is that all game hacks are scanning as the original game because the main scanning method was replaced by something not using complete ROM checksums and thus imprecise. To solve this All Roads Lead to customization of the scanner and amortization of the checksums.

From the lack of even response, a issue that all RA contributors are not interested in btw.

Another solution can be simplify the adding of patches to https://github.com/libretro/libretro-database/tree/master/dat from the community so the patch are availabl right there so the scanning is more simple.

'Just' pressure romhacking.net to release a good dat or do what i do and use a tool to keep your local patches up to date and submit the updates to libretro-database (even if the database is currently not used by RA, the tool is still useful to detect when i need updates). Though be prepared to get your commits rejected because fears that 'too large' a patch collection will slow the scanner (can't commit the checksums i have for NES or SNES here). Though it's arguably irrelevant anyway, since they're not being used.

And you probably are having a 'XY problem' moment here anyway. It's not the lack of libretro-database data that is sabotaging the recognition of hacks in RA (though ofc there are many patches/games/systems i'm not interested in so never contribute). It's the scan method (imprecise console specific serial vs precise checksum).