Retroarch: Switch: PSX Scanning not working

[AnalogMan151]

First and foremost consider this:

• Only RetroArch bugs should be filed here. Not core bugs or game bugs
• This is not a forum or a help section, this is strictly developer oriented

Description

PSX category not appearing on XMB after scanning a PSX BIN+CUE.

Expected behavior

Scanning directory of PSX BIN+CUE files is followed by a PSX menu option on XMB with scanned games.

Actual behavior

Scanning directories for PSX games does not add PSX game to XMB

Steps to reproduce the bug

1. Dump and verify BIN+CUE for a PSOne game (example: http://redump.org/disc/3160/)
2. Add verified BIN+CUE to SD
3. Scan directory using Import Content menu option
4. Observe as game is not added.

Bisect Results

No PSX games are working for me, despite the SHA1 and Cue file matching Redump records, the databases being updated and confirming that the game's ID code is in current database.

Version/Commit

You can find this information under Information/System Information

• RetroArch: 826c77d

Environment information

• OS: HorizonOS (Nintendo Switch)
• Compiler: NA

[RobLoach]

Don't think we have that entry....
https://github.com/libretro/libretro-database/search?q=%22Digimon+World%22&unscoped_q=%22Digimon+World%22

Mind making an issue on https://github.com/libretro/libretro-database about it? Would be great to update the serials across all redump systems.

[AnalogMan151]

Entry is in here: https://github.com/libretro/libretro-database/blob/master/dat/Sony%20-%20PlayStation.dat

game (
    name "Digimon World 2003 (Europe)"
    serial "SLES-03936"
    description "An RPG based upon Bandai's Digimon cartoon series, Digimon World 3 follows the adventures of Junior and his two friends, Teddy and Ivy, as they take part in a game of Digimon Online. Upon starting, the player will have to pick a trio of Digimon from the three initial sets available. Once in the world of Digimon Online however, the three friends are informed that the system is in dire need of repairs and that they'll have to stay until the repairs are complete."
    developer "Bandai"
    publisher "Bandai"
    releaseyear "2002"
    releasemonth "11"
    releaseday "15"
    users "1"
    genre "RPG"
    rom (
        serial "SLES-03936"
        image "Digimon World 2003 (Europe).cue"
    )
)

Should I still make the issue at https://github.com/libretro/libretro-database?

[RobLoach]

Oh great! Glad you found it... Serial scanning has always been spotty. Do you have the latest database update? Just tested with Tekken 2 on Linux and it seemed to work well.

Going to re-open this until we find the problem. Thanks for debugging.

[AnalogMan151]

I did Make sure to update to the latest database within the app as mentioned in the original post, also opened the compiled RDB and verified the entry was in there too. I've SHA1 checked the BIN file and it matches the one shown on http://redump.org/disc/3160/.

I downloaded the CUE file and made sure my BIN was named appropriately to what was in the CUE file. I then made sure the CUE file was named as it appears in the DAT entry. Other systems are working for scanning including SNES and GBA, so I know it's just PSX affected. I've also edited the CUE file to update the internal name to match that of the entry. (CUE file name and BIN file name inside CUE are Digimon World 2003 (Europe) (En,Fr,De,Es,It) while in the DAT it's Digimon World 2003 (Europe)). After renaming to match there was still no change. I have done both Directory and Single file scanning.

May I see the contents of your Tekken 2 CUE file so I can see if anything is missing from my CUE downloaded from http://redump.org/disc/3160/?

[RobLoach]

It's "Tekken 2 (USA) (v1.1).cue" from Redump: http://redump.org/disc/14017/

[AnalogMan151]

I have a copy of Tekken 2 (USA) (v1.1). I checked the SHA1'a and they match. I added it to my Switch and scanned both the directory and file, but no go. So even with the same game as the one you tested it still does not show up when scanned. May I ask how your files are named? Does naming have any impact or does it go solely on serial?

[RobLoach]

It's all the serial. The filename is pretty much ignored when it sees .iso. The code is around....
https://github.com/libretro/RetroArch/blob/43bbb5b2310757aa83ef0fcde5f63d672d2781cf/tasks/task_database.c#L653-L657

Make sure they're uncompressed too. Serial scanning doesn't work with .zip files.

[AnalogMan151]

I'm using BIN+CUE, so I assume the code I'm looking at is around here:
https://github.com/libretro/RetroArch/blob/5b4473b9489c6034fb88e78f6bf3775fa8448644/tasks/task_database.c#L627-L637
Looks like it grabs the file referenced by the cue (the BIN file) then it opens the BIN file, grabs the SYSTEM.CNF file and reads the boot line to get the serial. If that succeeds then it returns the database type. If it fails it it attempts the same thing but using a CRC32 instead of a Serial.

Looking at the serial retrieval code it seems fine too. Considering it works on Linux and not Switch I have to wonder if one of the functions are not working correctly or rely on a different definition or something. I'll see if anyone else has the same issue.

Any debug messages I can enable during the scan process to see what it's doing while processing the CUE file? Or can I tell it which database to use in case it's not detecting properly?

[i30817]

Some very early ps1 games don't have the serial in the executable filename (what the boot line in the system.cnf references), and instead use the iso format 'volume_identifier' to hold the serial. But i don't know if Tekken 2 is one of them (King's Field (Japan).bin is one of them, which btw is a different game than the USA version with the same name).

What i've often found with failures on the retroarch scanner is that it's bad at interpreting valid variations of the cue/bin format, for instance some Saturn translations turn the MODE2/2352 bins into MODE2/2048 bins (actually iso but lets not be pedantic).

And almost all of the parsers completely ignore 'mixed mode' (which is used on videos only anyway) because it substantially complicates filesystem reading - not that i'm completely sure that Retroarch actually bothers with a filesystem parser for the ps1 instead of treating the filesystem as a character array and seeking 'boot=something terminated with;\n\0' like a barbarian.

Anyway, the second to last paragraph means that Retroarch implementation of 'finding what system the iso/bin belongs to' with 'magic bytes' to find by serial failed horribly if the format was converted because of different sector padding. In fact, it didn't actually work at all and (i presume) would always fall down to checksum of the whole iso because the serial is kind of system dependent and the system was nearly always misclassified.

There is also a case of a false positive from origin, where the game was printed with a duplicate serial of another game by mistake (Urban Chaos (USA) is a - wrong - duplicate serial of Threads of Fate (USA) and has another serial on the database). There is a way to distinguish both with the volume label but i don't think the scanner ever bothered to hack it in.

[i30817]

https://gist.github.com/96674a3de8d9e4cb890e92cec3f36990

Here is the code i try to use to classify cd games...

this section for both ps1 and ps2:
https://gist.github.com/i30817/96674a3de8d9e4cb890e92cec3f36990#file-dumpid-py-L189

I keep thinking of proposing something similar for retroarch - especially to make SEGA consoles have a 'serial' too. I gave up on trying to understand what. the. fuck. process was used to make the serials on redump sega 'cds' consoles because they all have these 'normal' serials on the 'main info' pages but if you check them on the header they're a jungle of different offsets and different characters than advertised.

It would be easier to 'md5sum' the whole header 'first 256 bytes' habitually and scrape the header from the redump pages to associate, since it's there, so that's what i did on that code (except for the sega cd that has 2 headers, one for 'if the game was released on the megadrive' for some reason).

But it's understandably intimidating to make a scrapper for redump to extract the 'sega header' section to make a checksum and then propose all the info and code necessary for a true serial scan.

Gamecube and wii are even worse, because ofc nintendo has to be difficult and created something that isn't even a iso or keeps a header on the first sector (like sega did), so i didn't even try without libmirage support.

I honestly ended up wanting to force crc32 (or some other form of total hash) for everything, since they're at least ultra unlikely to give false positives, unlike serial scanning (which breaks as soon as you have a cd hack to identify). It's much slower but at least everything should get identified corrected if hardpatched.

edit: i opened a RFE for the 'force a hash check' option.