Retroarch: Thumbnails for Sony PSX, N64, Sega Saturn, Sega CD not properly scraping/adding to Games list.

[futurepr0n]

Hi RetroArch,

I am trying to add a bunch of games for their thumbnails and have ran into some problems. I aim to understand these problems better and then document them for the Wiki to help other struggling fans of RetroArch set up their libraries all nice!

Presently, I have had no issues when adding Sega Master System, SNES, NES, and TBGF16 - but when trying to add in libraries for PSX, N64 etc -I am not getting the behaviour I expect.

First - I have my directory structure as follows:
ROMS
|------NES
|------SMS
|------PSX
| |-----Amazing SpiderMan
|------N64
|------Sega CD
| |-----Sonic CD

So as you may note, in the PSX and Sega CD directories, there are sub dirs that contain the bin/cues. These are also named in full and resemble the titles to the BoxArt in the thumbnails directory. I found that "searching" the directory at the root level to "Sega CD", would not scrape the sub dirs. I then would scrape the sub directory - but this did not add the game to my collection. In the case of PSX however, I was able to add the games, and they show in XMB - however none of the box arts are showing.

I tried searching for more info on the website and other tutorials, requesting help on the retroarch subreddit, and finally now decided to request help as an issue to Git - so that I may help contribute in documenting this more thoroughly and aid other users through the process.

Please let me know if we can talk a bit about this, or if you can help lead me in the right direction.

Thanks!

[RobLoach]

Mind giving CRCs for the Spierman and Sonic games? Can check if they match in the DB.

[cudencuden]

rob also started https://github.com/libretro/RetroArch/issues/3139 to look into the recursive scan

[futurepr0n]

Is there anything else you can comment on - regarding the directory structure? it would see that RetroArch might not scan through SubDirs (I am using OSX version if that helps) - I tend to get no scan result, and have to go one level down into the folder (named the same as the bin/cue) and only then does it seem to scan.

For PSX games, as mentioned this successfully added them to a "library" but did not pair them to any Thumbnail art. Kinda strange.. I also had issues if I kicked off multiple thumbnails to be downloaded through the updater - that I had I guess a service lockup if one was unzipping while another concluded its download. I would check the Thumbnails directory and see the package didnt properly download and extract - and had to retry one at a time. Perhaps there is a way to queue these so that one does not extract if another is currently locking up the service?

Sega CD - Does not add to any special library, not to Genesis or its own.

cksum *
152349360 555815232 Amazing_Spider-Man_Vs_The_Kingpin(US-Genesis).bin
4192449800 877 Amazing_Spider-Man_Vs_The_Kingpin(US-Genesis).cue
28528996 247907856 WWF Rage in the Cage.bin
276792724 148 WWF Rage in the Cage.cue
2511050117 446976432 Bram Stoker's Dracula (combo pack 2.0 version) (U).bin
575456537 377683283 Bram Stoker's Dracula (combo pack 2.0 version) (U).bin.cpgz
4155718100 479 Bram Stoker's Dracula (combo pack 2.0 version) (U).cue
2366703144 2064 Bram Stoker's Dracula (combo pack 2.0 version) (U).nfo
2568400240 662353776 Ecco - The Tides Of Time (Sega CD) (U).bin
3032535492 1228 Ecco - The Tides Of Time (Sega CD) (U).cue

PS1 - Adds to its own library, but no Box Art.

3129425830 459220944 Apocalypse (USA) (Track 1).bin
2035405079 32634000 Apocalypse (USA) (Track 2).bin
3545206672 202 Apocalypse (USA).cue
3611499550 383679408 Cool Boarders 3 (USA).bin
1735794072 87 Cool Boarders 3 (USA).cue


N64 - Adds them to Library - no Boxart though as well

1489752577 16777216 Banjo to Kazooie no Daibouken (J) [!].z64
2879646290 16777216 Banjo-Kazooie (U) (V1.0) [!].z64
698503272 33554432 Banjo-Tooie (U) [!].z64
162631691 8388608 Doom 64 (E) [!].z64
212311328 8388608 Doom 64 (U) (V1.0) [!].z64
924396228 12582912 GoldenEye 007 (E) [!].z64
476594625 12582912 GoldenEye 007 (U) [!].z64
2555227654 33554432 Legend of Zelda, The - Ocarina of Time (U) (V1.1) [!].z64
1372727923 33554432 Legend of Zelda, The - Ocarina of Time (U) (V1.2) [!].z64
2957857084 12582912 Mission Impossible (E) [!].z64
2871800405 12582912 Mission Impossible (S) [!].z64
3968159361 12582912 Mission Impossible (U) [!].z64
1441554933 8388608 Monopoly (U) [!].z64
1504674843 16777216 NFL Blitz 2001 (U) [!].z64
2674793121 12582912 Star Wars - Shadows of the Empire (E) [!].z64
2319649526 12582912 Star Wars - Shadows of the Empire (U) (V1.0) [!].z64
51675283 12582912 Star Wars - Shadows of the Empire (U) (V1.1) [!].z64
3861665609 8388608 Super Mario 64 (E) (M3) [!].z64
1361803195 8388608 Super Mario 64 (J) [!].z64
419024783 8388608 Super Mario 64 (U) [!].z64
569959622 33554432 WWF No Mercy (E) (V1.0) [!].z64
3225374773 33554432 WWF No Mercy (E) (V1.1) [!].z64
502640314 33554432 WWF No Mercy (U) (V1.0) [!].z64

thanks @RobLoach

[bparker06]

I submitted a patch that implements recursive scanning per issue #3139 and it was merged on Sunday. This should be available in the next 1.3.7 release, or if you download the latest nightly you should be able to test it out. Recursive scanning is always on by default now.

[futurepr0n]

Great @bparker06 - any info on how I can maybe check my own CRC's against 'a' or 'the' database which pairs them to the BoxArt? no sense in having someone else check on my behalf (as offered by @RobLoach ) when I am perfectly capable of doing such a thing!

Appreciate the patch you worked on as well - but still curious about a few things - not sure if you can help...

If ROM's are scanned and added to a library in the XMB, is it possible to remove them? I have a lil Nintendo Satellaview library added now - but no Game Art is available for these, and for that reason I'd like to hide or delete them from the quick launch - or whatever that section would be referred to - in the XMB?

Also individual roms within a collection - for instance, I have a few JPN games that I also have USA versions - instead of deleteing them from the ROM directory and rescanning, now that they have been picked up by the Game Library, if I wanted to select an individual game/rom in the list and "remove" it - is this possible?

[bparker06]

As far as I'm aware, playlist entries cannot be modified from within the GUI, you would have to manually edit the files in ~/.config/retroarch/playlists or your relevant config directory.

I believe the databases are saved in a custom format, but they are derived from the DAT files here https://github.com/libretro/libretro-database/tree/master/dat which you can check individual CRC/MD5/etc. checksums from.

[RobLoach]

@futurepr0n Mind running a crc32 * on them instead? As @bparker06 explained, that's what it uses to match.

[futurepr0n]

No problem, I am a CRC n00b, my bad. Thank you.

b79bddd8    WWF No Mercy (E) (V1.0) [!].z64
43ba9e7e        WWF No Mercy (E) (V1.1) [!].z64
b33f44f0        WWF No Mercy (U) (V1.0) [!].z64

Sega CD
a009d981        Amazing_Spider-Man_Vs_The_Kingpin(US-Genesis).bin
152d19d0    Amazing_Spider-Man_Vs_The_Kingpin(US-Genesis).cue

961b3eba    Apocalypse (USA) (Track 1).bin
9d424890    Apocalypse (USA) (Track 2).bin
1fc2f0b6    Apocalypse (USA).cue

[RobLoach]

• Sega CD
  
  
  
  • Uses Redump, TOSEC, or Trurip. If you rip them using their documented methods, it'll work. As an alternative, you could download the Amazing Spiderman CUE sheet from Redump, and name the .bin file to what it expects, and it'll load correctly.
  
  
• N64
  
  
  
  • Looks like one of the GoodTools sets. They unfortunately don't have the No-Intro naming convention, so the thumbnail filenames don't match up.
  
  
• PSX
  
  
  
  • The PSX database currently only does single-image Serial-based reading. Looks like a redump cuesheet you have there. There's is a PR over at https://github.com/libretro/libretro-database/pull/251 to switch it to use Redump/TOSEC/Trurip. Would appreciate your thoughts on that in the PR.
  
  

[ghost]

The PSX box art library is not complete and those games you listed do not have a box art added yet. So of course you won't see box arts for them in menu.

[futurepr0n]

Sorry, not sure I follow @RobLoach - the GoodTools set is compatible, but I dont have a matching name convention? I couldnt really gather from the link whether there is anything I can do to make them compatible. Perhaps thats not the case.

For Sega CD and PSX - lil bit beyond my comprehension but looks like you have a pull request that could correct this issue - and I could possibly test it? Or you are asking other contributors for their feedback on the linked PR.

Thanks @Axiphel - I did see some of the thumbnails for the PSX though, but didn't post those particular ones I guess. Family Fued for instance. Anyways - it looks like Rob has addressed my issue as well, and I have bad romsets/dumps. Thank you for helping as well!

[RobLoach]

Sorry, not sure I follow @RobLoach - the GoodTools set is compatible, but I dont have a matching name convention?

Good64 works, just the thumbnails don't share the same names. The thumbnails follows No-Intro. We'd have to rename all the titles in this DAT to match No-Intro names:
https://github.com/libretro/libretro-database/blob/master/metadat/goodtools/Nintendo%20-%20Nintendo%2064.dat

For Sega CD and PSX - lil bit beyond my comprehension but looks like you have a pull request that could correct this issue - and I could possibly test it?

For Sega CD, you'll just have to download the .cue file from Redump.org , and use that instead of the one you have. The redump .cue files can be downloaded from their site.

For PSX, the fix is up over at https://github.com/libretro/libretro-database/pull/251 . You could post your thoughts or just say +1 over there :wink:

[futurepr0n]

Thanks @RobLoach hah - now is there anyway I can run a batch rename on my Good64 Romsets so that the names match and boxart will grab? Is there any chance you guys would like someone to help document this as part of the site wiki? or github wiki? I could take some time to compile this info into a section/faq... I appreciate the help!

[RobLoach]

It's not your N64 roms that are named wrong, it's the libretro-database goodtools DAT:
https://github.com/libretro/libretro-database/blob/master/metadat/goodtools/Nintendo%20-%20Nintendo%2064.dat

Would be neat to have a script to rename them all automatically somehow.

[futurepr0n]

Any chance I can help here? Still not understanding thoroughly, but I did a quick compare:

43ba9e7e        WWF No Mercy (E) (V1.1) [!].z64 

from my rom set - is equal to the entry in the database:

game (
    name "WWF No Mercy (E) (V1.1) [!]"
    description "WWF No Mercy (E) (V1.1) [!]"
    rom ( name "WWF No Mercy (E) (V1.1) [!].z64" crc 43BA9E7E )
)

It matches on complete name and CRC. What am I missing here? I've gotten fairly proficient with Python, wouldn't mind working towards some sort of solution / script. I just need to ramp up my understanding of how everything works within RetroArch, I haven't really familiarized myself with the source.

[bparker06]

He is saying that your filenames match the goodtools DAT... but the thumbnail uses the NoIntro DAT... which have different names and CRCs:

https://raw.githubusercontent.com/libretro/libretro-database/master/dat/Nintendo%20-%20Nintendo%2064.dat

This is the NoIntro DAT

[cudencuden]

rob mentioned that thumbnails are based of no-intro..which in your example, no-intro has:

game (
name "WWF No Mercy (Europe)"
description "WWF No Mercy (Europe)"
rom ( name "WWF No Mercy (Europe).n64" size 33554432 crc EEB166B4 md5 F58F64A8AD952459B2EE37F07004272B sha1 CADBCD5361DFFAF2AA17F0CB4C4CC799BDC8350A )
)

see https://github.com/libretro/libretro-database/blob/master/dat/Nintendo%20-%20Nintendo%2064.dat

[RobLoach]

To fix this, you would....

1. Go to the GoodN64 DAT over at https://github.com/libretro/libretro-database/blob/master/metadat/goodtools/Nintendo%20-%20Nintendo%2064.dat
2. Find the WWF No Mercy (E) (V1.1) [!] entry
3. Change the name and description to match the No-Intro naming scheme: WWF No Mercy (Europe)
4. Rebuild the database
5. Copy the new rdb in
6. Delete your ./.config/retroarch/playlists
7. Rescan the directory
8. See the new thumbnail appear for WWF

Again, I'm unsure of actually changing it in that DAT because we would loose those changes when updating GoodN64 in the future. Hence, having a script automagically update from GoodTools naming scheme to No-Intro would be pretty nice.

[ghost]

It would just be way simpler to replace your roms with those from the No-Intro rom set...

[futurepr0n]

@Axiphel - it would seem that way.. Is there not a benefit to an end user if maybe some of the thumbnail pairing was relaxed a bit so that you could select based on name criteria alone? I have the issue where I need to go to redump for some of my cue files and that might be a pain in the arse.

[cudencuden]

thumbnails detection is based on names...but no-intro names

[futurepr0n]

@cudencuden But the SegaCD checks require me to download new .CUE files - and that would probably mean I need to edit those .cue files to load up the .bin files I already got that for some reason my present .cue files are not being identified? So that would be a PITA no? I mean, I gotta go into every file and then replace the .bin file name or I gotta rename the .bin file to match the redump .cue

[RobLoach]

1. Download the CUE file
2. Rename the .bin image that you have to match the filename that the CUE file has specified

[futurepr0n]

Yeah - but doing this 100+ times @RobLoach ... Also was curious why the reason for such strict checking against files. What problem is it, if someone wanted to have boxart associated with a hacked rom that included a trainer, or or some other tweak/mod? Maybe an option or toggle for purists romsets, and then for a "take it as is" type of boxart association based soley on rom names for like 85% matching - would be pretty cool and handy for non-saavy users who may get a bum rom or not know that they aren't using the best dump.

Is it kinda for safety? If your roms dont match the crc check - you might be using something that could be a virus on your pc (as example)?

[bparker06]

It would be nice to be able to have boxart no matter what you might have done to your rom that's different from the checksum in the database... but I don't know of a sane way to accomplish that short of just fuzzy checking the name somehow, which seems error prone.

[RobLoach]

as curious why the reason for such strict checking against files

The Alternative Scanning Method feature has been suggested before. Would make it a lot easier if you don't want to match CRC data, just simply match a Folder up to a Core.

Benefits of CRC scanning:

• It gives clean titles for when you files arn't named correctly
• Allows detection of which system/core to use, no matter what folder/files you scan

Cons of CRC scanning:

• You need to make sure you have the right file
• Optical media (like Sega CD, PSX, PCFX-CD, etc), get tricky since there's a million different ways to rip a CD, all resulting in different CRC data. That's why there are projects out there like Redump, which documents a standard way to rip the media.

It would be nice to be able to have boxart no matter what you might have done to your rom

It will work, no matter how you've named your files. As long as the CRC matches the No-Intro DAT, then BoxArts wil work. GoodTools doesn't share the same naming convention, so we have to update the DAT to fit No-Intro naming scheme.

[kivutar]

Just name your boxart with the same name as your playlist entry, and it will display. You can write playlists manually or use a 10 lines script to generate them. Without CRC matching. Thumbnails are not dependant on the CRC or the database, they are just fetched using /.png

[futurepr0n]

@RobLoach I was going to try testing your Commit with the PS1 Thumbnail scanning, but am wondering is there any precompiled builds or do I need to have my whole environment set up so I can build for each platform I wanted to test on?

Thanks