Retroarch: Scan folders is not working with zipped files if there's no MAME or FBA cores installed

[sergiobenrocha2]

Scan is not working properly, try scan a folder and you'll get only a "scanning of a directory finished", it seems it just skipped all the files there.

I managed to scan only PSP games, a cue+bin psx and cave story.

Tested in the last android build, f-droid

[twinaphex]

@kivutar @bparker06 Can we confirm there is an actual issue here?

[lasers]

@twinaphex: I can confirm this with linux for two weeks!

[lasers]

Kivutar's report -- https://github.com/libretro/RetroArch/issues/3779

My report -- https://github.com/libretro/RetroArch/issues/3763

My forum report -- http://libretro.com/forums/showthread.php?t=6858

Somebody's report (almost a month ago) -- https://github.com/libretro/RetroArch/issues/3257

@sergiobenrocha2: Here's a temporary solution. Go back to stable version just to scan shit... then come back to git version when you want to. That's how I got my playlists.

[bparker06]

@lasers Can you tell us what git version you're using? I'm not able to reproduce this on Linux currently.

[lasers]

RetroArch: Frontend for libretro -- v1.3.6 -- df87fc7 --
Compiler: GCC (6.2.1) 64-bitBuilt: Oct 15 2016

I moved *.lpl off somewhere else before doing this test.

Music -- No go.

Pictures -- Scanning way too many files. I grab ImageViewer core to see if it works. It did. I looked at few RetroArch screenshots. I then quit because it was too many files. When I tried again with small directory this time, it won't get anything now.

Images -- No problem scanning isos. My PSP/PS1 games shows up.

Games -- It does not pick up anything. My games are all zipped and sitting inside directories. I tried unzipping one thing to scan again. It picked up just fine. I don't have any 7z archives right now.

Try moving your *.lpl somewhere and test again?

[bparker06]

I'm confused... you say games show up but they don't. Were you able to scan this same content before in a previous version? Could it be that your particular content just is not recognized by the database? The CRC has to match with known sets for games, like no-intro etc.

[bparker06]

I did not have any .lpl files when I tried scanning a directory full of zip files; it is working for me here so not sure how to reproduce your issue.

[lasers]

Is there a different CRC for zipped files or do you read inside the zipped files and compare CRC? Maybe No-Intro database is slightly outdated... by a year or so.

[lasers]

RetroArch: Frontend for libretro -- v1.3.6 -- 218d6f5 --
Compiler: GCC (6.1.1) 64-bitBuilt: Jul 19 2016

I didn't move *.lpl off somewhere else before doing this test.

Music -- Likely not part of this version. Still no go.

Pictures -- Likely not part of this version. It's scanning my p0rn album. Because it was too old, I don't have an option to see Picture tab nor the dirty files residing inside. The file content_image_history.lpl is still empty. I'll check later. Still no go?

Images -- No problem scanning isos. My PSP/PS1 games shows up. (Same)

Games -- I scanned few small game directories. No problem picking up multiple bogeys (over 400) hiding in their usual directories. No 7z archives right now for experimenting.

Try moving your .config/retroarch somewhere and test again? I don't know.

[kivutar]

I checked out master this morning, scanned my atari 7800 collection, it worked.

[lasers]

Scanned my Atari 7800 collection. Git no go, Stable go.

[sergiobenrocha2]

I can't reproduce this issue on linux (ubuntu 14.04), just on android 4.4.2

[lasers]

Clean profile. I'm able to scan contents right now. My problem was just like sergio. It skipped everything but images. I moved my old ~/.config/retroarch somewhere away for now.

[sergiobenrocha2]

Uh, I tried to do the same on 16.10, same behavior of android...

Both 14.04 and 16.10 were in a clean profile

[lasers]

I believe you. Do you use symbolic links? Any user-defined paths inside cfg? Anything you could think may be the reason for this issue? I checked permissions. I think it stopped scanning right after discs... Something could be throwing the scanner off?

We could use more verbose in the logging to see where it exactly stopped working and start reporting "Finished scanning" ?

[sergiobenrocha2]

I think it's not the case. You can't use them on android anyway.

[lasers]

Okay. I merged both dirs and now I'm unable to scan.

• Something is broken in my retroarch directory.
• There won't be any problem with initial scanning, only rescanning?

[sergiobenrocha2]

There's issue with scanning and rescanning

[kivutar]

I noticed that on Android I'm unable to scan zips.

[sergiobenrocha2]

moved to #3966

[RobLoach]

@sergiobenrocha2 I have noticed that for subfolders, it stalls for a bit when initially starting up.

[bparker06]

Here's what I have observed on the latest android build with no-intro snes games:

• after initial install, simply trying to scan before anything else will not find anything.
• updated core info files, still nothing
• updated databases, now it finds the files and creates a playlist, but scanning is very slow (haven't downloaded any cores to check its info files yet, this is what speeds up the scanning considerably)
• deleted the playlist, downloaded snes9x core, now nothing is found anymore.

Not sure why this is happening only on Android, Linux seems fine for me.

[bparker06]

I did find a workaround for now... you must have a core downloaded that supports .zip, like MAME or FBA. Once you have that core (plus the one related to the content you are trying to scan), and assuming your core info files and databases have been updated, then scanning should work.

[sergiobenrocha2]

Eh, I can confirm what @bparker06 said

[lasers]

I can confirm this issue too. https://github.com/libretro/RetroArch/issues/3836

[sergiobenrocha2]

Seems RA scans only if there's a core associated to the rom extension

Genesis Plus GX:
supported_extensions = "mdx|md|smd|gen|bin|cue|iso|sms|gg|sg"

PicoDrive:
supported_extensions = "bin|gen|smd|md|32x|cue|iso|sms"

They don't have .zip associated, so RA skips it.

Good catch guys :)

[rlabrecque]

Any update to this, nighties are pretty unusable without being able to make playlists of zipped files currently!

Seems like maybe it was this? 88c8c49ef89c76afb2349e5358e6afb3a4a571d6

[fr500]

You don't need playlists to use RA at all.
It's completely usable without playlist support.

[sergiobenrocha2]

@rlabrecque, there's a workaround, like bparker said: you need to install a core like mame or fba to get scan working with zipped files.

[ghost]

So there is a reason for why I can't scan .cue/.bin with beetle_psx_libretro.so ?
In fact it scan, but don't show them in retroach menu (and no playlist add it seems).
Even know than I can load them with retroarch -L /usr/lib/libretro/beetle_psx_libretro.so game.cue, and after that I can load them from the retroarch history.

Ps : Tried with retroarch stable and git

[lasers]

Update your database and try again. If it does not pick up, then that game might not be in the database. I think using Load Core and Load Content -> Select File have same effect of using that CLI command (then you can see it in the history too).

[ghost]

How I update my database? Sorry :s
I have tried already Load Core with Select Directory and Select File. It's not even in the history.

[lasers]

Main Menu --> Online Updater --> Update Database.

[ghost]

Not a single progress.
Forget to run from console sorry :

RetroArch [INFO] :: Parsing CUE file '/ssd/RetroGames/roms/psx/Herc's Adventures (USA)/Herc's Adventures (USA).cue'...
RetroArch [INFO] :: Found first data track on file '/ssd/RetroGames/roms/psx/Herc's Adventures (USA)/Herc's Adventures (USA).bin+0'
RetroArch [INFO] :: Reading first data track...
RetroArch [INFO] :: Comparing with known magic numbers...
RetroArch [INFO] :: Found disk label 'SLUS-00298'

That's all I've got when I scan a directory with beetle_psx core loaded.

PS : Didn't have any problem with gba,n64 or even nses games.
PSS: I really don't have the Playstion.lpl created.

[lasers]

I see that game in the database. SLUS-00298 too.

You're missing at least this...
RetroArch [INFO] :: Trying to write to playlist file: /home/lasers/.config/retroarch/playlists/Sony - PlayStation.lpl

What does Directory --> Playlists say?
Check Settings --> Playlist too. Enabled?
Settings --> Menu (or User Interface --> Menu) --> Show History Tab --> Enabled?

[ghost]

I think everything is right, my playlist directory is here /ssd/RetroGames/playlists (but it wasn't even working before I move the directory location).
http://pastebin.com/fmgvBnz6

Ps: I don't have any more information even with stable retroarch with debug flags.
PSS : I think I have a problem there : /usr/share/libretro/data, can't find Playstation 1. But I just update the database. (I downloaded them from libretro-database, but still not working)

End: Ahahah... Really sorry guys, that was really a database problem, just didn't have the right to write the new database... Thanks :)

[lasers]

If you didn't receive "Trying to write to playlist file" then I don't know if I can help you with this one... as I am unable to reproduce anything that might prevent "Trying to write to playlist file".

I'm suggesting a last attempt.
Back up your config. Try again with clean config and latest git. See if it's still same.

[ghost]

Thats' okay, I edited my last message.
I have to run retroarch as root user to update database.
Didn't see any message error before I run another update with the console enable, maybe a gui message will be a good idea.
Anyways thanks for that :)

[sergiobenrocha2]

"I have to run retroarch as root user to update database."

Crap, why did you do that? Just move the database dir to ~/.config/retroarch/database/

Settings tab --> Directory --> Content Database

[ghost]

Because I don't see a reason to duplicate the database. All user will have access like that.
Or there is a point to have a database for each user?

[RobLoach]

You could also chmod 777 the database folder if you want to have it accessible to everyone. The better way is to use ~/.config/retroarch/database , as @sergiobenrocha2 mentioned. How many users are on your system?

[ghost]

Currently only one, it's more for the principle.
Change the rights to give the control to the games group will be more sweet.
I just don't have already did it.
Why ~/.config/retroarch/database will be so perfect?
But I think we are going out of the subject no?

[RobLoach]

Why ~/.config/retroarch/database will be so perfect?

Because it is always writable by the current user.

[lasers]

I had troubles with adding Sinclair - ZX Spectrum +3 contents today due to this bug. For some reason unknown, libretro-super does not compile fsuae and puae cores... only to find out that they are WIP cores.

Adding ipf to supported_extensions in some-random-core.info will generate the playlist.

[sergiobenrocha2]

https://www.reddit.com/r/RetroArch/comments/5sbg0r/retroarch_not_scanning_directories_and_creating/

https://www.reddit.com/r/RetroArch/comments/5radva/roms_arent_scanning_and_i_have_no_idea_why/

https://www.reddit.com/r/RetroArch/comments/5kb74j/retroarch_wont_scan_any_roms/

https://forums.libretro.com/t/scanning-problem/5944

https://forums.libretro.com/t/retroarch-does-not-scan-the-folder-with-zip-archives/8804

https://forums.libretro.com/t/7z-files-and-scanning-directories/6710

Is it possible to tag this issue to fix in the next release?