The Import Content feature appears to be broken. Selecting scan by folder and file yield the same result. The GUI indicates it's stuck scanning the first file it comes across and remains at 0%, never progresses and must be restarted. Also happens when running as the root user
[What you expected to happen]
[What is actually happening]
Appeared to start with 1.8.7
Git Version: fda235518a
[INFO] RetroArch 1.8.8 (Git fda235518a)
[INFO] === Build =======================================
[INFO] CPU Model Name: AMD Ryzen 5 3400G with Radeon Vega Graphics
[INFO] Capabilities: MMX MMXEXT SSE SSE2 SSE3 SSSE3 SSE4 SSE4.2 AES AVX AVX2
[INFO] Built: May 26 2020
[INFO] Version: 1.8.8
[INFO] Git: fda235518a
[INFO] =================================================
[INFO] Version of libretro API: 1
[INFO] Compiled against API: 1
[INFO] SRAM will not be saved.
[INFO] Loading history file: [/home/
[INFO] Loading history file: [/home/
[INFO] Loading history file: [/home/
[INFO] Loading history file: [/home/
[INFO] Loading favorites file: [/home/
[INFO] Parsing CUE file '/home/
[INFO] Reading first data track...
[INFO] Comparing with known magic numbers...
[INFO] Found disc label 'SLES-01135'
[INFO] [config] Saved new config to "/home/
[INFO] Content ran for a total of: 00 hours, 00 minutes, 00 seconds.
[INFO] Unloading game..
[INFO] Unloading core..
[INFO] Unloading core symbols..
do you have the databases?
I would think so, I've always downloaded them from the internal updater thing.
/home/
total 65M
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-r--r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-r--r--. 1
-rw-r--r--. 1
-rw-r--r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-rw-r--. 1
-rw-r--r--. 1
This is also happening to me with RetroArch 1.8.7 on Arch Linux with all scan options except Manual Scan when selecting a specific core, that runs as expected (takes some time to start but scans all roms of that core very fast).
Same issue here with RetroArch 1.8.7 on Arch Linux on x64 system (on both computers intel core i7 and Ryzen 5). Downgrading to 1.8.6 make the import content scanner working. As soon as I upgrade to 1.8.7, the import content scanner is stuck at 0% and nothing happens.
Manual scan works on both versions.
Same problem on Manjaro. CPU at 100%.
Zero issue here on fda2355 or c2b8b3f1. Are you all using GCC 10? If so, that's likely the problem. GCC 10 has a couple massively breaking changes that have been causing headaches for a number of projects.
Mint 19.3 64-bit, GCC 9.3.0
I don't know the version of GCC used by Arch Linux to compile RetroArch. I just installed it from the official packet manager pacman.
It's unlikely to be GCC 10 in that case. Unfortunately I can't come up with anything specific, but if you manage any theories, I'm happy to check them out. My primary assumption is that it's either a permission or dependency issue, but it seems strange that it would crop up based on retroarch version in that case.
I think it is related with the access to the games databases. On an existing content library created from v 1.8.6, after 1.8.7 upgrade, when I click on the I to access the database information for a game, RetroArch crashes.
I don't think it is a permission issue because the game can be launched through the Load Content menu. For example, the User folder in the game folder is correctly created when using the Dolphin core. Also, I didn't touch any of the .rdb files between upgrade / downgrade. The owner is retroarch, group retroarch. Service is launched as retroarch, new created files owner are retroarch.
Something changed between 1.8.6 and 1.8.7.
I also completely removed retroarch and all configuration files then installed it again. Folder scanning sill doesn't work.
Scan is progressing when there is no game database. Though, as the database folder is empty, after scanning is done, no game is added in the library. As soon as I use the online updater to download the database, the folder scan does nothing and is stuck at 0%.
While I couldn't say -why- it would be important, try executing chown $USER *.rdb from the directory where your database files are being stored. That will change ownership to your user. It seems like it's halting on permission denial for the database files. I did verify that removing readability on the file will cause the exact behavior you're referring to.
EDIT: Could also just do chmod 644 *.rdb to give read perms to all. But I'd like to know what the perms on them are already.
No change here when chown to $USER, all .rdb files already have r permissions for everything.
Same for the directory I assume? Seems unlikely that'd be different though.
This problem just appeared for me as well though it appears a little different. With the ozone GUI, I get a progress bar that goes to 50%, then stops. For example, if I'm scanning a folder with 18 items in it, the progress bar stops at 18/36, and then uses 100% of a CPU core indefinitely. The program isn't frozen, and I can still move through the menu. I left it running for 15 minutes, and it never got any further.
I couldn't find anything in the logs or in stdout. Changing permissions on the database folder and files didn't help, unfortunately.
The first 18 are the archives you keep the games in. That's how it behaved when I set my Atari 2600 rdb to 000 and tried scanning. If you look, it caps out CPU usage on one core while that's happening.
chmod 755 ~/.config/retroarch/database
chmod 755 ~/.config/retroarch/database/rdb
chmod 644 ~/.config/retroarch/database/rdb/*
=> same result, scanning is stuck at 0%.
If I remove all *.rdb files, folder scanning is working (no games are added though).
In my test case, I try to scan a folder containing 3 GameCube roms. One game is .iso, the others are .gcz. If I put only the file Amstrad - CPC.rdb in ~/.config/retroarch/database/rdb, progression goes to 67% and is stuck analyzing the .iso file and never ends. No games are added in the library (it is normal, games are not in Amstrad CPC db).
Then I add the file Nintendo - GameCube.rdb in ~/.config/retroarch/database/rdb, if I launch folder scanning, it gets immediately stuck at 0% on the first .gcz it founds.
I think that Retroarch hangs as soon as it tries to match the game and the database entry. But I can't understand why.
Is it possible to try with other db formats, like .dat?
Also, I use Xorg, not wayland. I think the title of this issue is misleading, it seems to affect many Linux systems (Manjaro, Arch Linux and Fedora, at least), Xorg or Wayland.
I can confirm on Arch Linux as well, only manual scan works. The Arch Linux package is built with GCC 10, can't build with another GCC, and clang is a no go as well since compilation fails :/
I confirm the issue too.
"Scan Directory" option stucks at 0% on the first rom.
"Scan file" option has the same problem.
Only "Manual Scan" works.
Same behavior if the rom is compressed (zip or 7z) or uncompressed (tested with smc file - SNES rom)
Database has all permissions.
Using Xorg.
Arch Linux up to date (05-30-2020)
Retroarch version 1.8.8 installed with pacman from official repositories.
It was compiled with GCC 10.1.0 / 64-bit.
:/
Ah... I didn't think there was any way the AUR version would use GCC 10 so soon, when it's breaking every project in sight. I'm not sure if there's an issue open yet to handle GCC 10 fixes, but I'd bet good money this goes there if so. Frankly, building with GCC 10 at this point is a bad choice.
I think that @anderberin is saying that Arch Linux offical repos use gcc 10 to compile retroarch (and maybe all other softwares). AUR is a special repo where packages are compiled on the local machine. And I can say that my local gcc version is @10.1.0-2@. So, from the official repos or through a local compilation, the problem should be the same. I hope the retroarch project will support gcc 10.
Ah, my mistake. Seems my understanding of AUR is a bit lacking. Thanks for that info. Regardless, almost every large project right now is choking on GCC 10 in some way or other. Once my CPU's freed up I'll test a build myself, since I do have GCC 10 installed already for this very thing.
I think that @anderberin is saying that Arch Linux offical repos use gcc 10 to compile retroarch (and maybe all other softwares). AUR is a special repo where packages are compiled on the local machine. And I can say that my local gcc version is @10.1.0-2@. So, from the official repos or through a local compilation, the problem should be the same. I hope the retroarch project will support gcc 10.
Don't know is this help.
I compiled and it is happening the same to me.
I compiled then in debug mode, ran and performed the import content by folder.
It threw me this in the terminal.
Archlinux gcc 10.1.0
Retroarch GIT version 0fedfa220f
file trace
EDIT: Now compiled again and it works. Apparently there is something strange with GCC 10
I downgraded its version to 9.3.0.1
https://archive.archlinux.org/packages/g/
Just built Retroarch with GCC 10.1.0, can confirm this is a GCC 10 issue specifically.
@osirase I recommend updating the title to accurately represent the issue at hand. Something like RDB files are unreadable when compiled with GCC 10.
Yep, can also confirm this issue when compiling retroarch-git from the AUR with GCC 10.1.0 installed. Scanning never gets past the first file, attempting to view the database entries in Information -> Database Manager -> whatever database file says "No Entries to Display". I only found out about this because Manjaro has recently started providing 1.8.8 through their repo with this issue hehe.
Is there a way to compile with an earlier GCC version in the meantime without removing GCC 10 from my system?
GCC 9 (and other versions) can coexist with GCC 10. Get that installed (not sure exactly what method is needed on Arch), then export CC=gcc-9 CXX=g++-9 and rebuilding should spit out a usable version.
I can't find a way to get GCC 9 installed so I opted for 8 instead. It compiles and the databases now work again. Thanks
Hello, Manjaro user here.
I downgraded to CCG 9.3 and scanning is still not working.
Given that you typed CCG rather than GCC, if you made the same typo in telling it which compiler to use, you likely compiled it with GCC 10 unknowingly. You can check Information > System Information > Compiler to verify.
I just made the typo here. I did GCC. I reinstalled RetroArch from Manjaros server so I don't think I rebuilt RetroArch correctly. I'm still working on it.
I just made the typo here. I did GCC. I reinstalled RetroArch from Manjaros server so I don't think I rebuilt RetroArch correctly. I'm still working on it.
I use Arch.
I downgraded GCC to 9.3 and downloaded the retroarch PKGBUILD file (https://projects.archlinux.org/svntogit/community.git/tree/trunk?h=packages/retroarch)
Then, I rebuilt it with makepkg... and installed the package with "pacman -U 'package name'... And it worked!. I don't know how is done in Manjaro...
You can check the GCC version used to compile with the command:
$ retroarch --version
RetroArch: Frontend for libretro -- v1.8.8 -- 9552f87157 --
Compiler: **GCC (9.3.0) 64-bit** Built: May 31 2020
or using the interface: Main menu > Information > System Information: "Compiler"
I do not now if this is related, but the content browser cannot detect rom or iso files anymore. It does not matter if the RetroArch is version 1.8.6, 1.8.7, 1.8.8, or built from master branch. The content browser just stopped showing iso .cue, .ccd files, only zip files are shown. Debian 10, all versions are built from source.
I start to suspect that what is broken primarily, are the database files downloaded from the Online updater. The code may have issues, but my v1.8.6 built was working before I cleaned up the ~/.config/retroarch directory contents.
Pull requests welcome. I am not on GCC 10 so I don't have anything substantive to say here.
To be completely clear, it's GCC 10 compiling retroarch that's an issue. I didn't recompile the RDBs using GCC 10, nor would that happen as part of the build process. The same RDBs that work when retroarch is compiled with GCC 9 don't work when I recompile retroarch with GCC 10. The RDBs are fine.
On Fedora 32 with GCC 10.1.1 the database browser only displays entries when retroarch is compiled in debug mode.
edit
-Og: works-O1: works-Os: works-O2: broken-O3: brokenWell, libretro-db can be compiled in standalone form - all the files are contained in libretro-db. Perhaps that is a better way of debugging these issues. If you run Makefile inside libretro-db, it will compile a couple of standalone executables that will allow you to read RDB files, do queries on them, and read cursors too.
On Fedora 32 with GCC 10.1.1 the database browser only displays entries when retroarch is compiled in debug mode.
edit
* `-Og`: works * `-O1`: works * `-Os`: works * `-O2`: broken * `-O3`: broken
Does compiling in debug mode negatively impact the performance of RetroArch, or is the impact only during the compilation itself?
It would negatively affect performance in RetroArch. I really would not recommend using it at all.
As well as having the lists creation fail, it seems that the image updater is failing as well.
I hope someone can fix this issue soon, I deleted my lists earlier planning to recreate them, and now I can't. :(
I guess we have to use gdb to find where the code should be fixed...
This is not a bug in GCC optimization. There is an odd behavior in those optimizations but i think the underlying problem is a buffer overflow in RetroArch code.
GCC10 in -O3 and -O2 mode removes the following conditional statement
if (strncmp(header.magic_number, MAGIC_NUMBER, sizeof(MAGIC_NUMBER)) != 0)
in libretro-db/libretrodb.c.
The generated assembly code do the comparison and then directly jump to the error label regardless of the value.
The suspected bufferoverflow is in struct libretrodb_header declaration.
typedef struct libretrodb_header
{
char magic_number[sizeof(MAGIC_NUMBER)-1];
uint64_t metadata_offset;
} libretrodb_header_t;
IMHO should be
typedef struct libretrodb_header
{
char magic_number[sizeof(MAGIC_NUMBER)];
uint64_t metadata_offset;
} libretrodb_header_t;
The fix is really small but i can do PR if you prefer.
I have compiled retroarch with the suggested changes by @theowhy and both the a) Database information works and b) Scan folder/file works as expected.
gcc version 10.1.0 (GCC
[INFO] RetroArch 1.8.8 (Git 439aee616e)
[INFO] === Build =======================================
[INFO] CPU Model Name: Intel(R) Celeron(R) N4000 CPU @ 1.10GHz
[INFO] Capabilities: MMX MMXEXT SSE SSE2 SSE3 SSSE3 SSE4 SSE4.2 AES
[INFO] Built: Jun 14 2020
[INFO] Version: 1.8.8
[INFO] Git: 439aee616e
diff --git a/libretro-db/libretrodb.c b/libretro-db/libretrodb.c
index a82e2399b1..4976bdcbf2 100644
--- a/libretro-db/libretrodb.c
+++ b/libretro-db/libretrodb.c
@@ -75,7 +75,7 @@ typedef struct libretrodb_metadata
typedef struct libretrodb_header
{
- char magic_number[sizeof(MAGIC_NUMBER)-1];
+ char magic_number[sizeof(MAGIC_NUMBER)];
uint64_t metadata_offset;
} libretrodb_header_t;
Hey that's awesome news!
We'll push the change through then.
EDIT: Pushed.
So this has been in master now since 12 hours ago, can the people in this thread that were experiencing issues re-test this issue so we can see if it is resolved? Thanks.
While not specifically on Arch, I did reproduce the issue locally and have confirmed that I can now build with GCC 10 without the issue rearing its head.
That's cool to hear. What about @BugReportering ?
Fedora 32, GCC 10.1.1, it's working now. Thank you!
It is working on Arch now.
GCC 10.1.0
The package on official Arch repository was updated.
It's working now.
alright, seems to be resolved, so I'll close this issue.
Big thanks to @theowhy for tracking down the issue and identifying the fix!
Fixed! @theowhy are you able to link to the pull request that fixed this issue?
Looks like it was this commit: 685948368e6b28b97e1b91bdfe5921519359a033
I didn't do any pull request, the fix was directly made by @twinaphex in the commit pointed by @Sanaki.
Most helpful comment
This is not a bug in GCC optimization. There is an odd behavior in those optimizations but i think the underlying problem is a buffer overflow in RetroArch code.
GCC10 in -O3 and -O2 mode removes the following conditional statement
if (strncmp(header.magic_number, MAGIC_NUMBER, sizeof(MAGIC_NUMBER)) != 0)in libretro-db/libretrodb.c.
The generated assembly code do the comparison and then directly jump to the error label regardless of the value.
The suspected bufferoverflow is in struct libretrodb_header declaration.
typedef struct libretrodb_header { char magic_number[sizeof(MAGIC_NUMBER)-1]; uint64_t metadata_offset; } libretrodb_header_t;IMHO should be
typedef struct libretrodb_header { char magic_number[sizeof(MAGIC_NUMBER)]; uint64_t metadata_offset; } libretrodb_header_t;The fix is really small but i can do PR if you prefer.