Pcsx2: Suggestion: SQLite for storing everything (per-game configs,hacks,user preferences,game metadata)

[XenonPK]

After looking into GameDatabase.h and GameDatabase.cpp it seems to be a "pseudo-database" that uses hashes to find information, it does it's job. However wouldn't it be "cleaner" to use SQLite?
I know it's kind of a radical change to a lot of stuff within the emulator, and arguably slower(marginally) than the current low-level implementation, but It would also make development more versatile, looking forward.

1. For example, we could have an online database, (much like the wiki) , that people could contribute to (uploading game-specific settings from within the emulator with attached description and/or in-game screenshot) for every stable version.
2. Making game-specific settings the default, that are then overridden with user-defined settings (capable of reverting to that game's default settings at any time).
3. Periodically update the local games settings to match those found online(user specified changes take priority, ofc).

These three examples, if implemented , would make people contribute a lot more to the wiki.
Also , feel free to correct me if I misinterpreted the code.

[gregory38]

the pseudo db => bin/GameIndex.dbf
At least, it can be read with a basic text editor. Same for others config.

I guess you will need several db otherwise you need to handle a merge between user db (pad config for example) and general db (automatic gamefix).

[XenonPK]

Broadly, It would be like this:

User Config DB
->Gsdx Table
->SPU-2X Table
(...for each plugin...)

Gamefix DB
->Gsdx Table
->SPU-2X Table
(...for each plugin...)

• Use the entries as a stack, when we need to set the configuration, fetch the entry from the User Config DB first and then the Gamefix DB, into the "stack" we then pop the Gamefix DB entry from the stack and apply it, after that, apply the entry from the User Config DB.

I can see this causing some potential mayhem already , so I think a safer bet would be:

• When a game is loaded for the first time, read the config from the Gamefix DB. If the user changes the configuration, create a new entry in the User Config DB with all the currently defined values and from now on, for that game, pcsx2 no longer looks at the Gamefix DB, because everything is already defined in the user cfg, this also means, unless the user "reverts to default" the settings for that game won't change.

I think the best way of doing it is a hybrid of these two.
Use the first method when changing things like the internal resolution or toggling fxaa on/off because those are not fixes.
And use the second method for options like the half-pixel offset and crc hack, that are meant for fixing problems.

[avih]

The problem with storing config at a DB is that you can't edit it anymore by hand. It might be worth it if there's a good need for it, but I don't see such need (enlighten me).

Bottom line, what would PCSX2 and its users gain from moving the configs to a DB?

[XenonPK]

PCSX2 (as a project) would be able to automate the wiki and make it more convenient for users to contribute.
Users would be able to boot up a game listed as playable on the wiki and play it with settings tested and verified by other users to be working on that stable version without touching anything.
The most important thing (in my view) is making one config per game the default , as there is plenty of users that might find it convenient.
In a very specific example I dealt with recently, using ICE to integrate emulators into steam BPM I ran into a problem with God of War II and a bunch of other games that required specific hacks in order to fix ghosting and other issues.
I used the --cfgpath=$CUSTOM_GOWII_CONFIG_DIR in steam's launch arguments for that game on my account but then I realized I had to add it on all accounts in order to have it work for everyone.
Having an internal DB , with per-game settings solves that, and other similar minor end-user gripes.
The text file problem can be worked around by adding an export/import mechanism that exports json files , allowing the user to export the current game config into json, alter it at will and import it from the gui or command line arguments.
However, given all this, you are partly right, pcsx2 works fine as-is , and I do understand the if it ain't broke don't fix it point of view, but it provides some improvements to the user experience and potentially makes the emulator plug'n play for most games.

[MrCK1]

I don't think a "one-size fits all configuration" is ideal in an emulation environment. We don't know what hardware the player is using.

The default settings are intended to be the most compatible. The most recent hardware will usually have no problem running it; and the user can tweak if they feel necessary/are technically inclined to do so.

I think that higher level preset configurations will just narrow the "compatibility range" further and create more support tickets.

[avih]

All of these goals are viable (to different degrees), but none of those requires the settings to be stored at a DB, and all of those suggestions have many other issues to solve unrelated to where the data is stored.

A DB is just a method to store data, no better or worse than directories and INI files, except sometimes easier to search and cross data etc - if there's such need - but much harder to edit without the right tools. Since it seems to _me_ that you're mostly focused about config-per-game (which has been suggested countless times before), I don't see how a DB which stores config per game is any better than a a set of config dirs where each one is for one game.

I just don't see (yet?) how it helps TBH, but I can definitely see how it hurts.

[ssakash]

I agree with @avih , all these implementations are feasible with the current GameDB.

Also currently the GameDB doesn't support storing the values for the options which are exclusive to the plugins. However it should be possible to implement with some plugin -> core export functions.

[gregory38]

Well a db will be handy to avoid hundred of config file/directory. However it will be a nightmare for user to manage it. Various options are hidden, so you still need to update the ini file manually (otherwise gui will explode).

[avih]

Well a db will be handy to avoid hundred of config file/directory.

So is the current single-file GamesDB.

[ssakash]

However it will be a nightmare for user to manage it. Various options are hidden, so you still need to update the ini file manually (otherwise gui will explode).

What do you mean by various options are hidden ?

Though I do see why some people prefer multiple configuration files over a single GameDB. It looks much cleaner and the individual files could be looked through the git online editor properly, it would also be more sensible to have individual configuration files if support for lots of options are added. ( Plugin exclusive options )

However the benefits aren't worth enough to move from a single GameDB to multiple configuration files, at least that's my opinion.

[avih]

A DB enables efficient and structured data search, data manipulation, etc. Configs are just mostly static and have no need for efficiency as they're only read/updated on init and configs changes, and config changes are not in any hot code paths.

It seems to me that the OP main reason to want a DB is that it's a single file, but that's just not a good enough reason in my book, not to mention the elephant in the room that is that config-per-game subject which has been discussed many times and never took form.

If we ever decide to support config per game, the way we store it would not matter much anyway, and suggesting that moving the storage to a DB would make it easier to support config-per-game is a very incorrect statement.

[XenonPK]

Well, I meant it as a suggestion, when I thought of a structured way to store values, it was the first thing that came to mind, and I guess if it is implemented for one thing, it would make no sense to have different implementation for the rest of the use cases.
Of course we cannot expect the user to go ahead and create an SQL script every time he wants to change the configuration, but that can be abstracted , an alternative (similar behavior without db) would be libvirt, that uses xml and import/export command line tools, automatically validates the imported files and applies them.
Regardless of the backend, it's essentially an abstraction, in the case of libvirt , all it does is make it easy to manage virtual machines.
An SQLite implementation would create an unified way to manage a potentially big amount of configurations without creating a mess of files and directories, that can then be abstracted in a similar way that libvirt does with vm configs.
I am not saying it's the only way to do it, you are more experienced with the project than me, but, if pcsx2 were to go that route, it would be one way to do it.

[jcdenton2k]

Don't we already have this? Isn't that what the 'presets' were added in for this very purpose? A 'set it and forget it' type of thing?

Generally speaking most people are fine with the 'Safe' and 'Safest' presets then they can fiddle around with settings based on the helpful UI popups when hovering over them that explains what it does and how it works. Kudos to the UI people for putting those in; they are awesome!

So by default you are on the 'Safe' preset when you download PCSX2 and then you have to be technically-proficient enough to go in and set your keybinds and audio/video options as needed or you will have a suboptimal experience with just the defaults (since not everyone uses the same keybinds).

I agree that there is no 'one size fits all' and the existing 'presets' options do the job already. I believe if we could perhaps make it easier to contribute and test out 'Automatic Game Fixes' then that would be nice. It is not currently possible within the UI to even know what Automatic Game Fixes have been applied. Is it in the console output somewhere? Maybe we can add in a separate category for this in the console if needed?

This way, once we know what Automatic Game Fixes are being used then we know if any additional ones are necessary or if some fixes are no longer necessary for that specific game (due to advances in emulation accuracy and compatibility).

As far as custom configurations per game, there are already PCSX2 frontends that handle this quite well that may be useful to consider for your users.

The Unix/Linux philosophy is essentially every program does ONE thing really well. That way you have multiple small programs/scripts being used and called by other programs without the need to duplicate functionality. This seems to be the same philosophy that RetroArch has been partaking in with their universal GUI and separate emulator cores setup.

The reason the devs haven't merged PSP and PS1 functionality into PCSX2 is to reduce code complexity and maintenance. Do one thing and do it well then have plugins and other stuff to do the grunt work when needed.

I wouldn't mind an easier way to manipulate the GameDB but SQL may not be the best choice to use. This may take some time to discuss and mull over.

[avih]

It is not currently possible within the UI to even know what Automatic Game Fixes have been applied. Is it in the console output somewhere?

When automatic gamefixes are enabled - the console window title shows the number of automatic fixes (e.g. [2 fixes]), and the console log will show the fixes themselves soon after the game boot, e.g. the following shows up for ICO PAL:

(GameDB) Changing EE/FPU clamp mode [mode=2]
(GameDB) Changing VU0/VU1 clamp mode [mode=1]

[MrCK1]

Maybe we could print definitions with the mode number to reduce confusion?

ex: (GameDB) Changing EE/FPU clamp mode [mode=2] (Nearest)

[avih]

Not likely. The name of the value belongs to the GUI, and the value itself is applied (and printed to the console) at the PCSX2 emulation core which knows nothing about how it's being displayed to the users at the GUI, if at all.

[jcdenton2k]

@avih Thanks for the explanation. I'll keep a closer eye on the console and enable a few more of the options while I load up various games to see what happens. At least the DB file can be read in a text editor so it should be easy to just look up whatever is needed.

[ssakash]

Not likely. The name of the value belongs to the GUI, and the value itself is applied (and printed to the console) at the PCSX2 emulation core which knows nothing about how it's being displayed to the users at the GUI, if at all.

I don't see the problem with that, you can still hard-code the names of the values.

Not really related to the topic, though IIRC there was a certain bug where the console window title fails to update when Full boot is used. (I'm not sure if the issue is still present)

[jcdenton2k]

I can confirm the DataBase file opens up fine in WordPad (with proper formatting) and likely opens well in other open-source text editors as well. I suppose that the OP could try a simple parser script if they want to modify it for the purposes of their community. After all, this project is GPL so have at it :)

[avih]

Shall we close it as WONTFIX for now? we can always reopen it later or file another bug if it become relevant. For now I think so far we agree we don't want that.

[jcdenton2k]

Please feel free to close this issue. It has already been answered.
If @XenonPK wants to fork the emulator and do so then they are permitted to do it via the GPL license.

[avih]

Please feel free to close this issue.

That's a thing the OP could say, not another user which never wanted it to begin with ;)

But if there's an agreement among the devs that we're not doing this for the foreseeable future, then we can close it for now.

[avih]

OK, Closing for now. Feel free to reopen or request to reopen.