Citra: Using PhysicsFS for game, updates, and dlc

[TheRealYasri]

I am posting this here because I am not sure if any of the devs of citra have heard of this library.
https://icculus.org/physfs/

The standout parts that I think would be useful for citra are there...

1. For file reading, the programmer lists directories and archives that form a "search path"
2. a new archive that will override a previous archive on a per-file basis

You could when running a game file grab the update/dlc archive files in the same directory and put them in a single, transparent hierarchical filesystem for citra to use and it should do the work of picking the version of files from each archive for you.

I would think this would be easier then how cemu is doing it, were files have to be in certain directories with certain names or it will break for the end user. This way the end user just needs three archives in the same folder with the right names and citra can handle the rest internally.

This is all from a person who knows enough to be dangerous and had to fuss a lot with cemu to understand and to get things to work right, when I first tried to use it. Not sure totally how you all are planning on handling everything to do this.

Hope this was of some help /cheer

[yuriks]

Hi, thanks for the suggestion!

However, the way 3DS updates work and are implemented, Citra doesn't need to do any overlaying of files itself. We just provide the individual archives to the game, which then has to take care of loading the appropriate files. So I don't think PhysicsFS is applicable for us, sorry.

I'm gonna close this, but feel free to comment if I misunderstood.

[TheRealYasri]

How I understood the route you were going with was taking a CIA file "installing" it and placing the results in a folder that sticks around. Like cemu does with this as a example
mlc01\usr\title\\\aoc\
http://compat.cemu.info/wiki/Tutorial:How_to_install_a_DLC

What I am saying is take the files there and instead of keeping them loose, you archive them up and thru psyfs you take the game rom/archives and have it place them in a virtual fs in the correct spots and citra /games will not see it as anything more then just a standard correct directories(unless the devs want it too). I would think this would be more transparent to a end user and have less support requests. All they would need to have is a max of three files in the same directory named correctly. They would not have to dig into wiki's on how to find out where to place files and all that. So they could keep thing separated and more manageable.

/citra/citra.exe - Program Folder, that can just be nuked when a new version comes out
/games/GameName.3ds - The game itself
/games/GameName.3dsu - Game Update Archive
/games/GameName.3dsd - Game DLC Archive
/emusetting/citra/ - User changable folder that holds citra setting, saves, and other changing stuff

This way all the static stuff for a game would in the games folder as those three archives, anything that is modified would be thrown into the citra folder in the user directory and you would not put anything special into the program folder that cant be swapped without a second thought And you could have psyfs grab any needed files from those three places to build the virtual fs to run the game from and would give files in the user directory priortity over something out of the static archives if there is a conflict.

As a example if you want to nuke the cemu program folder for instance to install a new version it will also take the update/dlc and alot of other stuff with it unless you know to move them out. Right now for cemu I use a folder Junction making app to keep my files out of the program folder and on a different drive. Reason I am using cemu as a example is because you all seem to be heading down the same path. So do projects like retro arch to a extent.

File extention names, compression choice and the like would be up to you all to decide what to use. In the ideal world you would dump files off a 3ds in some form that could be put back onto a 3ds if you so choose but in this case you would drop the file/folder into a program that make a archive like above in a way in which you can reverse it.

Now if you could take the stuff off the 3ds and look into the file to get what you need without having to "install" it permanently, you could skip the hole "install" and backup the results step talked out above.

I hope this explains it a bit better, had to rewrite this like three times -_-::

[shinyquagsire23]

Well, I don't really think physfs would even be that useful for updates to be honest (having been the one to implement them). Updates aren't really one file, I mean yes .CIA files are a single file but as it is, but it's a container for multiple larger files. And for one of those files, there's the actual game update content which doesn't totally override the base game. The ExeFS and exheader partitions override the base game, but the RomFS from both games coexist and the update ExeFS just has to do the management between both RomFS partitions. Also, the game handles all RomFS parsing and RomFS is just one big blob of data, with Wii U all content/ files are just files on the filesystem, and IOS is able to overlay the update content/ on top of that much like PhysFS would do.

DLC is an entirely different story, with DLC the game doesn't actually say "hey, get me this file", it's more or less "Hey, open content index 0x8B from TID 0004008C000EDF00 and get me the RomFS blob," and on hardware content index 0x8B would translate to one of many separate .app files. The same actually goes for games on the home menu as well, the content for the manual of a game is entirely separate from the content of the game itself, and the home menu only asks for information of TIDs.

Would it be possible to load update and DLC data from a single CIA? Totally. Would it be trivial though? Maybe updates, but DLC is an absolute nope. The Application Manager service doesn't really have context of where the game is running, it only deals with the SD and NAND filesystems.

Comparisons to CEMU aren't totally fair either since everything on 3DS doesn't reside on the filesystem as separate files, most everything has its signature verified on 3DS (unlike Wii U) which means content comes in giant integrity-checked files. As it is, Citra's update/DLC implementation is more true to actual hardware and how 3DS games and applications expect things to be. RPCS3 does similar with its game updates if I remember correctly, I'd imagine there's similar stories there. Even for updates and DLC though, PhysFS would be unneeded overhead IMO.

[TheRealYasri]

As I write this I think I can see/understand how psyfs would not be useful, going to leave the old reply below for some context of my thought process and my new reply first.

---New Reply---
How I understand you, when you install updates and dlc, it puts all of it into a single file on the host computer that is the RomFS that combines all the updates/dlc installed as one big blob. It then takes that and the game which it sees as another Romfs blob. My question is will each individual games update/dlc installed blob be its own thing that can be sat in a directory next to the .3ds file, its matched for. Or will citra install several different games updates/dlc as a single uber blob.

If you can have each games dlc/updates as its own file blob that is used, that would be the end goal I wanted. So each game you run, it would act like a 3ds with just that game running with only its updates/dlc install on the sd card or internal storage.

The reason I made the topic was because of how cemu handles it and how next to impossible it is to figure out unless you go on a wiki to understand where to put tons of files and if anything is misspelled or put files in the wrong place, it will just does not see it. But if your not going that route and instead all you need is .3ds file+update/dlc blob or a game/update/dlc blob. That is all I wanted.

/cheers

----OLD REPLY---
Ok just read your reply and I think I over explained and mixed together to many possible features that could be used, but arent really the point, just extra things to use if needed. Or I am just being dense today and not totally getting it >.<;;;

I will list a few questions. Because I think my overall idea would be a boon to people using citra but I get I am not the one writing it. Just want to have a good dialog, which we are having so far ;)

1. When you "install" a update or dlc thru citra, will the end product made on the host computer be a single file or directory of files. Sounds like a single file that is the 3ds filesystem.
   1a. Can these installs be safely backup/restored/deleted individually on the host computers filesystem without citra or a game going nuts next time they are ran. Or will it be a giant mushed up pile.

2. Will there be a way in citra to keep the folder it uses for the installs elsewhere outside of the citra program folder. Without having to use workarounds like symlink or similar.
   2a. (Optional) Have a way in citra to select a folder to put all user made files, like save, config, etc in a selectable folder instead of the program or user directories, would be nice.

3. If everything is movable after install and has multiple files, that need to be in certain directories for the game or citra to see and use it. For the end user to be able more easily manage things and be less error prone on their end. I think it would be easiest to archive those files up.
   3a. Citra would grab the archives when a game is loaded and internally see the files in the correct directory structure.

4. The reason I think psyfs would be useful is the same reason it is useful for game mods. Each game would be in its own silo which can not get out. It would overlap the game,update, dlc archives as static files on the virutal filesystem and any files that are changed/created would be put in the users directory into a folder just for that game.
   4a. So the next time the game is ran it would mount the game then updates and then dlc which are the same as the first time. And then overtop all that it would mount the folder in the user directory with anything that has been changed, like saves etc.
   4b. The game/cirta should not see anything going on, it will just think its the only game installed.

5.Some games do this for mods. You download the mod and just throw the archive/s into a folder. Launch the game it sees the mods, then select to load one or several of them, it then transparently to the game mounts the archives needed along with needed program and user directories with any file changes made are put into a folder for each different mod so they do not effect other mods down the road.