Devilutionx: proposal: change save archive format from MPQ to JSON

[mewmew]

This proposal is about updating the character save archive format (i.e. single_1.sv and multi_1.sv) from MPQ archives to JSON format. The benefit of this is better debug-ability, likely easier modability and extensibility (e.g. store @qndel pixel light sources) of the save file format. The downside is that with added extensibility, it is likely that mods may add custom fields of their own, making the save formats between mods incompatible.

The envisioned process is as follows:

1. Develop a tool for converting from the old to the new save archive format (MPQ -> JSON).
2. Implement support for loading these JSON files, and support loading both the old and the new format.
3. (optional) deprecate the old format by applying automatic conversion?
4. ....
5. profit.

[mewmew]

Copy from https://github.com/sanctuary/djavul/issues/3#issue-565558540:

Old format (MPQ archive)

Save files of Diablo 1 are stored as MPQ archives, and their contents differ based on game mode; Single Player or Multi Player game.

Single Player Save Archive

Save archive file name: single_NN.sv (MPQ archive)

Save archive file contents:

• hero: hero information
• game: game state information
• templNN: temporary state of dungeon level NN
• tempsNN: temporary state of set level NN
• permlNN: permanent state of dungeon level NN
• permsNN: permanent state of set level NN

Multi Player Save Archive

Save archive file name: multi_NN.sv (MPQ archive)

Save archive file contents:

• hero: hero information

New format (directory with JSON files)

Single Player Save Archive

Save archive directory name: single_NNN.sv (file system directory)

Save archive directory contents:

• hero.json: hero information
• game.json: game state information
• templNN.json: temporary state of dungeon level NN
• tempsNN.json: temporary state of set level NN
• permlNN.json: permanent state of dungeon level NN
• permsNN.json: permanent state of set level NN

Multi Player Save Archive

Save archive directory name: multi_NNN.sv (file system directory)

Save archive directory contents:

• hero.json: hero information

[julealgon]

Do we intent to provide any sort of protection against tampering on these files? I think the idea of using MPQ was also to offer some protection against people just manipulating stuff arbitrarily.

I'm not entirely sure myself if that's a good idea, just wondering if you put any thought into it.

[julealgon]

Additionally, the original files seem to be incredibly fast to save and load. Do you think we can keep that in place when converting to Json? Do we have any benchmarks?

[AJenbo]

There are so many tools available for editing saves and cheating in general that that isn't really a worry. My biggest issue would be compatibility, and having the ability to move games back and forth, without having to use a converter, seams very helpful, especially when testing and comparing.

On the other hand, it makes it easy to extend and debugging the game itself.

Performance-wise I'm not sure there would be that big of a difference. MPQ isn't that different from a ZIP, so the performance difference would largely be down to the speed of encryption/decryption vs encoding/decoding.

[julealgon]

There are so many tools available for editing saves and cheating in general that that isn't really a worry.

Fair enough. Was asking for curiosity to be honest, as I tend to agree.

MPQ isn't that different from a ZIP, so the performance difference would largely be down to the speed of encryption/decryption vs encoding/decoding.

The big difference I think would be in direct memory mapped structures vs json serialization/deserialization. It's hard to get more efficient than binary, but I guess PCs are up to the task today compared to 1996 😆

[AJenbo]

We do not directly memory map the structures, instead we read from a byte stream. This was nessesery for compiling for anything that isn't 32bit x86 (I think ARM 32bit might have been mostly compatible).

[julealgon]

We do not directly memory map the structures, instead we read from a byte stream.

Sorry, I used the wrong term here. I meant reading bytes directly into the native types without performing any conversions.

[AJenbo]

Well, that's why I would say the difference is between the speed of encryption/decryption and json-encoding/decoding those would be the more pronounced differences between the two implementations.