Mtasa-blue: Poor engineReplace* performance

[Wolfee-J]

Describe the bug
engineReplace* has very poor performance in comparison to standard SAs loading and replacing. When replacing any sort of model there is always a FPS drop, and in some cases the entire MTA client simply freezes up. While this is not a massive issue with smaller amounts of things being replaced when larger maps are being created it can take up to 15 minutes for all of elements to be loaded in fully.

To reproduce
https://github.com/CodyJL/JStreamer
https://github.com/CodyJL/JVice-City
Extract Objs and [VC} into your resources, start Objs and VC2, join and watch it take up to 4 minutes to load.

Expected behaviour
engineReplace* should replace in the background in pulses synchronously as San Andreas does; a map like VC (270 mb) should take at most 30 seconds and the client shouldn't completely freeze up.
(MTA loads up and joins a blank server in 40 seconds flat, SA being a 4GB game is nearly 17 times bigger)

Screenshots

-- Here's a graph showing how SA should load --

Additional context
Vice City takes 2.7-5.3 minutes to load (270 mb, 2866 unique elements)
Liberty City takes 4.3-6 minutes to load (1833 unique elements)
Alien City takes 27-40 minutes to load. (Large, but less then a GB) (5305 unique elements)
DFFs/ TXDs/ COLs replaced average less then half a mb.

Replacing when an element streams in cuts FPS in 4ths and some times freezes up the client entirely.

[Addlibs]

Would enabling asynchronous loading work with engine replaces, or is that only for models loading when streaming in?

[Wolfee-J]

Only when an element streams in.

[Wolfee-J]

Enabling asynchronous loading actually makes it 2x slower.

[qaisjp]

What do you mean by "slower"? As in, "takes longer to load", or "freezes the game for longer"?

Note that the wiki page says:

This function enables or disables asynchronous model loading. Enabling asynchronous model loading may reduce the small pauses that occur when a new model is displayed for the first time. However, it can cause the new models to appear slightly later than they might have otherwise.

[Wolfee-J]

Freezes for longer; goes from 2.7 minutes to 5.8.

[qaisjp]

Interesting. I wonder if it's the disk read that's causing the freeze. We should run a profiler.

[Wolfee-J]

Disk read is normal while joining; infact, it does not seem like MTA uses many resources at all when connecting. Memory usage shoots up to 800mb however that is the extent of it.

[tederis]

I think there should be the way to load RW resources in a separate thread and only then actually replace a model. Judging by recent experience a HDD is the bottleneck in such issues.

[AlexTMjugador]

I agree that HDD performance is a bottleneck in some way, but the way I see it, it probably isn't the bottleneck what causes these differences with unmodified GTA: SA load times.

AFAIK GTA: SA is a single-threaded game, and doesn't spawn a new thread for resource loading, but still it manages to show much better load times. So, even if moving that to a new thread improves performance a bit (because in the end the engine is not thread-safe and things have to be serialized), it doesn't appear to be the whole solution to this.

I think the best we can do now is profiling, so we know where the bottleneck could be, as @qaisjp said. Or perhaps look at the good old MTA: Eir branch to see if it has fixed already?

[tederis]

GTA:SA uses a bunch of techiques to achive max loading performance. First of all the RW resources stored in a compact IMG archive. It eliminates excessive actuator's work so instead of jumping from sector to sector we have almost continuous reading process. In the second place RW reads file not entirely in a moment but splits it into small pieces and then reads them consequentially. And it's absolutely opposit to what MTA is doing. The MTA does reads individual files that requires a lot of HDD movement. Moreover a file will be loaded at once without any slicing and especially threading. So I think there no way to smooth loading with the current implementation.

[AlexTMjugador]

Thank you for your insightful response. It gives me the idea that implementing IMG archive format support with streaming for loading a bunch of RW resources in MTA could translate to a very noticeable load performance boost for these use cases. But that, of course, exceedes what the current file-by-file implementation is designed to achieve, so it definitely is some heavy duty to do 🙂

[Wolfee-J]

I think even without IMG loading pulsed loading could massively speed up the loading times. I have noticed that engineLoad* is incredibly quick while engineReplace* is incredibly slow due to it forcing all content to load within the same frame. However IMG loading could further speed up loading times by allowing it to load all of the data from a single location instead of jumping around between multiple loaded files.

[TheNormalnij]

engineReplace* has good performance. My world loader need 20 seconds to load 800 mb. It is possible to pack all files in one, and you need to optimize your scripts. There are no problems. It works now, and i can proof it.

[Pirulax]

Its totally possible to just add a callback to engineLoad* functions, and load the files on a separate thread. Would that help?

[Wolfee-J]

Would potentially help. When loading them the MTA client has a really heavy pause and puts a lot of load on the single core (Also excuse the fowl language in previous comment, was going through some crappy times)

[PlatinMTA]

engineReplace* has good performance. My world loader need 20 seconds to load 800 mb. It is possible to pack all files in one, and you need to optimize your scripts. There are no problems. It works now, and i can proof it.

send proof ty

[TheNormalnij]

I pack files of vice city map in IMG archive. Map loaded in 6 sec. Wolfee-J's map loaded in 30 sec.
What i do:

• I pack files in one big. IMG is good for this, TAR archives maybe better. File operations with small files are to much slow (HDD need ~12 ms to get access to file! It's 78 sec for 6500 files). It's visible when MTA downloading small files and when client will be math CRC hash.
• I load content into memory sequentially as it is located in the archive. it's faster than loading it separately.
• Models will be replaced without any special magic
• I use lua files to store objects positions and replacement data.
• I use lua optimizations: local variables, memoize, minimal oop...
  Callback to engineLoad* will not speed up loading 6500 files. But it will not lock main thread.

^{sorry for my english}

Edit:
You can also optimize textures. With this Wastelands map loaded in 1 sec on my machine.
see tederis comment about HDD.

[Pirulax]

Well Ive looked at the code..
It loads the whole file in the netcode, then it checks if the hashes match(thats again 2 loads, as it calculates the CRC for it as well, even tho its not used), then, it loads it again into memory for GTA.
No wonder why its so slow, especially on a HDD.