Trinitycore: World Server Memory Consumption

[Animalthug]

Description:

World server high memory consumption

Current behaviour:

Clean build, world server is consuming 2.5GB of memory.
https://image.ibb.co/dgP8Yc/consumption.jpg

Expected behaviour:

I noticed when I initially started on a clean TrinityCore source (aprox month ago) world server was at 800mb-1GB stable with scripts. Now I have noticed when compiling with scripts it peeks at 4GB and without 2.5GB which seems extremely high. I am uncertain of what the "standard" would be, thus I assume it is an issue. Can someone please confirm? Thanks

Steps to reproduce the problem:

1. Compile clean source
2. Run MySQL database
3. Run WorldServer.exe
4. CTRL+ALT+DEL view Task Manager - Running Process - Worldserver.exe

Branch(es):

3.3.5

TC rev. hash/commit:

TrinityCore rev. 2a4541803223 2018-04-23 20:33:14 +0200 (3.3.5 branch) (Win64, Release, Static)

Operating system:

Windows 10 64bit

[Kittnz]

BaseMapLoadAllGrids && InstanceMapLoadAllGrids = 1?

[Animalthug]

Both are = 0

Everything is clean, have not touched a single config (other than MySQL access) or part of the code.

[Aokromes]

If database grows a lot obviously core will take more ram.

[Killyana]

7.x and 4.3.4 uses 1.1GB each

[funjoker]

2.5GB is "high" ? :D
Google Chrome almost uses more

[Animalthug]

I can understand the growth part, that is why I asked. But does it seem feasible in the current state to use that amount of memory? What should be the expected amount, just curious, as I am running a VPS with 4GB total, something to base future upgrades off of, thanks @Aokromes

[Killyana]

If you starts the server for the first time after rebooting your PC you will notice:
"Starting objects pooling system..." taking too much time, more than 50% of the server loading time, it's probably related to that, Pooling + dynamic spawns needs rewrites anyway.

[Animalthug]

The only way I noticed this issue was within the past few days I notice the server would pause longer on pooling and query than normal, as well lag in the server that was not due to latency so I investigated further and remove all scripts/edits and started with a clean source, after realizing that was not the case I noticed the memory consumption and realized the worldserver.exe was maxing out the VPS memory (with scripts).

[Animalthug]

What I mean by "not the case" is that pooling/query still took as long to load as I was used to with or without scripts.

[ccrs]

Everything is clean, have not touched a single config (other than MySQL access) or part of the code.

[...] and remove all scripts/edits and started with a clean source [...]

the plot thickens

[Animalthug]

analyzing complete? @ccrs

The build is clean from source.

This means, completely deleting the entire source folder and cloning the source again, which removes anything other than the source files.

[ghost]

i waill to tested on linux

use ram for last tc : 56.0 % = Cloner: 2a45418
use ram for old tc: 34.6 % = Release: TDB64

0 player onlines offline mod 127.0.0.1

image:

[jackpoz]

could you post your worldserver configs (removing the username and passwords) ?
I know Massif (by Valgrind on linux) is a very useful tool to see which objects are using memory, probably VS has a similar feature.

[ghost]

I did not make any changes worldserver . configs

Just tested for help with the report

offline mod on local: 127.0.0.1

[DoctorKraft]

can you try with CacheDataQueriesat 0 in your worldserver.conf?

[ghost]

Did not make a difference

If I tested my mistake, test yourself

What I've tested and I think it's having trouble in the new version

[Rochet2]

@irgc what is the commit of this "older tc" you compare to?
EDIT: I see you edited your post to include the info.

[ghost]

@Rochet2

my bad

old = older:D? :laughing:

That is old this > https://github.com/TrinityCore/TrinityCore/archive/TDB335.64.zip

That is last this > 2a45418

[Rochet2]

OLD rev ed63d15dfe956360851018c55a61d6d27dd743c4
Memory graph: https://i.imgur.com/G72aTQg.png
Function memory:
https://i.imgur.com/BjC3PpL.png
https://i.imgur.com/6l3PgC2.png

NEW rev 71188062314ee94402d9d3e228fd5b46e8865b18
Memory graph: https://i.imgur.com/lib9O11.png
Function memory:
https://i.imgur.com/JLVPnZE.png
https://i.imgur.com/7lwV0Gi.png

DB compare: https://i.imgur.com/cDjHVJd.png
Startup memory graph compare: https://i.imgur.com/tZFjx0J.png
Task manager compare: https://i.imgur.com/LgvYY9a.png

Some conclusions:
DB has not really grown that much, only 0.4MB difference.
Memory graphs show that both are at 1.7GB. (which is probably a ceiled value)
Task manager shows real RAM consumption for old 1677MB and new 1677MB.

settings:
mmaps disabled, vmaps enabled, CacheDataQueries = 0, separate clean DB for both, 64bit compile

[Rochet2]

Updated my post above, would say I cannot reproduce based on that.
Especially see the task manager compare.
The difference I had before was due to launching from different folder with different config on for the task manager compare. Now that they have same config (with different DB) they match just fine, like all other data matches.
DB size and RAM usage and all seem to match between the core from 18.2 and 23.4

However ofcourse this does not tell anything about actual usage during runtime / real usage.

Update: It seems that after idling for some time (lets say 10-20 minutes) the RAM usage has dropped a lot ..
And that is by the new core from 23.4: https://i.imgur.com/hwb8Dqy.png, https://i.imgur.com/QljgUOd.png https://i.imgur.com/UrBGLio.png
Attaching debugger shows that the programs are still using 1.7GB, so the RAM usage must have dropped because of memory packing or moving to disk.

[Killyana]

I always get 2.7GB of ram usage when the server starts: https://imgur.com/a/TJicjTq
```
TrinityCore rev. 71188062314e 2018-04-23 23:35:31 +0200 (3.3.5 branch) (Win32, Release, Static)
Using SSL version: OpenSSL 1.0.1e 11 Feb 2013 (library: OpenSSL 1.0.1e 11 Feb 2013)
Using Boost version: 1.59.0
Using MySQL version: 5.5.39
Using CMake version: 3.10.1
Compiled on: Windows
Automatic database updates are disabled for all databases!
Worldserver listening connections on port 8085
Realmlist (Realm Id: 1) configured in port 8085
VMAPs status: Enabled. LineOfSight: 1, getHeight: 1, indoorCheck: 1
MMAPs status: Enabled
maps directory located in ./maps. Total size: 252191207 bytes
vmaps directory located in ./vmaps. Total size: 588247501 bytes
mmaps directory located in ./mmaps. Total size: 2229000480 bytes
Using frFR DBC Locale as default. All available DBC locales: frFR
Using World DB: TDB 335.64

[Rochet2]

@Killyana what version? is it 335? also with clean DB? CacheDataQueries = 0? - I did have this set to 0. seems you have 32bit, not sure how that affects. Mmaps enabled?

Can you try with same as me? mmaps disabled, vmaps enabled, CacheDataQueries = 0, clean DB

[Killyana]

Server info added. And DB and core are clean.
EDIT: I found CacheDataQueries = 1 (by default)
I change it to 0 and now the server use only 1.3 GB
Store cached static data queries, was added here https://github.com/TrinityCore/TrinityCore/pull/18637 but it was supposed to use only 300 MB extra of Ram not 1.4 GB

I delete all pooling Data and spawn_group Data, and it changes nothing.

[Animalthug]

Can confirm that setting CacheDataQueries = 0, reduces the memory a LOT.

https://image.ibb.co/fGzaEH/consumption2.jpg

[jackpoz]

my 4 years old desktop has 16 GB of ram, a server should have no problems using 1.4 GB of ram

[boveloco]

These are my mem consumptions with rev 3cd7851cd95 on 3.3.5. 57% memory. !

[ccrs]

@Animalthug you better choose your words properly

[Aokromes]

2 GB of ram are to low to run TC properly.

[ccrs]

the query pre creation was done quite some time ago, so it should not be the reason of this "sudden" RAM use increase, maybe some other commit has caused it.

Again, need more research, similar to https://github.com/TrinityCore/TrinityCore/issues/21876#issuecomment-383921320
Thanks @Rochet2 for bringing true data to the discussion.

The rest is and will be considered unrelated.

[jackpoz]

it could be useful to have a wiki page (or maybe there is already) stating the best configuration for low memory environments and high memory ones to get the best out of TC in both cases.
for this particular use case set CacheDataQueries to 0

[ccrs]

@jackpoz the only fear I have about this is a memleak, do you have any test environment?

[Killyana]

Tbh 1.4GB looks too high, the values announced by @xinef1 was 300MB only, so something turned wrong some time after it was implemented.

[ccrs]

ariel kinda expanded xinef's original idea, I dont think that value can be taken as reference

[Killyana]

Yes I found the PR: https://github.com/TrinityCore/TrinityCore/pull/18999
So this is probably intended.

[Animalthug]

@ccrs I always choose my words wisely, thanks.

Closed? What was the answer/solution, or is 2.5GB-4GB to be expected with default config?

[illfated]

To quote jackpoz:

for this particular use case set CacheDataQueries to 0

[jackpoz]

do you have any test environment?

just run TC with Address Sanitizer or Valgrind if you want a report about memory leaks

[ghost]

Confirm

version: https://github.com/TrinityCore/TrinityCore/commit/d920e5b0e4567913b81d5b86400ada8c9107a55d

use memory: 3G 374MB

[jackpoz]

actually this issue was tackled with https://github.com/TrinityCore/TrinityCore/commit/deabd4ab453ee8cf1f515bdb32bf9cf9de7f0591 , reducing a lot the memory usage of CacheDataQueries

[ghost]

Yes, I saw .

why a lot of RAM is being used؟!
use memory: 3G 374MB
uptime: 2day

[Aokromes]

unusued ram, wasted ram.

[jackpoz]

what are you GridUnload, BaseMapLoadAllGrids, InstanceMapLoadAllGrids and GridCleanUpDelay settings ? Do you use HotSwap and/or reload tables at runtime ? do you have custom item entries with very high IDs (is this still even an issue ?) ?
You could create a user dump and inspect with a debugger where the memory is used.

[ghost]

There are options to reduce the use of RAM on worldserver.cfg?

[Aokromes]

https://github.com/TrinityCore/TrinityCore/issues/21876#issuecomment-384080193

[illfated]

@Hir0shi : to be specific, look up this section in worldserver.conf:

#
#   CacheDataQueries
#        Description: Server caches data queries at startup. 
#                     Can be disabled if not enough memory is available.
#        Default:     1 - (Enabled)
#                     0 - (Disabled)
#
#

CacheDataQueries = 1