Pocketmine-mp: Region-based world formats leak file resources

Created on 9 Jul 2017 · 20Comments · Source: pmmp/PocketMine-MP

Issue description

The Server just froze with the error message attached below. When I'm trying to connect, it keeps saying "synchronizing packets". The server also can't be stopped with CTRL + C. The world has been recently converted from MCRegion to PMAnvil, the error didn't happened with MCRegion.

Steps to reproduce the issue

Using a huge creative world and about 40 players
The error occurs very seldom

OS and versions

PocketMine-MP: 2726f2a0119a09df1433ac63dc3b5f4d584ceada
PHP: 7.0.21
Server OS: Ubuntu 16.04 LTS
Game version: PE/Win10

Plugins

Test on a clean server without plugins: is the issue reproducible without any plugins loaded?

Didn't found it out yet because this issue happens very seldom and no player would play on a plot server without plugins.

Crashdump, backtrace or other files

[20:53:21] [Server thread/CRITICAL]: ErrorException: "file(/proc/self/maps): failed to open stream: Too many open files" (EXCEPTION) in "/src/pocketmine/utils/Utils" at line 223
[20:53:21] [Server thread/EMERGENCY]: An unrecoverable error has occurred and the server has crashed. Creating a crash dump
[20:53:21] [Server thread/CRITICAL]: Error: "Class 'ClassNotFoundException' not found" (EXCEPTION) in "/src/spl/BaseClassLoader" at line 131
[20:53:21] [Server thread/CRITICAL]: Could not create crash dump: Class 'ClassNotFoundException' not found
[20:53:21] [Server thread/INFO]: Stopping other threads
[20:53:21] [Server thread/EMERGENCY]: An unrecoverable error has occurred and the server has crashed. Creating a crash dump
[20:53:21] [Server thread/CRITICAL]: Error: "Class 'ClassNotFoundException' not found" (EXCEPTION) in "/src/spl/BaseClassLoader" at line 131
[20:53:21] [Server thread/CRITICAL]: Could not create crash dump: Class 'ClassNotFoundException' not found

Core Fixed

Source

jarne

Most helpful comment

@dktapps Here's the output now:

OFLD.trace.txt

Hope it helps.

feikede on 30 Dec 2017

🎉2 👍2

All 20 comments

Are these 40 players scattered over a very large area?

Also can you execute ulimit -n and report the output. Thanks.

dktapps on 9 Jul 2017

Yes, nearly every of these 40 players is on his own plot and they are at different positions of the world.

ulimit -n returns 1024.

jarne on 9 Jul 2017

Please try loading this plugin onto your server: https://github.com/dktapps/OpenFilesLeakDebugger

If the plugin succeeds in catching the error, it will var_dump a nice big list of open files into the console. Please paste that list in a gist or something so I can take a look.

dktapps on 13 Jul 2017

Alright, it's running on the server where the bug occurred now. It could just take some time because this bug is really seldom.

jarne on 13 Jul 2017

my server crash with same error
1 player were online
1 error since 48 hours server online
no crashdump created

server log

PocketMine-MP 1.6.2dev-318 | Online 1/50 | Memory 215.03/3.01/413.18/1202.21 MB @ 10 threads | U 2.04 D 1.23 kB/s | TPS 20 | Load 6.88% [36m[04:54:27] [m [31m[Server thread/CRITICAL]: ErrorException: "file(/proc/self/maps): failed to open stream: Too many open files" (EXCEPTION) in "src/pocketmine/utils/Utils" at line 226 [m

screen log

017-07-30 [04:54:04] [Server thread/INFO]: Steve[/XX.XXX.212.249:54176] logged out due to Kicked by admin. Reason: Login timer expired!
2017-07-30 [04:54:27] [Server thread/CRITICAL]: ErrorException: "file(/proc/self/maps): failed to open stream: Too many open files" (EXCEPTION) in "src/pocketmine/utils/Utils" at line 226
2017-07-30 [04:54:27] [Server thread/EMERGENCY]: An unrecoverable error has occurred and the server has crashed. Creating a crash dump
2017-07-30 [04:54:27] [Server thread/CRITICAL]: Error: "Class 'pocketmine\CrashDump' not found" (EXCEPTION) in "src/pocketmine/Server" at line 2161
2017-07-30 [04:54:27] [Server thread/CRITICAL]: Could not create crash dump: Class 'pocketmine\CrashDump' not found
2017-07-30 [04:54:27] [Server thread/CRITICAL]: Error: "Class 'pocketmine\event\plugin\PluginDisableEvent' not found" (EXCEPTION) in "src/pocketmine/plugin/PharPluginLoader" at line 135
2017-07-30 [04:54:27] [Server thread/INFO]: Disabling SimpleAuth v1.8.1-dev1
2017-07-30 [04:54:27] [Server thread/CRITICAL]: Error: "Class 'pocketmine\event\plugin\PluginDisableEvent' not found" (EXCEPTION) in "src/pocketmine/plugin/PharPluginLoader" at line 135
2017-07-30 [04:54:27] [Server thread/INFO]: Disabling SimpleAuthHelper v2.0.5-dev1
2017-07-30 [04:54:27] [Server thread/CRITICAL]: Error: "Class 'pocketmine\event\plugin\PluginDisableEvent' not found" (EXCEPTION) in "src/pocketmine/plugin/PharPluginLoader" at line 135
2017-07-30 [04:54:27] [Server thread/CRITICAL]: Error: "Class 'pocketmine\event\level\LevelUnloadEvent' not found" (EXCEPTION) in "src/pocketmine/level/Level" at line 539
2017-07-30 [04:54:27] [Server thread/EMERGENCY]: Crashed while crashing, killing process

totor29 on 30 Jul 2017

my server crashed today with same error
2 player were online
1 error since 24 hours server online
no crashdump created

log
017-08-03 [01:40:39] [Server thread/CRITICAL]: ErrorException: "file(/proc/self/maps): failed to open stream: Too many open files" (EXCEPTION) in "src/pocketmine/utils/Utils" at line 226
2017-08-03 [01:40:39] [Server thread/EMERGENCY]: An unrecoverable error has occurred and the server has crashed. Creating a crash dump
2017-08-03 [01:40:39] [Server thread/CRITICAL]: Error: "Class 'pocketmine\CrashDump' not found" (EXCEPTION) in "src/pocketmine/Server" at line 2161
2017-08-03 [01:40:39] [Server thread/CRITICAL]: Could not create crash dump: Class 'pocketmine\CrashDump' not found

screen log
]0;PocketMine-MP 1.6.2dev-339 | Online 2/50 | Memory 224.39/3.23/490.43/1414.68 MB @ 11 threads | U 0.26 D 1.13 kB/s | TPS 20 | Load 5.86%]0;PocketMine-MP 1.6.2dev-339 | Online 2/50 | Memory 224.39/3.23/490.43/1414.68 MB @ 11 threads | U 0.65 D 1.27 kB/s | TPS 20 | Load 5.11%]0;PocketMine-MP 1.6.2dev-339 | Online 2/50 | Memory 224.39/3.23/490.43/1414.68 MB @ 11 threads | U 0.73 D 1.32 kB/s | TPS 20 | Load 5.1%]0;PocketMine-MP 1.6.2dev-339 | Online 2/50 | Memory 224.4/3.23/490.43/1414.68 MB @ 11 threads | U 3.54 D 2.01 kB/s | TPS 20 | Load 5.92%[36m[01:40:39] [m[31m[Server thread/CRITICAL]: ErrorException: "file(/proc/self/maps): failed to open stream: Too many open files" (EXCEPTION) in "src/pocketmine/utils/Utils" at line 226[m
[36m[01:40:39] [m[31m[Server thread/EMERGENCY]: An unrecoverable error has occurred and the server has crashed. Creating a crash dump[m
[36m[01:40:39] [m[31m[Server thread/CRITICAL]: Error: "Class 'pocketmine\CrashDump' not found" (EXCEPTION) in "src/pocketmine/Server" at line 2161[m
[36m[01:40:39] [m[31m[Server thread/CRITICAL]: Could not create crash dump: Class 'pocketmine\CrashDump' not found[m
libgcc_s.so.1 must be installed for pthread_cancel to work
bash : ligne 1 : 19220 Abandon ./start.sh

totor29 on 3 Aug 2017

Hello,

I started to get this same "libgcc_s.so.1 must be installed for pthread_cancel to work" error after I started using latest Ubuntu Server and running the latest PMMP .phar. I been searching for days, installing different things to see what works but ultimately eventually get the same errors as everyone above.

luitron on 23 Nov 2017

I get this error daily. On a Mac OS X Sierra with ulimit -n of 256 and also on a raspberry 3 with SuSE Linux aarch64 with ulimit -n of 1024. Changing the ulimit does not really help.
But it is definitely related to the size of the world we're playing in. A workaround is to run ./start.sh in a bash loop.

feikede on 29 Dec 2017

@feikede how thoroughly does this world get explored? what kind of distances are travelled and what timeframe in?

If you can reproduce this regularly, it would be nice if you could load this plugin onto your server and feedback the results when the server crashes. You can get a prebuilt artifact from Poggit (login with github to view).

It'll throw a lot of red text into the console with filenames when the error occurs, and then stop the server. It's the red text I'm interested in.

dktapps on 29 Dec 2017

@dktapps I have installed the OpenFilesLeakDebugger from your github repo on my Mac OS X Sierra server. As soon as it happens I'll paste the red text here. Your other questions are really hard to answer. I don't know how big the world is, it's from minecraftworldmap.com but I can't remember which one.
But we are 4 persons, playing about 1 hour a day each.

feikede on 29 Dec 2017

@feikede that's fine, the plugin will spit out any relevant information when the bug next occurs. Thanks.

dktapps on 29 Dec 2017

@dktapps Here's the output now:

OFLD.trace.txt

Hope it helps.

feikede on 30 Dec 2017

🎉2 👍2

That's fantastic, thanks very much @feikede !

dktapps on 30 Dec 2017

This confirms what I originally thought - there is a file descriptor leak with Region-based worlds, which I initially thought was harmless (because they are supposed to get collected by GC sweep every half-hour or so). Apparently it is _not_ harmless.

This will require a new Region management system to ensure that the regions are cleaned up properly when no longer in use.

dktapps on 30 Dec 2017

🎉2

@dktapps Fine! Let me know if I can be of any further help.

feikede on 30 Dec 2017

Postmortem on this: I thought that level GC was triggered every half-hour by the GC sweep. It is not.

The only time regions get cleaned up is when the GC is triggered under low memory conditions. This means that if your server never reaches a low memory condition, the regions will simply keep accumulating as you explore, until the server crashes.

dktapps on 2 Jan 2018

The referenced commit should fix the bug, please test and report back.

dktapps on 2 Jan 2018

I have a similar error when playing a lot of players and some randomly triggered this error https://github.com/pmmp/PocketMine-MP/issues/1183

Perhaps it is not related, but this error is killing me, it appears so often that I'm sick of this