Ts3audiobot: Unable to run alongside Java VM

[Thomseeen]

Describe the bug
I am unable to start the Java VM alongside the Audio Bot.
This is true for

• starting any Java program before the bot -> Bot won't start, see logs
• starting any Java program when the bot is running -> Java won't start, see logs

I'm unsure whether this is a bug/issue with your bot or some setup thing as dotnet and java don't like each other. Not being the most advanced Linux user, I tried everything I could find about checking thread- and memory-limits, but without finding anything that's wrong. Maybe someone on here can reproduce my problem or knows, what exactly could cause this.

To Reproduce

1. Start the bot
2. Start Java VM, i.e. with java -version -> crash

OR

1. Start any Java program (I originally tested with a "bigger" program, but the issue is also present with a program that just prints something in an endless loop and with limited memory for the VM and single threadstacks).
2. Start the bot -> crash

Version
Version: 0.10.1+7
Branch: master
CommitHash: 65aa9a18ca5f72f0be9adc6aa93571b9291a9398

Platform

• Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0 x86_64)
• VServer with 8GB RAM and 4 VCores
• dotnet --info:
  
  
  
  • Host (useful for support):
    
    Version: 3.1.1
    
    Commit: a1388f194c
  
  
  • .NET Core runtimes installed:
    
    Microsoft.NETCore.App 2.2.8 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
    
    Microsoft.NETCore.App 3.1.1 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  
  

Log
Starting TS3AudioBot after any Java program

2020-01-24 19:46:08.5672| INFO||Setup.LogHeader [============ TS3AudioBot started =============]
2020-01-24 19:46:08.6953| INFO||Setup.LogHeader [ Date/Time: Freitag, 24. Januar 2020 19:46:08
2020-01-24 19:46:08.7085| INFO||Setup.LogHeader [ Version: 0.10.1+7/master/65aa9a18
2020-01-24 19:46:08.7103| INFO||Setup.LogHeader [ Platform:   (64bit)
2020-01-24 19:46:08.7115| INFO||Setup.LogHeader [ Runtime: .NET Core (2.2.8) ServerGC:True GC:SustainedLowLatency
2020-01-24 19:46:08.7115| INFO||Setup.LogHeader [ Opus: libopus 1.1.2 (x64)
2020-01-24 19:46:08.7124| INFO||Setup.LogHeader [==============================================]
2020-01-24 19:46:09.6812|FATAL||Core.ExceptionHandler Critical program failure!
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Threading.Thread.StartInternal()
at TS3AudioBot.Helper.TickPool.AddWorker(TickWorker worker) in C:\projects\ts3audiobot\TS3AudioBot\Helper\TickPool.cs:line 53
at TS3AudioBot.Helper.TickPool.RegisterTick(Action method, TimeSpan interval, Boolean active) in C:\projects\ts3audiobot\TS3AudioBot\Helper\TickPool.cs:line 41
at TS3AudioBot.Helper.Environment.SystemMonitor.StartTimedSnapshots() in C:\projects\ts3audiobot\TS3AudioBot\Helper\Environment\SystemMonitor.cs:line 27
at TS3AudioBot.Core.Run(Boolean interactive) in C:\projects\ts3audiobot\TS3AudioBot\Core.cs:line 121
at TS3AudioBot.Core.Main(String[] args) in C:\projects\ts3audiobot\TS3AudioBot\Core.cs:line 55
2020-01-24 19:46:09.7232| INFO||Core.Dispose TS3AudioBot shutting down.
2020-01-24 19:46:09.7345| INFO||WebServer.Dispose WebServer is closing

Starting any Java program after TS3AudioBot

[0.084s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
at java.lang.Thread.start0(java.base/Native Method)
at java.lang.Thread.start(java.base/Thread.java:803)
at java.lang.ref.Reference.<clinit>(java.base/Reference.java:303)

[Flakebi]

It doesn’t seem to be a general incompatibility, I can successfully run java -version while a bot is running.
What does your memory usage look like? (e.g. with free -h)

[Thomseeen]

Memory seems fine:

free -h
              Gesamt   belegt    frei        gemns. Puffer/Cache verfügbar
Speicher:        8,0G         77M        7,8G        1,1M        100M        7,9G
Auslagerungsspeicher:          0B          0B          0B

Thread limit seems fine too:

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1030709
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 62987
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

ps -eo nlwp | tail -n +2 | awk '{ num_threads += $1 } END { print num_threads }'
73



md5-e7d616274bb4a864e9e0c2c7463884b5



lsof -u teamspeak3 | wc -l
346

The only thing I can see and slightly wonder about, is the 20.2GB use of virtual memory the bot is using according to htop.

[Splamy]

This is indeed a weird case. The only related thing I could find is this: https://stackoverflow.com/questions/16789288/java-lang-outofmemoryerror-unable-to-create-new-native-thread
Since both runtimes threw while trying to create a new thread:

java.lang.OutOfMemoryError: unable to create native thread

at System.Threading.Thread.StartInternal()

but 73 really isnt a lot of threads, so it would be weird if it would hit the thread limit here.

From what I see in the stacktrace of the bot, this should be the first thread which the bot manually starts. However the dotnet runtime probably will start a few threads too before that.

So far my observations.
Stuff you can try:

• Try running the bot with --no-llgc, this will use the default garbage collector which (afaik) allocates less threads and cleans up more regularly
• In the TS3AudioBot.runtimeconfig.json file set the runtimeOptions.configProperties."System.GC.Server" to false this should allocate less threads and reserve less memory per thread (see msdn)
• Try starting a lot of processes and see if it has the same effect of the jvm/dnc runtime (The bot with 1 running client on my server uses 19 threads)

The only thing I can see and slightly wonder about, is the 20.2GB use of virtual memory

That's normal for dotnet core, since it's only virtual memory it's not really allocated. The real used physical memory will be adjusted and can also shrink again when freed,

[Thomseeen]

So I tried the --no-llgc option and that runtime option without any positive results.

To check for the thread/task limitation is just opened a bunch of ssh-connections, after about 11 I wasn't able to spawn any more and was unable to start the JVM with the same error as listed above... I had to close half of the ssh-connections to be able to start java -version... so I guess there is some weird task or thread limitation in place, which I can't find. Guess I will try to contact my hoster's support.

[Thomseeen]

Solved; it was a problem with the threads being limited by Ubuntu.

Problem:

My hoster uses Virtuozzo which limits the maximum processes per VM-instance in /proc/user_beancounters - numproc. This value is 400 for my type of server, which doesn't explain the problems I get at around 60 tasks. It appears Ubuntu takes this value and sets the maximum allowed tasks, according to systemctl show --property=DefaultTasksMax, to 15% of that by default. 400*0.15=60 so that's where the 60 tasks-limit comes from.

Solution:

DefaultTasksMax can be changed in /etc/systemd/system.conf (120 for me now). Everything works as expected now.