Detekt: OOM on multiple invocations

[arturdryomov]

Hello there.

Me and @artem-zinnatullin are working on the Bazel Detekt rule. The rule can work as a persistent worker. This means that we run Detekt multiple times in a single process. We did an internal testing at scale (hundreds of Bazel packages) and found that the rule gets stuck at some point. I’ve analyzed the memory usage and seems like we are getting near OOM and a lot of time is spend by GC trying to free a bit of memory. Fortunately enough I was able to reproduce it in an isolated environment (see steps below).

I think it might be related to #1873. Seems like KotlinCoreEnvironment objects are not disposed.

Steps to reproduce

1. Create a directory named detekt-oom-sandbox.

2. Create a file detekt-oom-sandbox/Detekt.java.

   import io.gitlab.arturbosch.detekt.cli.Main;
   
   final class Detekt {
   
       public static void main(String args[]) {
           for (int iteration = 0; iteration < 1_000; iteration++) {
               System.out.println(iteration);
   
               Main.buildRunner(args, System.out, System.err).execute();
           }
       }
   }
   

3. Create a file detekt-oom-sandbox/kotlin/main.kt.
   kotlin fun main() { println("Hi") }

4. Create a file detekt-oom-sandbox/run.sh.

   #!/bin/bash
   set -eou pipefail
   
   DETEKT_VERSION="1.3.1"
   
   curl --location "https://repo.maven.apache.org/maven2/io/gitlab/arturbosch/detekt/detekt-cli/${DETEKT_VERSION}/detekt-cli-${DETEKT_VERSION}-all.jar" --output "detekt.jar" --time-cond "detekt.jar"
   
   javac Detekt.java -classpath detekt.jar
   java -Xmx16m -XX:+HeapDumpOnOutOfMemoryError -classpath detekt.jar:. Detekt --input kotlin/
   

5. cd detekt-oom-sandbox && bash run.sh

6. Observe (around 280 iteration)!

   java.lang.OutOfMemoryError: GC overhead limit exceeded
   Dumping heap to java_pid6461.hprof ...
   Heap dump file created [27347847 bytes in 0.195 secs]
   Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
   

Opening the produced heap dump shows something like this in MAT.

[schalkms]

Thanks for testing detekt at a large scale and for describing the observed problems.
The changed function in the referenced PR #1873 is only called in test sources. Hence, it's not causing the problem.
I guess the used KotlinCoreEnvironment should be handled like in #1873 or do you have a better idea @arturbosch? The approach taken in #1873 seems to be impossible for this problem at the first glance.

[arturdryomov]

Just tested a change which seems to resolve this. Gonna open a PR ASAP.

[schalkms]

Just tested a change which seems to resolve this. Gonna open a PR ASAP.

Awesome 👍! The gold medal🥇for the best PR today is yours.

[arturdryomov]

Plot twist: there were no PRs today...