Coursier: Possible memory leak in Tasks.scala

[vil1]

I have a relatively large multimodule project (30+modules) which, thanks to hadoop, transitively depends on about half of maven-central.

When I perform project-wise operations like sbt update (or hitting Refresh Project in intelliJ), the JVM ends up in OutOfMemoryError (GC overhead limit exceeded). The same operation with coursier plugin disabled works fine (as long as wasting 10+ minutes can be considered as fine :) ).

Upon heap dump investigation (which is definitely not one of my strongest skills), it appears that the heap contains millions (40+) of HashMap entries, many of whose nearest GC-root seems to be the reportsCache created here.

The version used is 1.0.0-M14-9

Illustration : http://imgur.com/a/PS37A

[alexarchambault]

@vil1 If you feel like giving a try to removing reportsCache and see if it works for you, go for it, I think it's redundant. I can't have a look at this right now.

[vil1]

I'm in the process of doing it, sort of. As I don't know the implications enough, I've just changed reportsCache to a WeakHashMap which should fix the problem.
I've a bit of struggle using my locally published 1.0.0-SNAPSHOT, but that shouldn't last too long.

[vil1]

Bad news : the WeakHashMap change described above, and even a complete removal of reportsCache do not fix the problem (confirming that my heap dump analysis skills are pretty weak).

I'll keep investigating

[vil1]

I finally managed to work around the problem using the following JVM options for SBT :

-XX:+UseG1GC
-XX:MaxGCPauseMillis=200
-XX:ParallelGCThreads=4
-XX:ConcGCThreads=4
-XX:InitiatingHeapOccupancyPercent=50

(Thanks to @samklr's advice)

[alexarchambault]

Thanks for having investigated! A bit worrying that coursier puts such a load on the GC though, but that can be addressed in a second time.

[vil1]

This may also be very specific to the combination of coursier internals and the structure of my project (many submodules, with internal dependencies between them, and transitive dependencies to a lot of libraries through hadoop, spark and pals)

[alexarchambault]

https://github.com/coursier/coursier/pull/462 is merged (and seemed to address this for you... are 1.0.0-RC2 or 1.0.0-RC3 fine?). Feel free to re-open if you still face memory issues.

[vil1]

I confirm, the memory problems seem to be fixed by #462
Thx a lot