Leakcanary: ArrayIndexOutOfBoundsException in MutableByteSubArray

[nobeh]

Description

Trying to run the example at https://square.github.io/leakcanary/shark/#generating-a-heap-analysis-report on 11G hprof file (not from Android but a Java process):

Caused by: java.lang.ArrayIndexOutOfBoundsException: 250655009
    at shark.internal.UnsortedByteEntries$MutableByteSubArray.writeLong(UnsortedByteEntries.kt:187)
    at shark.internal.UnsortedByteEntries$MutableByteSubArray.writeId(UnsortedByteEntries.kt:135)
    at shark.internal.HprofInMemoryIndex$Builder.onHprofRecord(HprofInMemoryIndex.kt:291)
    at shark.HprofReader.readHprofRecords(HprofReader.kt:444)
    at shark.internal.HprofInMemoryIndex$Companion.createReadingHprof(HprofInMemoryIndex.kt:391)
    at shark.HprofHeapGraph$Companion.indexHprof(HprofHeapGraph.kt:245)
    at shark.HprofHeapGraph$Companion.indexHprof$default(HprofHeapGraph.kt:217)

Unfortunately, due to privacy, I am not able to share the heap dump here.

Version Information

• LeakCanary version: 2.0
• Android OS version: not applicable
• Gradle version: not applicable
• Kotlin: 1.3.72
• JVM: 1.8.0-252-amzn

[pyricau]

Thanks for the report! I've worked on making LeakCanary JVM compatible, but it definitely lacks enough real usage to find and fix edge cases.

If you can't share the hprof, any chance you could investigate what's going on and offer a fix, or reproduce with a sample program that creates a huge hprof?

[hadilq]

I'm interested in this issue, but I'm not sure if I understand it correctly! Should we implement HprofInDiskIndex to use instead of HprofInMemoryIndex for this case? Then maybe some configs to switch to this option may solve this problem, yes? To test it we just need to check that this new implementation would work as HprofInMemoryIndex works but of course slower! In the end, having a HprofInDiskIndex makes sense as here we're talking about memory leaks and in such situation there's no memory to save the indices.

[pyricau]

Great question! In theory LeakCanary can handle large heap dumps anyway because it never loads the entire heap dump in memory, the whole point is to keep low memory usage. Though there can definitely be edge cases we haven't found yet.

That being said, here, we're not running out of memory, we're trying to write past the UnsortedByteEntries.entries array size. That's not supposed to happen, as that array is supposed to grow as needed. append() is called first and performs a capacity check and otherwise expands the array and then the rest of the data is written for that entry.

I wonder if we've overflow an int somewhere. Can find out more without debugging though.

btw you should be able to open the LeakCanary project, replace some of the tests hprof files with your own and debug it on your own :)

[pyricau]

@nobeh have you found time to investigate this some more, or should I close?

[pyricau]

I tried running LeakCanary on a 2Gb hprof and didn't run into this. Not sure what it is about. I'll close until someone can repro and we can fix.