Ghidra: DWARF DIE Cache Option

[astrelsky]

Is your feature request related to a problem? Please describe.
Currently the options in the DWARF Importer are to either preload all dies or to limit the max number of dies. This poses a problem for binaries with extensive amounts of DWARF info as pre-loading all the dies causes an OutOfMemoryError yet not pre-loading would take days if not weeks or more to finish.

Describe the solution you'd like
I am assuming from the available options that there is no caching of the dies occurring. If there is I have yet to stumble across it.

Describe alternatives you've considered
Run the DWARF importer for days and hope it eventually finishes before my patients runs out. This is bad for productivity though.

Additional context
@dev747368 libwebkit2gtk-3.0-25-dbg is an excellent stress test for the DWARF importer. As a reference I am launching Ghidra with the -Xmx28672m vmarg and am still running out of memory. (That gets rounded to about ~30Gb)

Also to load that file the .got and .plt.got sections need to be removed due to #2416

[dev747368]

Pre-loading all DIEs isn't a performance option. It exists because there were tool chains that produced DIEs that referenced DIEs in other compile units. The normal mode of operation is to load the DIEs for a compile unit, process them all, discard them, go to the next compile unit, and lather, rinse repeat.

The slowness you are seeing is probably due to data type resolution / conflict handling. :) Can't get away from that it seems.

There have been a few changes that might help, and there is one change hopefully hitting before 9.2 with the naming of anon data types that might reduce the amount of work.

[dev747368]

I will use your example binary as well in the future.

[astrelsky]

I see. It was impossible to tell as I'd run out of memory when pre loading all the dies.

If some dies are being referenced from other compile units wouldn't caching them help with that? This way if they are referenced the cache could be checked and if it isn't there then the compile unit gets reloaded. It would definitely help for when this occurs and you don't have enough ram to pre-load them.

Also changes to anonymous types are good. I'm not a fan of them or the .conflict(inf) yet I understand the necessity.

[dev747368]

Do you remember where in the various steps (per the progress meter) it bogs down / starts taking lots of memory?

[astrelsky]

Do you remember where in the various steps (per the progress meter) it bogs down / starts taking lots of memory?

While checking the pre-conditions I think or just after. I can double check tonight.

[dev747368]

I saw it take a bit of time to do the pre-condition checking, but after those several minutes, I'm into the part where its importing types in each compilation unit. I haven't really gone above 2gb of used memory either.

[astrelsky]

I saw it take a bit of time to do the pre-condition checking, but after those several minutes, I'm into the part where its importing types in each compilation unit. I haven't really gone above 2gb of used memory either.

I'm stuck waiting on "DWARF Move Types" at the moment and will have to let it run overnight. I modified the dwarf conflict handler so I could get at least some of the data I was looking for. I'll rebuild ghidra, re-import and re-run tomorrow to see exactly where in the dwarf analysis the memory issue is occurring. I may modify the task monitor to log the changes in message to the application log so I don't need to monitor it.

[dev747368]

Ah. Yeah, you are right at the end. FYI, closing the data type manager UI may help a little by reducing the number of queries going into the data type db tables.

[astrelsky]

Ah. Yeah, you are right at the end. FYI, closing the data type manager UI may help a little by reducing the number of queries going into the data type db tables.

I closed all the windows and it seemed to help a bit. It finally finished around 9:00 am so I'll rebuild and run again as described above later.

2020-11-06 09:01:43 ERROR ghidra.app.plugin.core.analysis.DwarfLineNumberAnalyzer Unexpected Exception: Index values do not match! java.lang.IllegalStateException: Index values do not match!
    at ghidra.app.util.bin.format.dwarf.line.StatementProgramInstructions.executeExtended(StatementProgramInstructions.java:114)
    at ghidra.app.util.bin.format.dwarf.line.StatementProgramInstructions.execute(StatementProgramInstructions.java:71)
    at ghidra.app.plugin.core.analysis.DwarfLineNumberAnalyzer.added(DwarfLineNumberAnalyzer.java:75)
    at ghidra.app.plugin.core.analysis.AnalysisScheduler.runAnalyzer(AnalysisScheduler.java:186)
    at ghidra.app.plugin.core.analysis.AnalysisTask.applyTo(AnalysisTask.java:39)
    at ghidra.app.plugin.core.analysis.AutoAnalysisManager$AnalysisTaskWrapper.run(AutoAnalysisManager.java:688)
    at ghidra.app.plugin.core.analysis.AutoAnalysisManager.startAnalysis(AutoAnalysisManager.java:788)
    at ghidra.app.plugin.core.analysis.AutoAnalysisManager.startAnalysis(AutoAnalysisManager.java:667)
    at ghidra.app.plugin.core.analysis.AutoAnalysisManager.startAnalysis(AutoAnalysisManager.java:632)
    at ghidra.app.plugin.core.analysis.AnalysisBackgroundCommand.applyTo(AnalysisBackgroundCommand.java:58)
    at ghidra.framework.plugintool.mgr.BackgroundCommandTask.run(BackgroundCommandTask.java:102)
    at ghidra.framework.plugintool.mgr.ToolTaskManager.run(ToolTaskManager.java:315)
    at java.base/java.lang.Thread.run(Thread.java:832)

2020-11-06 09:01:43 INFO  ghidra.app.plugin.core.analysis.AutoAnalysisManager -----------------------------------------------------
    DWARF                                  76952.021 secs
    DWARF Line Number                          0.009 secs
-----------------------------------------------------
     Total Time   76952 secs
-----------------------------------------------------

DWARF Import: 1, Nevada: 0

[astrelsky]

@dev747368 The memory usage does explode during "DWARF Checking Preconditions". The JVM was taking 17294.6 MB before I killed it since my computer was lagging.
application.log

Also now that I have sat through all of that I now notice that almost every section is uninitialized including .text, .data and .rodata :/

[dev747368]

Yeah. This is a debug image. It looks like they are created with a "strip --only-keep-debug".

The memory usage during precondition is worrying. My JVM only had 4gb to begin with and didn't go above 2 while doing most of its work. (I didn't allow the entire import to finish as I kept interrupting it to debug some data type resolve stuff)

One thing with java memory management, if you give it an excessive amount of memory, it will use it, even if it could have managed with a smaller amount... it just delays the GC until later.

[astrelsky]

Yeah. This is a debug image. It looks like they are created with a "strip --only-keep-debug".

The memory usage during precondition is worrying. My JVM only had 4gb to begin with and didn't go above 2 while doing most of its work. (I didn't go to completion as I kept interrupting it to debug some data type resolve stuff)

One thing with java memory management, if you give it an excessive amount of memory, it will use it, even if it could have managed with a smaller amount... it just delays the GC until later.

So I lowered the memory to 8GB and this does appear to be the case. I do eventually get an OutOfMemoryError though. I'm assuming it occurs in the next stage as it vm is still trying to manage all the dies and eventually cannot handle it any longer.

[dev747368]

FYI, I did track down an issue with comparing structures that contain bitfields. With this fixed, the number of .conflicts will reduce dramatically.

[astrelsky]

@dev747368 this isn't directly related to the topic here but I just had a thought about having the DWARF conflict handler check to see if the two types came from the same die. Since the equality check can be quite expensive for large composite types this could be an effective means of improving performance if it does occur.

[dev747368]

I am working on large DWARF issues and have queued up several improvements. I've got your example down to <6gb ram needed and (excluding the "moving types" step at the end) to 4'ish hours.

RE: DIE to data type comparisions... It does re-use types in some situations (which effectively gets to what you are saying), but looking over the code right now there are a few more places it could do it.

[astrelsky]

I am working on large DWARF issues and have queued up several improvements. I've got your example down to <6gb ram needed and (excluding the "moving types" step at the end) to 4'ish hours.

RE: DIE to data type comparisions... It does re-use types in some situations (which effectively gets to what you are saying), but looking over the code right now there are a few more places it could do it.

Iirc the dwarf importer initially imports everything into /DWARF/UNCATEGORIZED. Since you can only import the dwarf info once and all conflicts would have been resolved in the initial import I think the conflict handler can be set to KEEP_HANDLER during the "move types" step if it isn't already. I'm not currently looking at any code and am going of the top of my head so I could be remembering wrong.