Ghidra: Task :SourceCodeLookup:buildHelp failed

[xiaoyinl]

Describe the bug
Building the current master (bda4e1dc19bec94efa11e016894a0e20436983a7) fails. The failed task is Task :SourceCodeLookup:buildHelp.

To Reproduce
Build the master branch of this repo.

Environment (please complete the following information):

• OS: Windows server 2019 (GitHub Actions)
• Java Version: 11.0
• Ghidra Version: 9.2

Additional context

Task :SourceCodeLookup:buildHelp
INFO  Using log config file: jar:file:/D:/a/buildGhidra/buildGhidra/ghidra/Ghidra/Framework/Generic/build/libs/Generic.jar!/generic.log4j.xml (LoggingInitialization)  
INFO  Using log file: C:\Users\runneradmin\.ghidra\.ghidra_9.2_DEV\application.log (LoggingInitialization)  

[JavaHelpFilesBuilder] Generating Help Files for: [file:///D:/a/buildGhidra/buildGhidra/ghidra/Ghidra/Features/SourceCodeLookup/src/main/help/help/]
[JavaHelpFilesBuilder] Generating map file: file:///D:/a/buildGhidra/buildGhidra/ghidra/Ghidra/Features/SourceCodeLookup/build/help/main/help/SourceCodeLookup_map.xml...
[JavaHelpFilesBuilder]  finished generating map file
[JavaHelpFilesBuilder] Generating TOC file: SourceCodeLookup_TOC.xml...

Unexpected error building help module files:

java.lang.RuntimeException: Unresolved definitions in tree!
    at help.OverlayHelpTree.initializeTree(OverlayHelpTree.java:175)
    at help.OverlayHelpTree.printTreeForID(OverlayHelpTree.java:135)
    at help.OverlayHelpTree.printTreeForID(OverlayHelpTree.java:127)
    at help.validator.LinkDatabase.generateTOCOutputFile(LinkDatabase.java:200)
    at help.JavaHelpFilesBuilder.generateTOCFile(JavaHelpFilesBuilder.java:188)
    at help.JavaHelpFilesBuilder.generateHelpFiles(JavaHelpFilesBuilder.java:88)
    at help.GHelpBuilder.buildJavaHelpFiles(GHelpBuilder.java:160)
    at help.GHelpBuilder.build(GHelpBuilder.java:96)
    at help.GHelpBuilder.main(GHelpBuilder.java:73)

[astrelsky]

I wasn't able to reproduce this. Maybe try cleaning, re-fetching the dependencies and building again.

[xiaoyinl]

Thank you, @astrelsky! I tried that, but the building still failed. Here is the GitHub Actions workflow file I used to build ghidra: https://github.com/xiaoyinl/ghidraCI/actions/runs/350800632/workflow

This is the full log: logs_2.zip

I can build it successfully on my local Linux system, but not on GitHub Actions with Windows OS.

[Tim---]

Hi,

I have the same problem here. Apparently this was introduced by this recent commit which tried to fix a dirty hack.

The original hack was introduced because the TOC "dependency" helper uses a Hashmap for each tocitem, using the text attribute as the key.
However, if two tocitem share the same text attribute, they will collide, like these two:

https://github.com/NationalSecurityAgency/ghidra/blob/bda4e1dc19bec94efa11e016894a0e20436983a7/Ghidra/Features/Base/src/main/help/help/TOC_Source.xml#L259

https://github.com/NationalSecurityAgency/ghidra/blob/bda4e1dc19bec94efa11e016894a0e20436983a7/Ghidra/Features/Decompiler/src/main/help/help/TOC_Source.xml#L71

So the last one was slightly modified in the hacky version:

https://github.com/NationalSecurityAgency/ghidra/blob/798c3abf4280c74d350c22307886b6b806efb578/Ghidra/Features/Decompiler/src/main/help/help/TOC_Source.xml#L71-L72

Now, the recent commit remove the hack by properly implementing the tree by using the id instead of the text attribute.
However, OverlayHelpTree still uses getTOCItemExternalsByDisplayMapping(), which is indexed by text (and for which we seem to get a collision for the "Comments" key).
I guess this is why it still fails. Reintroducing the hack resolves it.

Well, I didn't look further for now, but I hope it helps narrow down the problem.

[dragonmacher]

I also cannot reproduce the issue.

I will point out that our gradle build system does not correctly handle dependency management for the help artifacts. While working on this change, I had to run a full clean before building help:

gradle clean buildHelp

@Tim--- that was actually a pretty close summary. The real issue, was that we are using a TreeSet for organizing the nodes. The old sort comparator would consider 2 nodes with the same 'text' attribute to compare as 0. This would cause the set to drop one of those nodes. We now include the id in the sorting.

[Tim---]

@dragonmacher I admit that I didn't dig far enough into the code to understand the big picture :). However, I think that this is all trashed by getTOCItemExternalsByDisplayMapping().

Indeed, OverlayHelpTree uses this method to get the TOC items:

https://github.com/NationalSecurityAgency/ghidra/blob/bda4e1dc19bec94efa11e016894a0e20436983a7/Ghidra/Framework/Help/src/main/java/help/OverlayHelpTree.java#L43-L49

If we dig a little deeper, we see that this return a Map, keyed by text attribute:

https://github.com/NationalSecurityAgency/ghidra/blob/49c2010b63b56c8f20845f3970fedd95d003b1e9/Ghidra/Framework/Help/src/main/java/help/validator/location/HelpModuleCollection.java#L291-L296

Both the Comments and AnnotateComments will have the same key. In my setup, it first puts the Comments item, then erases it with the AnnotateComments item.

main[1] print tocItemProvider.getTOCItemExternalsByDisplayMapping().get("Comments")
 tocItemProvider.getTOCItemExternalsByDisplayMapping().get("Comments") = "<tocitem id="AnnoteComments"
        text="Comments"
        target="DecompilePlugin_AnnoteComments"/>
        TOC file="/help/Decompiler_TOC.xml
"

Since the Comments item has children, each child will populate parentToChildrenMap["Comments"]:

https://github.com/NationalSecurityAgency/ghidra/blob/bda4e1dc19bec94efa11e016894a0e20436983a7/Ghidra/Framework/Help/src/main/java/help/OverlayHelpTree.java#L107-L117

However, buildChildren(Comments) will never be called, since the Comments node has been erased by AnnotateComments. So parentToChildrenMap will not be empty at the end, raising the exception:

https://github.com/NationalSecurityAgency/ghidra/blob/bda4e1dc19bec94efa11e016894a0e20436983a7/Ghidra/Framework/Help/src/main/java/help/OverlayHelpTree.java#L171-L176

jdb confirms:

main[1] print parentToChildrenMap
 parentToChildrenMap = "{Comments=[<tocitem id="Displaying Comment History"
        text="Displaying Comment History"
        target="CommentsPlugin_Show_Comment_History"/>
        TOC file="/help/Base_TOC.xml
, <tocitem id="Delete Comments"
        text="Delete Comments"
        target="CommentsPlugin_Delete_Comments"/>
        TOC file="/help/Base_TOC.xml
, <tocitem id="Add or Edit Comments"
        text="Add or Edit Comments"
        target="CommentsPlugin_Edit_Comments"/>
        TOC file="/help/Base_TOC.xml
, <tocitem id="Navigating in Codebrowser through Comments"
        text="Navigating in Codebrowser through Comments"
        target="CommentsPlugin_Navigating_through_Comments"/>
        TOC file="/help/Base_TOC.xml
]}"

I think this will only occur if AnnotateComments overrides Comments in getTOCItemExternalsByDisplayMapping(). If it's the other way around, this should not crash because AnnotateComments does not have child nodes. This would explain why you can't reproduce. Maybe adding a child item to AnnotateComments would trigger the bug consistently ?

In the end, I guess the correct way would be to return a Map keyed by ID (or Text + ID ?) in getTOCItemExternalsByDisplayMapping(), but I'm not familiar enough with the code to know if it will break things down the road.

[dragonmacher]

but I'm not familiar enough with the code to know if it will break things down the road.

That is always the risk :slightly_smiling_face:

I will use your info to dig a bit deeper into the issue. If ordering is the issue, then I should be able to write a test to expose that.

[dragonmacher]

Thanks @Tim--- for the details. This made it much easier to track down and fix. (It's amazing when you see code you wrote years ago and it looks completely foreign.) This will be fixed with the next master branch update.

[Tim---]

Well thank you for looking into this and fixing the bug :).

[maxgerhardt]

This issue is currently preventing building Ghidra with the automated container method (https://github.com/dukebarman/ghidra-builder)

2 languages successfully compiled

> Task :SourceCodeLookup:buildHelp FAILED
INFO  Using log config file: jar:file:/files/ghidra/Ghidra/Framework/Generic/build/libs/Generic.jar!/generic.log4j.xml (LoggingInitialization)  
INFO  Using log file: /home/dockerbot/.ghidra/.ghidra_9.2_DEV/application.log (LoggingInitialization)  
[JavaHelpFilesBuilder] Generating Help Files for: [file:///files/ghidra/Ghidra/Features/SourceCodeLookup/src/main/help/help/]
[JavaHelpFilesBuilder] Generating map file: file:///files/ghidra/Ghidra/Features/SourceCodeLookup/build/help/main/help/SourceCodeLookup_map.xml...
[JavaHelpFilesBuilder]  finished generating map file
[JavaHelpFilesBuilder] Generating TOC file: SourceCodeLookup_TOC.xml...


Unexpected error building help module files:

java.lang.RuntimeException: Unresolved definitions in tree!
        at help.OverlayHelpTree.initializeTree(OverlayHelpTree.java:175)
        at help.OverlayHelpTree.printTreeForID(OverlayHelpTree.java:135)
        at help.OverlayHelpTree.printTreeForID(OverlayHelpTree.java:127)
        at help.validator.LinkDatabase.generateTOCOutputFile(LinkDatabase.java:200)
        at help.JavaHelpFilesBuilder.generateTOCFile(JavaHelpFilesBuilder.java:188)
        at help.JavaHelpFilesBuilder.generateHelpFiles(JavaHelpFilesBuilder.java:88)
        at help.GHelpBuilder.buildJavaHelpFiles(GHelpBuilder.java:160)
        at help.GHelpBuilder.build(GHelpBuilder.java:96)
        at help.GHelpBuilder.main(GHelpBuilder.java:73)

is the fix already pushed for that?

[dragonmacher]

The fix is now in.