Ghidra: NoClassDefFoundError when compiling GhidraScript

[astrelsky]

Describe the bug
NoClassDefFoundError occurs when attempting to run a GhidraScript

To Reproduce
Steps to reproduce the behavior:

1. Attempt to run any GhidraScript

Attachments

stack trace

ERROR Task Error: compiling script directory - Uncaught Exception: java.lang.NoClassDefFoundError: Could not initialize class aQute.bnd.osgi.Analyzer  (Task.java:133)
java.lang.NoClassDefFoundError: Could not initialize class aQute.bnd.osgi.Analyzer
    at ghidra.app.plugin.core.osgi.GhidraSourceBundle.generateManifest(GhidraSourceBundle.java:824) ~[Base.jar:?]
    at ghidra.app.plugin.core.osgi.GhidraSourceBundle.compileToExplodedBundle(GhidraSourceBundle.java:1024) ~[Base.jar:?]
    at ghidra.app.plugin.core.osgi.GhidraSourceBundle.build(GhidraSourceBundle.java:605) ~[Base.jar:?]
    at ghidra.app.plugin.core.osgi.BundleHost.activateAll(BundleHost.java:708) ~[Base.jar:?]
    at ghidra.app.script.JavaScriptProvider.loadClass(JavaScriptProvider.java:116) ~[Base.jar:?]
    at ghidra.app.script.JavaScriptProvider.getScriptInstance(JavaScriptProvider.java:81) ~[Base.jar:?]
    at ghidra.app.plugin.core.script.GhidraScriptComponentProvider.getScriptInstance(GhidraScriptComponentProvider.java:642) ~[Base.jar:?]
    at ghidra.app.plugin.core.script.GhidraScriptComponentProvider.doRunScript(GhidraScriptComponentProvider.java:624) ~[Base.jar:?]
    at ghidra.app.plugin.core.script.GhidraScriptComponentProvider$5.run(GhidraScriptComponentProvider.java:611) ~[Base.jar:?]
    at ghidra.util.task.Task.monitoredRun(Task.java:126) ~[Generic.jar:?]
    at ghidra.util.task.TaskRunner.lambda$startTaskThread$1(TaskRunner.java:94) ~[Docking.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
    at java.lang.Thread.run(Thread.java:832) [?:?]

Environment (please complete the following information):

• OS: Microsoft Windows [Version 10.0.19041.388]
• Java Version: 14.0.2
• Ghidra Version: 9.2 aac0c2229f29395e58ea2d77d0a63360548c1a47

Additional context
I had just run gradle clean, deleted the flatRepo, ran gradle --init-script gradle/support/fetchDependencies.gradle init, rebuilt ghidra and still encounter this.

Also other GhidraScript classes in the same directory as the GhidraScript being run that are not imported should not be recompiled. I am encountering compile errors do to outdated scripts in the same directory.

@ryanmkurtz

[ryanmkurtz]

The scripting engine was just rewritten by not me. Unfortunately I am not familiar with this code anymore to know what's going on or what the intent of certain behaviors are (like why all scripts get recompiled).

[astrelsky]

The scripting engine was just rewritten by not me. Unfortunately I am not familiar with this code anymore to know what's going on or what the intent of certain behaviors are (like why all scripts get recompiled).

Oh my mistake. I thought the changes were from #1960 but it doesn't appear to by the case.

[astrelsky]

@jpleasu this appears to be due to a change between 34a030173cb819d659b3521da6e62526f0b76f18 and 1a598f26b2c31c6a69eb07cacbcc112d1b8da257

[jpleasu]

I'm guilty -- trying to recreate the issue now.

It seems you're not getting the new dependency: https://github.com/NationalSecurityAgency/ghidra/blob/master/Ghidra/Features/Base/build.gradle#L31

Compiling everything in the directory is intended -- the new scripting model treats scripting directories as modules, everything in a directory will be built once and cached. In the new model, even sub-packages will be recompiled as necessary.

A lot of failing scripts in a directory shouldn't break anything, it'll just make noise (the compilation errors).

[astrelsky]

I'm guilty -- trying to recreate the issue now.

It seems you're not getting the new dependency: https://github.com/NationalSecurityAgency/ghidra/blob/master/Ghidra/Features/Base/build.gradle#L31

Compiling everything in the directory is intended -- the new scripting model treats scripting directories as modules, everything in a directory will be built once and cached. In the new model, even sub-packages will be recompiled as necessary.

A lot of failing scripts in a directory shouldn't break anything, it'll just make noise (the compilation errors).

That is strange as Ghidra/Features/Base/libbiz.aQute.bndlib-5.0.0.jar is present in my ghidra installation and is present in my classpath when debugging.

[ryanmkurtz]

Did you do a prepdev?

[astrelsky]

Did you do a prepdev?

Yes. The jar is in my ghidra installation and I can see it in "JAVA DEPENDENCIES" in my debugger.

Edit: I don't remember if I did the prepDev before or after running gradle clean. Also, I'm not running from the repo as I can't because the load is too much for my laptop.

[jpleasu]

@astrelsky we haven't been able to reproduce this issue -- have you made any progress?

[astrelsky]

@astrelsky we haven't been able to reproduce this issue -- have you made any progress?

I re-setup a local repo and recloned only the changes for jdk14 support. (Changes all Record to db.Record). I rebuilt everything, cleared the dev and osgi folders in C:\Users\astre\.ghidra\.ghidra_9.2_DEV and launched ghidra. I ran the AskScript.java and it compiles everything and then sits there and does nothing. I have to kill the jvm to close ghidra. After relaunching ghidra and trying to run it it then produces the error "Unable to locate script class: AskScript.java". Running AskScript.py works without a problem.

[astrelsky]

@jpleasu I just noticed this when starting up the vscode debugger. It may/may not be relevant. The library is coming from my gradle cache because I had gradle download the sources for debugging.

...
INFO  Opened project: --- (FileActionManager.java:346)
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.felix.framework.ext.ClassPathExtenderFactory$DefaultClassLoaderExtender (file:/C:/Users/astre/.gradle/caches/modules-2/files-2.1/org.apache.felix/org.apache.felix.framework/6.0.3/18d02dd467607cb61a8cf77c1847a733a417da76/org.apache.felix.framework-6.0.3.jar) to method java.net.URLClassLoader.addURL(java.net.URL)
WARNING: Please consider reporting this to the maintainers of org.apache.felix.framework.ext.ClassPathExtenderFactory$DefaultClassLoaderExtender
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
...

I also always got the warning for cglib as well so it may not be related.

[astrelsky]

@jpleasu I'm no longer receiving the NoClassDefFoundError after purging my ghidra build and installation and starting fresh with the minor jdk14 build changes. I'm now receiving a NoSuchMethodError likely because org.slf4j:slf4j-api:1.5.0 is being picked up from yajsw-stable-12.12 instead of org.slf4j:slf4j-api:1.7.25

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.felix.framework.ext.ClassPathExtenderFactory$DefaultClassLoaderExtender (file:/C:/Users/astre/Documents/ghidra_9.2_DEV/Ghidra/Features/Base/lib/org.apache.felix.framework-6.0.3.jar) to method java.net.URLClassLoader.addURL(java.net.URL)
WARNING: Please consider reporting this to the maintainers of org.apache.felix.framework.ext.ClassPathExtenderFactory$DefaultClassLoaderExtender
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
INFO  Packed database cache: C:\Users\astre\AppData\Local\Ghidra\packed-db-cache  (PackedDatabaseCache.java:64)
...
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/C:/Users/astre/Documents/ghidra_9.2_DEV/Ghidra/Features/GhidraServer/data/yajsw-stable-12.12/lib/extended/vfs-webdav/slf4j-jdk14-1.5.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/C:/Users/astre/Documents/ghidra_9.2_DEV/Ghidra/Features/GraphServices/lib/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: slf4j-api 1.6.x (or later) is incompatible with this binding.
SLF4J: Your binding is version 1.5.5 or earlier.
SLF4J: Upgrade your binding to version 1.6.x.
ERROR Task Error: compiling script directory - Uncaught Exception: java.lang.NoSuchMethodError: 'org.slf4j.impl.StaticLoggerBinder org.slf4j.impl.StaticLoggerBinder.getSingleton()'  (Task.java:133)
java.lang.NoSuchMethodError: 'org.slf4j.impl.StaticLoggerBinder org.slf4j.impl.StaticLoggerBinder.getSingleton()'
    at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150) ~[slf4j-api-1.7.25.jar:1.7.25]
    at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124) ~[slf4j-api-1.7.25.jar:1.7.25]
    at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412) ~[slf4j-api-1.7.25.jar:1.7.25]
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357) ~[slf4j-api-1.7.25.jar:1.7.25]
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383) ~[slf4j-api-1.7.25.jar:1.7.25]
    at aQute.bnd.osgi.Processor.<clinit>(Processor.java:98) ~[biz.aQute.bndlib-5.0.0.jar:?]
    at ghidra.app.plugin.core.osgi.GhidraSourceBundle.generateManifest(GhidraSourceBundle.java:824) ~[Base.jar:?]
    at ghidra.app.plugin.core.osgi.GhidraSourceBundle.compileToExplodedBundle(GhidraSourceBundle.java:1024) ~[Base.jar:?]
    at ghidra.app.plugin.core.osgi.GhidraSourceBundle.build(GhidraSourceBundle.java:605) ~[Base.jar:?]
    at ghidra.app.plugin.core.osgi.BundleHost.activateAll(BundleHost.java:708) ~[Base.jar:?]
    at ghidra.app.script.JavaScriptProvider.loadClass(JavaScriptProvider.java:116) ~[Base.jar:?]
    at ghidra.app.script.JavaScriptProvider.getScriptInstance(JavaScriptProvider.java:81) ~[Base.jar:?]
    at ghidra.app.plugin.core.script.GhidraScriptComponentProvider.getScriptInstance(GhidraScriptComponentProvider.java:642) ~[Base.jar:?]
    at ghidra.app.plugin.core.script.GhidraScriptComponentProvider.doRunScript(GhidraScriptComponentProvider.java:624) ~[Base.jar:?]
    at ghidra.app.plugin.core.script.GhidraScriptComponentProvider$5.run(GhidraScriptComponentProvider.java:611) ~[Base.jar:?]
    at ghidra.util.task.Task.monitoredRun(Task.java:126) ~[Generic.jar:?]
    at ghidra.util.task.TaskRunner.lambda$startTaskThread$1(TaskRunner.java:94) ~[Docking.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
    at java.lang.Thread.run(Thread.java:832) [?:?]

Also I only get the error message while debugging. When not in the debugger the error does not appear, the script doesn't run after compiling and I am unable to close ghidra after trying to run a script..

[jpleasu]

On linux, building with jdk14, gradle buildGhidra task, I can reproduce the hang.. from eclipse, I am seeing NoClassDefFoundErrors that makes no sense:

ERROR Task Error: compiling script directory - Uncaught Exception: java.lang.NoClassDefFoundError: ghidra/app/plugin/core/osgi/BuildError java.lang.NoClassDefFoundError: ghidra/app/plugin/core/osgi/BuildError
    at ghidra.app.plugin.core.osgi.GhidraSourceBundle.buildError(GhidraSourceBundle.java:204)
    at ghidra.app.plugin.core.osgi.GhidraSourceBundle.tryBuild(GhidraSourceBundle.java:780)
        ...

BuildError is a class in the same package as GhidraSourceBundle.. and every directory with a GhidraSourceBundle.class on my machine, in a jar or otherwise, also has a BuildError.class.

[jpleasu]

The JavaCompiler task closes the main classloader on call. If I override the close method of GhidraClassLoader to do nothing, everything works for me. I don't know if classloading is lazier in jdk14 or if closing the loader in the task is new, but I'm still working at it.

[jpleasu]

The removal of this test causes the framework bundle's classloader to be closed. I'll post a patch soon.

I haven't encountered what appears to be a classpath order issue with slf4j. Are you using the gradle ecilpse task?

[astrelsky]

The removal of this test causes the framework bundle's classloader to be closed. I'll post a patch soon.

I haven't encountered what appears to be a classpath order issue with slf4j. Are you using the gradle ecilpse task?

Yes. I have things layed out strangely in the gradle script so I can debug from vscode. Considering it doesn't occur outside of the debugger I may just need to reorder the dependencies.

[astrelsky]

@jpleasu deleting Ghidra/Features/Base/lib/slf4j-api-1.7.25.jar fixes the problem in the debugger. Might be something with eclipse.jdt.ls. Strangely enough I'm able to run scripts in the debugger without a problem with the file removed but not when using ghidra normally.

[jpleasu]

@astrelsky the issue arises when JavaCompiler.CompiliationTask#call is invoked before required classes are loaded. So a) if you aren't compiling or b) you've managed to load all necessary classes first you won't see the issue. Try "cleaning" first from the bundle manager (the old script directories dialog).. this will force the compile. Then restart Ghidra with fresh classloaders and immediately try to run a script from the cleaned directory, or re-enable the cleaned directory.

if you _are_ able to compile from eclipse without the fix, make sure you're actually running on jdk14 from eclipse.

[astrelsky]

@astrelsky the issue arises when JavaCompiler.CompiliationTask#call is invoked before required classes are loaded. So a) if you aren't compiling or b) you've managed to load all necessary classes first you won't see the issue. Try "cleaning" first from the bundle manager (the old script directories dialog).. this will force the compile. Then restart Ghidra with fresh classloaders and immediately try to run a script from the cleaned directory, or re-enable the cleaned directory.

if you _are_ able to compile from eclipse without the fix, make sure you're actually running on jdk14 from eclipse.

I actually cannot use Eclipse on my machine as the workspace never finished loading. I've given up on Eclipse a long time ago. When running in vscode I can confirm that the eclipse.jdt.ls is running on jdk14.

Launching ghidra normally from ghidraRun.bat. It took a while but I managed to get the python interpreter open and loaded so I could execute ghidra.util.classfinder.ClassSearcher(True, monitor) which would force refresh the classes and ensure everything is loaded. After it's completion I opened the BundleManager and hit refresh. After it "finished" I had the same results where most of ghidra became non-functional, in particular the gui elements and I could not close ghidra without ending the vm process. Upon relaunching eclipse the issue immediately reoccured as soon as ghidra attempted to restore the bundle state.

[jpleasu]

@astrelsky ClassSearcher will load extension point classes, not, for example, the bndlib classes or BuildError, I wouldn't expect it to make a difference. Actually priming the classloader with all of the classes used after the JavaCompiler task completes would be pretty tedious.

I think we've addressed all of the issues you've raised and I've prepared the patch. Is there anything else?

[astrelsky]

@astrelsky ClassSearcher will load extension point classes, not, for example, the bndlib classes or BuildError, I wouldn't expect it to make a difference. Actually priming the classloader with all of the classes used after the JavaCompiler task completes would be pretty tedious.

I think we've addressed all of the issues you've raised and I've prepared the patch. Is there anything else?

Not that I'm currently aware of.

[ghidra1]

Fix has been merged