Ghidra: Issues with running Ghidra tests in CI

[kant2002]

I setup mirror for the Ghidra, and currently able build Ghidra on Linux and Mac, on public CI.
But when I add unit tests and even worse integration tests, I have an problem with running tests in CI environment. I completely understand that this may be not your concern, but maybe you have some advice based on your existing experience with running CI for Ghidra.

Let's start with running gradlew unitTestReport
Here the error messages which generated on Linux

java.lang.IllegalArgumentException: java.net.MalformedURLException: unknown protocol: ghidra
java.lang.NoClassDefFoundError: Could not initialize class sun.awt.X11.XToolkit
java.lang.NoClassDefFoundError: Could not initialize class docking.widgets.table.constrainteditor.EnumConstraintEditor
java.lang.NoClassDefFoundError: Could not initialize class docking.widgets.table.constrainteditor.UnsignedLongRangeConstraintEditor
java.lang.NoClassDefFoundError: Could not initialize class docking.widgets.table.constrainteditor.DoubleValueConstraintEditor
java.lang.NoClassDefFoundError: Could not initialize class docking.widgets.table.constrainteditor.DateRangeConstraintEditor
java.lang.NoClassDefFoundError: Could not initialize class docking.widgets.table.constrainteditor.IntegerRangeConstraintEditor
java.lang.NoClassDefFoundError: Could not initialize class docking.widgets.EmptyBorderButton
java.lang.NoClassDefFoundError: Could not initialize class javax.swing.RepaintManager
java.lang.NoClassDefFoundError: Could not initialize class docking.DockingUtils
java.lang.NoClassDefFoundError: Could not initialize class java.awt.GraphicsEnvironment$LocalGE
java.awt.AWTError: Can't connect to X11 window server using ':0.0' as the value of the DISPLAY variable.

java.lang.AssertionError: Failed to parse expected composite (see log)

All except latest one clearly from inability to see X11 server in CI environment. Last one not clear for me since I have to debug that, or at least take a look (not have time yet).
What I want to have right now is ability to run tests which does not involved in the UI part. So here my quuestions:

1. How can I run unit tests without running test which involved in UI. I attempt to run gradew unitTestReport -Djava.awt.headless=true and try to play with many other different combinations without luck. But funny things I found that you have that: https://github.com/NationalSecurityAgency/ghidra/blob/master/gradle/javaTestProject.gradle#L150 ,
2. Is it possible to make java.awt.headless=false switch configurable somehow? Preferable in a way which easy to change in CI environment?
3. If I annotate tests which are require to use X11 with some category will this change be accepted by Ghidra? There ~450 tests failing due X11 dependency now. I promise to update documentation to reflect that this is possible, or do any similar activity if this is required to make that happens.
4. How can I run unit tests just for single project in Ghidra? I assume gradlew unitTestReport -PDB but I it runs tests for Base and other projects, which PDB depends on. This is defeats purpose running tests for single project. This allow me to run less tests in one build. Public build has a limit of 1 hour, but I want to run integration tests as well and they do not fit in that time limit when I simply run gradlew integrationTestReport

These build errors prevent me from at least partial validation of the status of my fork.

Link to build with errors, in case sombody interested: https://codevision.visualstudio.com/Ghidra/_build/results?buildId=10106&view=ms.vss-test-web.build-test-results-tab
In case if somebody super-interested here link to build definition: https://github.com/kant2002/ghidra-official/blob/dev/.azure/build.yaml

[devnoname120]

Have you tried using a virtual framebuffer such as Xvfb?

You would do something like this:

Xvfb :99 &
export DISPLAY=:99

[dev747368]

Yes, Xvfb is what we use.

[dragonmacher]

This is an example of how you can run a single test using gradle (running from the repo root directory):

gradle :Docking:test --tests TaskDialogTest -x sleighCompile -x buildHelp

'TaskDialogTest' is the test name to be run in this example
'Docking' is the module name.

'test' signals a unit test; 'integrationTest' would signal an integration test. Unit tests live under '/src/test'; integration tests live under '/src/test.slow'

The '-x' options are to tell our current build system to ignore those items that are known to be slow and are not necessarily needed for any given particular test.

[dragonmacher]

We do have other scripts we use to run our full test suite. These scripts handle all of the complicated environment setup. For reasons that are unclear to me at the moment, these are not included in this repository.

[dragonmacher]

Regarding the X11 annotation comment, this should not be needed. Ultimately, our test harness is designed such that the developer only needs to extend the correct type of abstract test. After that, the test harness understands how to configure the environment and run the tests. For this to work correctly, we rely on the aforementioned test setup scripts.

[kant2002]

Thanks to all of you guys. It really helps me move forward. I still have issues with Xvfb, but for now I suspect that this is more related to Azure Dev Ops.

@devnoname120 when I run Xvfb :99 &
I strangely have following error during build.

> Failed to create MD5 hash for file '/home/vsts/work/1/ghidra.bin/Ghidra/Features/GhidraServer/yajsw-stable-12.12.zip' as it does not exist.

Not sure why ghidra.bin appears in the path. If something is ticking for any of you, I would appreciate help.

@dragonmacher Command line for running tasks for separate projects would be enough for me to play with for now.

[kant2002]

Now I have 7 failed tests:
2 tests

testFullyZoomedOutOption
testFullyZoomedOutOption

failed at the

at ghidra.app.plugin.core.functiongraph.AbstractFunctionGraphTest.assertZoomedIn(AbstractFunctionGraphTest.java:2170)

Which corresponding to the following code

    Rectangle cursorBounds = v.getCursorBounds();
    Window graphWindow = windowForComponent(getPrimaryGraphViewer());
    Rectangle windowBounds = graphWindow.getBounds();
->> assertTrue(windowBounds.contains(cursorBounds));

This is somehow UI related I suppose.

Another 5 tests

testSimpleUnion
testComplexStructureWithBitFields
testSimpleStructureWithBitFields
testComplexStructureWithBitFields2
testMoreComplicatedStructure

failed with message Failed to parse expected composite (see log) and in the logs there indication that somehow structures different.

For example

14:08:52 ERROR Expected and result differ: expected line 5, column 20   (CompositeTestUtils.java:69) 
14:08:52 ERROR Expected composite:
   /union
    Aligned
    Union union {
        0   uchar   1   a   ""
        0   union_s_1   1   _s_1   ""
        0   union_s_2   1   _s_2   ""
        0   ushort   2   f   ""
     }
     Size = 2   Actual Alignment = 2
     /union/union_s_1
     Aligned
     Structure union_s_1 {
        0   uchar:4(0)   1   b   ""
        0   uchar:4(4)   1   c   ""
     }
     Size = 1   Actual Alignment = 1
     /union/union_s_2
     Aligned
     Structure union_s_2 {
        0   uchar:4(0)   1   d   ""
        0   uchar:4(4)   1   e   ""
     }
     Size = 1   Actual Alignment = 1   (CompositeTestUtils.java:79) 
     14:08:52 ERROR Result composite:
     /union
     Aligned
     Union union {
        0   uchar   1   a   ""
        0   union_s_1   2   _s_1   ""
        0   union_s_2   2   _s_2   ""
        0   ushort   2   f   ""
     }
     Size = 2   Actual Alignment = 2
     /union/union_s_1
     Unaligned
     Structure union_s_1 {
        0   uchar:4(0)   1   c   ""
        1   uchar:4(0)   1   b   ""
     }
     Size = 2   Actual Alignment = 1
     /union/union_s_2
     Unaligned
     Structure union_s_2 {
        0   uchar:4(0)   1   e   ""
        1   uchar:4(0)   1   d   ""
     }
     Size = 2   Actual Alignment = 1   (CompositeTestUtils.java:80) 
14:08:52 DEBUG  ✖ FAILED Test: CompositeMemberTest - testSimpleUnion ✖     (AbstractGenericTest.java:113) 

[ghidra1]

CompositeMemberTest tests are currently passing in master. Are you using the latest?

[kant2002]

I update yesterday, and now 10 commits behind master, but I see these tests failing and earlier. I assume that this maybe something missing in my CI setup.

[kant2002]

Now, I would say that only testFullyZoomedOutOption does not working. Other tests are working fine.

[ryanmkurtz]

Some of our tests aren't written quite correctly, and are susceptible to time-related failures. We fix these as we find them. testFullyZoomedOutOption is probably one of these.

[kant2002]

thanks for information. I will keep an eye on it. but for now, I think this issue can be closed. At least I have better understanding. @ryanmkurtz if you can take a look at #832 where I made attempt to capture small tips which you give to me in the docs.

[ryanmkurtz]

K I'll take a look, thanks.