Libgdx: It would be possible to unit test a GDX game if not for one thing.

[travishaynes]

The headless backend helps with writing tests, up until you need access to something that uses OpenGL, like a SpriteBatch. Some people get around this by mocking Gdx.gl with something like Mockito.

I personally think that mocking an entire library like OpenGL is bad practice. So my solution is to just create a hidden window that has an active OpenGL context. Check out this gist for an example. With that I can test anything that uses OpenGL. No need to mock anything.

This would be possible out of the box if not for one simple thing: The backend starts the main loop inside the constructor. That's the only reason I had to write the TestApplication class in that gist. I'm talking about this code right here.

If that was in a separate method, you could easily unit test a game just by instantiating the application with an invisible window, and disabling the audio.

ApplicationListener game = new Game();

Lwjgl3ApplicationConfiguration config = new Lwjgl3ApplicationConfiguration()
config.setInitialVisible(false);
config.disableAudio(true);

Lwjgl3Application testApplication = new Lwjgl3Application(game, config);

The only difference would be that you wouldn't start the app and run the main loop. Then anything that makes calls to Gdx.gl would be available to your tests because you'd actually be testing against a real OpenGL context.

Then the desktop launcher would just need one extra step to start the game. For example, if the method that starts the main loop is called run:

new Lwjgl3Application(new Application(), config).run();

[tommyettinger]

I kinda doubt the libGDX devs will want to break compatibility for every single existing game for this. Unit tests, like the ones libGDX uses to test PRs, sometimes need to run on headless machines and don't have this as a valid option anyway. Invisible and silent unit tests also have a limited range of behavior they can actually detect; games often need a human being to point out that, say, too many zangulorgs are spawning in an area, or in a visually-obvious pattern, and not enough blanyayools spawn in the same place, or their AI makes poor decisions. When unit tests are an excellent option, they are often checking non-subjective, non-perceptual things, like "Was an Exception thrown and uncaught during routine data structure manipulation?" or "Does the name entry validation function reject the empty string as a name?" Sometimes these need OpenGL to check, like when examining errors in shader compilation, but many more OpenGL issues need to be caught on corner-case hardware than can be caught on one machine with invisible graphics.

It sucks, but I don't see unit testing catching on in game development (outside of situations where the logic can be tested independently) any time soon.

[travishaynes]

@tommyettinger You make a good point about not breaking backwards compatibility. I disagree with you on the rest of what you said. Testing a game presents all of the exact same challenges you face testing anything with a graphical user interface. I've unit tested games before. The scenarios you mentioned were exactly the kind of things I did test.

To make it backwards compatible, we just need to add a configuration option. We could call it startOnInitialize, which would default to true. Then you just have to set that to false in your tests.

[tommyettinger]

So unit tests can catch issues from when a user runs on a Core 2 Duo with integrated-only graphics? What about low-end Android phones with eccentric GLSL compilers? Can they tell when sounds are playing over each other at deafening volume? Can they predict usability challenges on screens with unusual aspect ratios, or very high DPI? Can they tell what parts of the picky Android audio situation with libGDX are expected problems with libGDX (or Android at a particular OS version) and which are problems in the tested code? What about Linux users running a distro and WM that came out yesterday? What about MacOS users who could have the rug pulled out from a critical OpenGL API in the next OS version? When issues get reported by users, they very frequently are related to one or more of these.

I worry about getting a false sense of security from passed unit tests when the application isn't being tested in the wild.

P.S. How did you unit test for AI making poor decisions? Seems like a challenging topic under active research.

[travishaynes]

@tommyettinger You might like this article. As the author puts it, TDD is merely a way to help us developers. Like you, the author doesn't think unit testing OpenGL is really all that beneficial.

I opened this issue to point out that it would be possible to write unit tests with an active OpenGL context out of the box if only the main loop was refactored away from the constructor. I didn't intend to start a debate about whether or not TDD is a good design principal for game development. If you want to continue that discussion, feel free to shoot me an email.

With an active OpenGL context, you could access any part of the GDX library in your tests without exceptions. That's the main benefit this would provide. To those who do write unit tests, it would be extremely helpful.

[Darkyenus]

I know that you mentioned that you don't want to mock OpenGL, but since testing OpenGL is pretty much wasted time anyway and you don't need OpenGL to actually render anything, wouldn't it be easier to implement unit testing mode into the headless backend? It seems to be much better suited to what you want to do and modifying the main loop there would be much less controversial than in any other backend (also, it would be much safer and more reliable).

Headless backend does not currently set Gdx.gl to anything, but it would be an easy change to set it to something that does nothing. You could even do some basic sanity checking there, if you wanted to.

[crykn]

The problem with mocking OpenGL is that some classes depend on certain returned values (see for example GLFrameBuffer#L240). In those cases it would definitely be easier to just use a modified headless client.

A compromise would be to make loop() protected, so a subclass could just override it. Currently a subclass needs to override the constructor and as (nearly) all methods called inside of the constructor are private (loop(), cleanupWindows(), cleanup()), their code has to be copied as well.

[travishaynes]

Years ago I was using the JGLFW backend in my tests, because it was setup that way. That made it really easy to test things. Unfortunately, that back-end was removed, so I've found myself having to hack into the LWJGL3 backend to get the same results.

[Darkyenus]

I reckon that it would not be too hard to implement believable mocking for OpenGL for some basic results, just so that everything thinks that everything was successful. And the idea to open up HeadlessApplication for modification and hacking seems reasonable as well. (I would probably be against same opening up in the other backends for the sake of future compatibility.)

[travishaynes]

Why mock something you don't have to? It's no trouble at all to create a real OpenGL context that you can call from your tests. This is basically what that would look like using GLFW directly:

glfwInit();
glfwWindowHint(GLFW_VISIBLE, GLFW_FALSE)
long windowHandle = glfwCreateWindow(640, 480, "Test", 0, 0);
glfwMakeContextCurrent(windowHandle);

After that any call to OpenGL is valid. Nothing is actually rendered to the screen, because the window is hidden.

The most difficult part of this is creating the back-end for GDX, which has already been done. It just needs to be refactored a little to make it easier to work with.

[Darkyenus]

It's no trouble at all to create a real OpenGL context

It is trouble on a headless linux machine, when you want to run unit tests in parallel, when you are doing some code that messes with GL state or native initialization, etc.

[Tom-Ski]

I agree with @Darkyenus, and this would typically be the most common way to be running unit tests anyway, on some headless CI. That said, if someone wants to add support for unit testing without mocking for whatever reason, as long as it doesn't harm or cause huge breaking changes, I don't see anything to go against this.

Wouldn't just being able to override the loop method provide a solution in your case?

[travishaynes]

@Darkyenus I guess trouble was the wrong noun to use. What I meant was simple. Running the tests in parallel in a headless environment would indeed be very challenging.

@Tom-Ski Yes, being able to override the main loop would provide a solution for this case, as well as the cleanup methods, as @crykn mentioned

[MobiDevelop]

It really depends on what you want to test I guess. Testing how some 'thing' handles various OpenGL states is an entirely different animal than testing application code.

Testing application code probably doesn't (normally) care too much about the weird OpenGL handling on various setups. That said, it's probably not really unit testing but integration or functional testing at that point (for the semantics nerds, like me).

[NathanSweet]

You guys are really writing tests?!

It's obviously very difficult to test UI. IMO the most that's worth doing is a smoke test that runs the app (as a user would, with OpenGL) and simulates some user actions to exercise various code paths. This tells you if a build is worth testing further. Other than that, the best thing to do is separate testable app code from UI, then you can test the parts that are reasonable to test and leave the UI to the test monkeys (users).

[travishaynes]

@NathanSweet How do you write any code without tests? How does that even work?

[NathanSweet]

I just roll my face on the keyboard and then let the users find any bugs. I find it helps a lot to avoid writing bugs in the first place.