Godot: Automatic behavior testing

[shackra]

I posted this in the forums and I'm making a request in the bug tracker so maybe the devs can keep this under their radar.

As I software developer, I tend to driven my development of software thought tests to assure the correctness of behavior with the code I'm writing and because this method of software development helps me find bugs when I introduce them by accident during the development process.

So, I was wondering why Godot have no automatic testing facilities to allow game developers assure the game behaves as they want. I see this feature for Godot not only useful during development, but also afterwards, specially for those authors writing books or making on-line classes on Godot: for them automatically checking what APIs are outdated or which lines of code don't work as they supposed to in new versions of Godot, making it easy to updated their material where it is required. Nowadays is very easy to find outdated code for different programming languages or frameworks, and Godot isn't going to be an exception.

In my my mind, this could consists of:

• a way to simulate player input (touch in screen with finger or mouse pointer, button or key pressed, action triggered)
  
  
  
  • A way to move a node using simulated inputs until the node reach the target point, maybe?
  
  
• a way to gather nodes properties after the input is sent and allow value comparison and yell error when the developer see it fit.
• a way to set fixtures and tear down them before and after each test.
• a way to take screenshots between steps of the test.
  
  
  
  • does it makes sense to record a video during each test? I mean, in the gamedev context.
  
  
• The tests should be triggered from the command line (say: godot --test-node nodescene.tscn --test-script mytestscript.gd or godot --run-tests in the current game project directory and let Godot search a folder called "testing" with GDScripts defining the tests).

[bojidar-bg]

To be fair, Godot probably supports all of this (as long as globals and autoloads aren't used too much), you just have to bundle it in a nice GDScript framework :smile: :

• Simulating player input is already possible by feeding custom InputEvent-s docs
  
  
  
  • Since inputs would count as actions just fine (might need slight tweaking in case they don't), nodes would move exactly as in the real game... unless you start pressing real inputs (workaroundable, as one can make a fake SceneTree, and pass only custom inputs)
  
  
• Node properties of all kinds can be gathered via functions (get_pos...) or as properties (get("transform/pos")...). Comparing those properties should we relatively easy
  
  
  
  • Since properties might change between observations (or something else might happen, better not to risk it), one might pause the tree before running through the nodes.
  
  
• Setting up fixtures is probably something scenes should do. To tear a fixture down, kill the scene (or even the scene tree), then instance another one (Note: might need killing/resetting global singleton nodes).
  
  
  
  • Again, a fake SceneTree is going to be easier to work with
  
  
  • Also, worth to note, anything that makes use of OS time, or ticks since start is not mockable, so code wisely :wink:
  
  
• Making screenshots is very easy in Godot, and saving them should be simple as well docs
  
  
  
  • Videos are going to be harder to do, though making a 30fps one doesn't seem impossible (this might deserve another issue if it is really important).
  
  
  • Instead of videos, you can make a complete copy of the tree saved as a scene, with scripts removed, and add an AnimationPlayer with a Animation resource generated from all (or at least changed) properties of all nodes in the scene. Added and removed nodes might be simulated with hiding and showing, or with func call tracks.
  
  
• Triggering a script from the command line is already possible with the -s flag docs. Probably a custom scene that does everything would be more useful though, since it would be able to have access to Singleton autoloads.
  
  
  
  • As an alternative solution, you can check OS.get_cmdline_args() for --run-tests in the main scene, and then change into the testing scene.
  
  

[Zylann]

Small note: a single scene doing all tests won't be efficient because a test could crash the engine or propagate errors for next tests, which would force to fix issues one by one.

[vnen]

The hardest part in all of this IMO is making scenes and scripts completely isolated to be able to unit test, especially the singletons. Inherited and instanced scenes can get in the way, though if you rely on placeholder scenes and do a lot of signal passing you might be able to manage it. You will likely need to abstract away all the unmockable parts, like OS and FileSystem functions.

If you build a script which extends MainLoop, it's relatively easy to follow @bojidar-bg's advice to create a SceneTree manually and control the passing of frames by calling _iteration(delta) and _idle(delta). This also allows you to pass fake input events.

[shackra]

@vnen ok, let's say I do as @bojidar-bg suggests: how would I trigger the tests?

NOTE: This may sounds like a silly question, but bear with me.

[ghost]

First of all thank you for your report and sorry for the delay.

We released Godot 3.0 in January 2018 after 18 months of work, fixing many old issues either directly, or by obsoleting/replacing the features they were referring to.

We still have hundreds of issues whose relevance/reproducibility needs to be checked against the current stable version, and that's where you can help us.
Could you check if the issue that you described initially is still relevant/reproducible in Godot 3.0 or any newer version, and comment about it here?

For bug reports, please also make sure that the issue contains detailed steps to reproduce the bug and, if possible, a zipped project that can be used to reproduce it right away. This greatly speeds up debugging and bugfixing tasks for our contributors.

Our Bugsquad will review this issue more in-depth in 15 days, and potentially close it if its relevance could not be confirmed.

Thanks in advance.

Note: This message is being copy-pasted to many "stale" issues (90+ days without activity). It might happen that it is not meaningful for this specific issue or appears oblivious of the issue's context, if so please comment to notify the Bugsquad about it.

[shackra]

Currently Godot 3.0.2 has only this reference on anything related to testing:

--test <test>: Run a unit test ('string', 'containers', 'math', 'render', 'multimesh', 'gui', 'io', 'shaderlang', 'physics', 'oa_hash_map').

But there is the Gut project that seems to cover the requirements of this feature request.