Cxbx-reloaded: Regression Testing Tool

[LukeUsher]

As it is very easy to cause regressions when editing the OOVPA data structures, it would be nice if we had some tooling we could run to find symbols which were previously matched, but are no longer matched in new builds.

I was thinking we could have a simple command line utility that takes two log files outputting by Cxbx-Reloaded and compare the HLE symbol output. This utility should report when a symbol address differs, or when a function that was previously located is no longer found in the latest build.

In order to easily automate this, it would be good if there was a command line parameter to Cxbx-Reloaded itself that only performs the HLE symbol scan and no other activity.

An extract from a log file is shown below, to give an idea of what needs to be parsed. It would be useful if the _XXXX suffix could be disregarded, then the utility would still work even if we move a signature to a different XDK.

*******************************************************************************
* Cxbx-Reloaded High Level Emulation database last modified Sat Mar 18 22:14:13 2017
*******************************************************************************

HLE: Detected Microsoft XDK application...
HLE: Starting pass #1...
HLE: * Searching HLE database for XAPILIB version 1.0.5558... Found
HLE: 0x00155248 -> SetThreadPriority_3911
HLE: 0x001552F0 -> XRegisterThreadNotifyRoutine_3911
HLE: 0x001560B5 -> XMountUtilityDrive_4432
HLE: 0x001F4563 -> XInitDevices_5233
HLE: 0x001F5309 -> XGetDevices_4134
HLE: 0x001F532B -> XGetDeviceChanges_5233 (Disabled)
HLE: 0x001F5002 -> XInputOpen_4361
HLE: 0x001F5064 -> XInputGetCapabilities_5558
HLE: 0x001F523C -> XInputGetState_5558
HLE: 0x001F52AF -> XInputSetState_5233
HLE: 0x00153A64 -> XLaunchNewImageA_5558
HLE: 0x0018EA4E -> GetTimeZoneInformation_3911 (Disabled)
HLE: 0x001F5BC5 -> XID_fCloseDevice_5558 (XRef)
HLE: 0x001F5058 -> XInputClose_5558
HLE: 0x0018EBB0 -> OutputDebugStringA_3911
HLE: 0x001A12D0 -> EmuD3DDeferredRenderState
HLE: 0x001A0F60 -> EmuD3DDeferredTextureState
HLE: * Searching HLE database for D3D8 version 1.0.5558... Found
HLE: 0x0019D880 -> Direct3D_CreateDevice_4627

[Sarkie]

Can you upload another log to compare, what's there and what's not usually?

Here's my first regex attempt.

[LukeUsher]

The entries that are there/not there normally vary on a per-game basis, the general idea is that if an address of a function changes between two log files from the same game, or if a function is detected the old log but not in a new log, it needs investigating.

That said, I'll upload a collection of log files after work

[Sarkie]

ah right, understood I believe.

tool.exe oldgameid.log newgameid.log

"ERROR
Latest log has the following missing:
XInputGetState
EmuD3DDeferredTextureState

Latest log has the following added:
XNewMethodNameHere
"

"Address of method XYZ has changed"

[LukeUsher]

Nice.

Knowing which functions have been added will also help identify titles that have potentially improved based on the changes too, so this would help with both cases!

[jarupxx]

How are things going? I found a application to differs the logs. The name is DF and the developer is MYON.
Homepage is Japanese but I believe application is displayed in English.

You can download it by pressing the green button 3 times.

• Download from here
• Go to the download page
• Download now
  http://www.vector.co.jp/soft/win95/util/se113286.html
  

[PatrickvL]

Thanks for the tip. I'm using BeyongCompare (a commercial tool), but there are lots of alternatives http://alternativeto.net/software/beyond-compare/ for a diff tool like that.

[jarupxx]

I wrote a batch file to reduce regression testing effort. Run this batch file, logs are generating automatically.
Please specify the folder according to your environment.

SET WORKING_FOLDER=D:\git\Cxbx-Reloaded\build\win32\Release\
SET MASTER_FOLDER=D:\git\Cxbx-Reloaded-master\build\win32\Release\
SET APP=your\diff\tool.exe

SET WORKING_EXE=%WORKING_FOLDER%Cxbx.exe
SET MASTER_EXE=%MASTER_FOLDER%Cxbx.exe

reg add "HKCU\SOFTWARE\Cxbx-Reloaded" /v KrnlDebugFilename /t REG_SZ /d #3925-WORKING.txt /f
Start /MIN %WORKING_EXE% "C:\3925\default.xbe"
ping 127.0.0.1 -n 20
taskkill /F /IM Cxbx.exe

reg add "HKCU\SOFTWARE\Cxbx-Reloaded" /v KrnlDebugFilename /t REG_SZ /d #3925-master.txt /f
Start /MIN %MASTER_EXE% "C:\3925\default.xbe"
ping 127.0.0.1 -n 20
taskkill /F /IM Cxbx.exe

reg add "HKCU\SOFTWARE\Cxbx-Reloaded" /v KrnlDebugFilename /t REG_SZ /d #4134-WORKING.txt /f
Start /MIN %WORKING_EXE% "C:\4134\default.xbe"
ping 127.0.0.1 -n 20
taskkill /F /IM Cxbx.exe

reg add "HKCU\SOFTWARE\Cxbx-Reloaded" /v KrnlDebugFilename /t REG_SZ /d #4134-master.txt /f
Start /MIN %MASTER_EXE% "C:\4134\default.xbe"
ping 127.0.0.1 -n 20
taskkill /F /IM Cxbx.exe

%APP% %WORKING_FOLDER%#3925-WORKING.txt %MASTER_FOLDER%#3925-master.txt
%APP% %WORKING_FOLDER%#4134-WORKING.txt %MASTER_FOLDER%#4134-master.txt

[donwayo]

I'm working on a sort of database for Xbox xbe's, and then use this to drive regression tests. So far the xbe database is coming along nicely.

My plans for the regression tool are:

• [x] Parse xbe dump files created by Cxbx and Dxbx and create a database of them.
• [x] Parse xbe files and extract information from them like Cxbx dump files.
• [x] Verify xbe file signature to filter out unreliable data.
• [x] Automatic cxbx binary download from Appveyor (can be filtered by tags, pull requests, etc.)
• [x] Attempt to run a set of games/samples in the new cxbx binary build (like jarupxx's above script)
• [ ] Collect the HLE detection debug output, store it in the database, and compare it to a baseline.
• [ ] Upload results to database so all devs can access them.
• [ ] Provide a script for doing this off-line as well.

You are welcome to try it out, just send me a message on Gitter after you have signed in with Github so I can activate your account: https://xbox-db.herokuapp.com/