Cxbx-reloaded: Dolphin (5849), DolphinClassic (5849), Fire (5849) and Gamepad (4134) Regression

[Voxel9]

All XDK Samples listed in this issue's title (and probably more) will boot without any issues, albeit broken graphics, in cdc5d5cd and older commits. However, as of ecded47 these samples seem to no longer boot at all; only exceptions occur.

Example of the Gamepad sample working on the former, then being unable to boot on the latter:

KrnlDebug.txt (KrnlDebug log for ecded47 build)

[PatrickvL]

It's quite strange how an improvement in memory querying seems to lead to not booting these XDK samples anymore. Are you sure it's that exact commit that broke it?

[PatrickvL]

I suspect these regressions are actually caused by one of the OOVPA changes @jarupxx made. I'm sure this will be corrected soon enough.

[Voxel9]

I tested jarupxx's last commit before your MemoryManager improvement, and can confirm that the XDK Samples could boot just fine, so yeah, I am certain that the exact commit was ecded47.

[PatrickvL]

Okay, thanks for verifying that. To be absolutely clear - is this https://github.com/Cxbx-Reloaded/Cxbx-Reloaded/commit/ad659ca408a99870d0b1d6e4e6e3e1c2e8d5b535 the commit in which the XDK samples still ran?

[Voxel9]

Yes, that's the one :)

[LukeUsher]

I'll do some hardware testing of the memory manager behavior next week, see if the results from Cxbx and real hardware differ.

[PatrickvL]

Perhaps QueryAllocationSize shouldn't return remaining size in bytes, but something else, like remaining size in pages, or the size of the original allocation (either in bytes or in pages), or the base address of the allocation, or even something more obscure?

[LukeUsher]

My plan is to allocate some memory via various methods, one being MmAllocateContiguousMemory, run QueryAllocationSize on them, dump the results to a text file on the HDD then compare the text files generated by Cxbx-Reloaded and real hardware, that should provide some pointers on what we're doing wrong

[PatrickvL]

Btw, thanks @Voxel9, for bringing this under our attention, as it's exactly the little things like this, that will truly get us accurate emulation - it's much appreciated!

[LukeUsher]

I believe that MmQueryAllocationSize returns the total size (in bytes) of the pages used by the memory region pointed to by the input. Rather than calculate the remaining size of the buffer and return that, I believe that it finds the base address of the allocation.

Eg. MmAllocateContiguousMemory(0x1000) returns 0x8001000.

MmQueryAllocationSize(0x8001050) would look through a page table (or our memory block info), find the page that contains this address, and return the total size of the pages the allocation uses (in this case 0x1000). Currently we return the remaining space in the page.

Just something to play with until I find time for the HW testing

[LukeUsher]

Update: changing
ret = info->block.size - ((size_t)addr - (size_t)info->block.addr);
to
ret = info->block.size;

Fixed the GamePad sample for me.

[PatrickvL]

@Voxel9 Can you verify that PR #363 fixed these samples' regressions for you too?

[PatrickvL]

@LukeUsher I'm still interested to see what would be returned if the call mentioned an address from a page after the starting page

[LukeUsher]

That's what the hardware testing will hopefully discover

[ObiKKa]

I've tested it. I was able to boot it up with the latest master build. But original poster above should do likewise. The D-Pad controls didn't work on the main menu. So I entered the Gamepad screen and saw this shot below. I think graphics seem to be missing in the center - I can only see the ring which moves when I use the left analog stick buttons on my keyboard. Right analog stick's button presses worked also but seemed to be slow to register on the screen's text lines perhaps.

Also tried the Hardware Video mode. The analog nub is emulated a bit better, more surfaces showing. But broken in two pieces in the middle.

KrnlDebug_(2017-04-13).txt

[Voxel9]

Yep, can confirm the majority of the regressions have been fixed. But a lot of the graphics seem to have broken since they were last working, especially in the 4627 samples (probably side-effects of previous commits); the 5849 samples seem to have remained the same though.

Dolphin (4627) Before fix:

Dolphin (4627) After fix:

FuzzyTeapot (4627) Before fix:

FuzzyTeapot (4627) After fix:

As for Gamepad, it seems that both 4627 and 4134 end up displaying none of the gamepad model at all anymore for me:

Gamepad (4627/4134) Before fix:

Gamepad (4627/4134) After fix:

Only thing I have noticed is that Fire (5849) still seems to be in regression, hence why I said the majority of the samples no longer regress.

[LukeUsher]

Still investigating, but the graphical corruption in the XDK samples seems to have started when we introduced the Memory Manager. It would seem there is still something we don't have quite right yet. Perhaps we really do need to implement emulated page tables.

[PatrickvL]

@LukeUsher The most important change the introduction of the new memory manager brought, is that since then addresses in contiguous memory (0x80000000 and up) are returned - this could lead to new code-paths being taken, perhaps even conversion back to non-GPU addresses (by anding with 0x7FFFFFFF) for some operations. If that's the case, we should devise a solution that would map these addresses to the same data. For that, memory-mapped files could be a solution, as those can be mapped onto multiple virtual memory addresses. For this we do need to use an address-range that's both available _with_ and _without_ the high-bit set.

[LukeUsher]

Mapping these same addresses is not really possible, eg: 0x80010000 would be 0x00010000, which resides within our XBE, also I don't think this is the issues (although it could be related, we possibly still have some code laying around in Cxbx that makes the < 0x80000000 assumption.

[PatrickvL]

But we could raise the addresses in the 0x80000000 range up with 128MB, which would circumvent this problem.

[BenNottelling]

Is this still a valid issue?

[PatrickvL]

XDK Samples and Tutorials are (and will stay) a very important "first-line-of-defense" test-set, as these are small (so can be ran through quickly) and use various methods of rendering. Our focus should be to get them all running flawlessly, as that will significantly raise the quality of larger executables too.

[BenNottelling]

@PatrickvL Haha I'm just asking if these still have regressed I know these are important :)

[PatrickvL]

@BenNottelling As far as I know, these are still regressed. I've added the "high-priority" label to this issue, as for all issues marked with "regression", this is the most important one.

[PatrickvL]

This issue was moved to Cxbx-Reloaded/game-compatibility#146