Pcsx2: Smuggler's Run: Missing HW Shadows/Incorrect GS Dump Playback

[MrCK1]

PCSX2 version:
v1.5.0-dev-2104-g49840bb

PCSX2 options:
Defaults, Safest

Plugins used:
Defaults. Both GSdx plugins used in the emulator and GSDumpGUI are matching versions

Description of the issue:
In Smuggler's Run, shadows near the player have never been rendered in any HW renderer. In addition to this however, the GS Dump shows an outline of the shadow in HW mode, where they were missing in-game. In all instances, distant shadows are still rendered normally.

How to reproduce the issue:
In-game: Boot game, drive to an area with shadows.

GSDump: Select any HW renderer, find first frame and observe shadows. As a side note, you can see the alpha texture of the trees shadows flicker at the top when stepping to another frame in the GSDump.

GSDump
SmugglersRun_ShadowsHW.zip

Last known version to work:
Never

PC specifications:
Intel Core i5-6600k, GTX 1070, Windows 10 Home 64-bit

In-Game (HW):

GSDump (HW):

[MrCK1]

Forgot to add in the report that both dumps I captured in SW and HW have the playback issue. The dump that was uploaded was captured in the HW renderer.

I also got some more info from GSdx here that could be of use in the future. :)

Complex Alpha Test
Alternate ATE handling: ate_RGB_then_ZA

[MrCK1]

Additional tests on real hardware shows that the game uses pre-baked shadows at a distance and transitions to stencil shadows (I hope I got this right) when the player gets closer. I believe this is why the shadows are disappearing when the player gets too close in HW renderers.

Here's an in-game example

50/50 (Top half is pre-baked, bottom half is stencil shadow)

Full image

Transition layer (stencil shadow is missing):

Pre-baked shadow (photo was cropped, but taken further away)

[gregory38]

Technically there is no stencil on GS.

Is SW renderer impacted (in game) ?
Did you try zerogs ?

[MrCK1]

SW renderer is not impacted.

I haven't tried zerogs yet, but some tests a while back with older versions of GSdx around 0.9.6; and they didn't have working shadows in HW either.

[gregory38]

If sw renderer is fine, it is likely a readback of GS memory by EE which is later transferred again to GS. I added some logs on readback and texture is dumped

[MrCK1]

@gregory38 So is there anything else I should get? I had no luck with captured frames from my graphics debugger, it doesn't get trace logs or anything unfortunately.

[gregory38]

Enable gsdx hidden dump option. Do you see gl error in your debugger (which is btw)?

[MrCK1]

I use RenderDoc, it's cross-platform as well: https://renderdoc.org

"Enable API validation" is the option used in the program to get the GL debug logs, that's how I got the logs from Guitar Hero earlier. Unfortunately, this game doesn't print any GL debug, I don't get the individual drawcall frames either. It's a hit or miss depending on the game itself. In addition, I use GSdx-dbg when capturing.

I'll post example later frames later.

[MrCK1]

For some reason I can't edit my original comment, guess Github is glitching out.

Anyways, I got a successful frame capture after some tinkering. For some reason, the missing alpha textures are positioned in an odd orientation. As for debug messages, here are the Medium severity ones:

[gregory38]

All gl messages are stored in a debug file (should be current dir). It will greatly help you to understand what happen. Debug build is quite verbose because you have all TC information.

The above message is about a texture depth effect. Typically the green channel of depth buffer (bit 8-16) are copyied into the alpha channel of a color buffer. Hint if the detection of the effect doesn't work, it is wrongly replaced by a FBMSK emulation. If you spot those message, there are likely a bug.

You need to check the code of InvalidateLocalMem. This function will transfer code from the GPU to the GS mem which could be send to EE. Check the code there are some GL related messages.

[MrCK1]

Here's the log: SmugglersRun_opengl_debug_hw.txt

2nd Log:
GSdx_opengl_debug_hw.txt

[gregory38]

Log analysis. There is a color effect based on channels shuffle (from ba to rg). Around draw call 13. The source is located at 0x1180 and the destination is located at 0x1200. My feeling is that we have some kind of overlapping of color buffers/textures.

Edit: next step, to confirm the behavior, is to get the resolution of the color buffer. The width can be extrapolated from FBW register value (...._context.txt file) i.e. width = 64 * FBW. Above address 0x1180/0x1200 are in block unit.

[MrCK1]

@gregory38

Here's the context for drawcall 13. If you need 14, ect. let me know ;)

FRAME
    FBP (*32):0x1200
    FBW:16
    PSM:0x2
    FBMSK:0x0

[MrCK1]

New GS Dump: smugglersrun_badtex.zip

This is the raw dump with all the drawcalls, contexts, ect.
smugglersrun_debug_contexts.zip

[gregory38]

Analysis main issue is about a texture channel implementation.
Input: 0x0/0x1180
Output: 0x80/0x1200
Effect: permutation of RG and BA channels

Tricks are

• Input and output overlap
• The 0x80 offset will change page/pixel order

Permutation is easy to implement. But overlap/pixel positions will be tricky.

I'm not sure the game rely on overlapping. So I propose that we implement an hle dedicated shader to implement color channel shuffles. I'm afraid that shadow position will require the pixels position update too. But we will see if at least some shadows appear.

[sunkin351]

HW Shadows are also missing in TLoS 2, just thought I'd mention that here, seeing that both games seem to suffer from the same issue.