Cxbx-reloaded: D3DDevice_CreatePixelShader issue

[PatrickvL]

At least two games suffer from an issue with CreatePixelShader;
Cold Fear - see https://github.com/Cxbx-Reloaded/game-compatibility/issues/434
and Dino Crisis 3 : see https://github.com/Cxbx-Reloaded/game-compatibility/issues/114#issuecomment-343683008

Cold Fear crashes due to expecting the HANDLE returned by CreatePixelShader to be a pointer to valid memory, containing the pixel shader definition. Currently, our implementation of CreatePixelShader returns a handle to the host pixel shader, which is not correct behavior.

Cold Fear crashes while executing the following code (located at 0x0004F0F0 within the .text segment)

   D3DDevice_CreatePixelShader(&pPSDef, &pHandle);
    for ( i = dword_37D198; memcmp((const void *)(pHandle + i), &pPSDef, 0xF0u); dword_37D198 = i )
      i += 4;

[PatrickvL]

According to some feedback, "it's not only Cold Fear and Dino Crisis 3 that have this problem, Panzer Dragoon Orta and Jet Set Radio Future also have this (actually putting the CreatePixelShader = 0 in the .ini of JSRF fix the tag problem and make the game fully playable (no more glitch, other than the character legs, everything else look fine), PDO have the same problem after pushing the start button in the mission, everything become white and impossible to play, modifying this line again, fix this issue."

[Meowson]

When you say to insert "CreatePixelShader = 0" in the .ini of JSRF, where does one find the .ini? Ive tried looking through the files of JSRF but couldn't find it. Thank you.

[PatrickvL]

You'll find it in the "%appdata%/Cxbx-Reloaded/HLECache" folder

[LukeUsher]

The fix for this issue would be to stop patching CreatePixelShader completely, and instead move shader conversion to the SetPixelShader functions instead. (Or better yet, at draw time)

[PatrickvL]

All PixelShader related patches are gone, except for SetPixelShader, which uses a trampoline. Commit https://github.com/Cxbx-Reloaded/Cxbx-Reloaded/commit/4637f4da5eff38afcf4281081ea7c28f8d7179d5 resolves this issue.