Rpcs3: Persona 5 (Vulkan): native fp16 has graphical glitches on Vega (RADV and AMDVLK)

[aqxa1]

Description

With a Vega 56 on Linux, native fp16 support exhibits graphical glitches (glitches and inaccurate colours).

Screenshot with latest RPCS3 git (using Mesa RADV git, although the symptoms are the same with AMDVLK):

Screenshot when disabling native fp16 with Vulkan (shader_types_support.allow_float16 = false) (or reverting to a build before PR #5860):

Here's an RSX capture although I couldn't figure how to replay it, so I'm not sure if it's valid.

Bug does not affect an RX 550 (so looks to be Vega specific) and also doesn't affect the OGL renderer for either card.

System information

Gentoo Linux AMD64
RX Vega 56
Mesa RADV Git (LLVM 9)
RPCS3 git - 43f919c04b6f6ce6bad1cf61d9e641147d5f4475

[kd-11]

There is already a config option for this in the yml file. You don't need to edit code.
Does this also happen on windows? Looks like vega has even more hw issues than expected. Need more people with similar issues to come forward so we can find the common denominator. Fp16 can be forcefully disabled on vega in such a situation.

[aqxa1]

Does this also happen on windows?

Not sure, I haven't booted Windows in a while, but I can test it.

[aqxa1]

Bug doesn't appear on Windows, with AMD drivers 19.2.3 and 19.6.3.

Interestingly, applying the mod from here: https://amicitia.github.io/mods/p5 works around the problem as well (maybe removing some effect that uses fp16).

[kd-11]

The engine uses FP16 heavily, the only thing that can cause that issue is incorrect handling of extended values e.g clamp(NaN, x, y) which should return NaN. This must be something with the codegen, so its likely another LLVM bug. The patch removes some post-processing effects that can generate NaN values which the game does handle properly by itself but if the codegen incorrectly handles NaNs and INFs then we have to just disable that path entirely.

[aqxa1]

I submitted a RADV bug report here: https://bugs.freedesktop.org/show_bug.cgi?id=111039

[aqxa1]

Workaround, use the experimental ACO shader compiler (make sure it's a recent compile as a bug was fixed that prevented the game from launching).

I can't say if it introduces other bugs, but it does workaround this particular problem.

[kd-11]

Workaround included in https://github.com/RPCS3/rpcs3/pull/6164

[aqxa1]

It would be nice to keep it enabled with ACO at least (the driver string includes "ACO"), although I guess you don't want to keep too many driver overrides in the codebase.

[Whatcookie]

ACO doesn't support native half floats yet, so it won't make a difference for now.
see: https://vulkan.gpuinfo.org/displayreport.php?id=6274#extensions (We're looking for the VK_KHR_shader_float16_int8 extension)
Needs retesting after they expose half float support.

If it works on windows, then it likely works on AMDGPU-PRO as well, however the device string reported is exactly the same as amdvlk, so sadly we can't filter between these drivers by device string.

I used this PKGBUILD to install AMDGPU-PRO on my system: https://github.com/Tk-Glitch/PKGBUILDS/tree/master/amdgpu-pro-vulkan-only
Obviously you can't use the PKGBUILD itself on gentoo, but you can manually follow the steps in the script to extract it from the .deb and try it for yourself.

[aqxa1]

Ah okay, that makes sense. So it's only working with ACO because it doesn't support the feature yet, and is falling back to the non-native form.

EDIT: @Whatcookie I can confirm that AMDGPU-PRO works, and yeah, shame about the device strings being the same. I assume disabling it at the code level means it can't be overridden from the config file? I suppose I can just keep a local branch, although I might stick with RADV anyway since it seems to work fine without native float16 (and I get to keep Freesync enabled).

I also added a renderdoc trace to the RADV report, although I'm not sure it will help since I can't reproduce the bug when renderdoc is enabled.