Rpcs3: shaders: Hash collision issue on cache

[prr-123]

Revision: 0870862898f01f3bbb506c4a49165c0e7477e912

Game tested: BLUS30011 - Tony Hawk's Project 8

Renderer: Vulkan + OpenGL

Problem:
The same shader is found more than once in the shader logs. I think that instead of using the already compiled fragment shader, a new one is compiled from the RSX instructions. The check used by the cache lookup might be inaccurate.

Sample:
shaderlog.zip

[kd-11]

Its a hash collision issue actually. The cache tests and detects the code is different, but then when storing the new shader it overwrites the previous slot, making the first shader now non-existent. Its been known for a while now, I'll see what can be done about it. Some titles are hit very hard by this, generating several shaders per second infinitely even when the game is idle.

[Megamouse]

Yeah i sometimes notice shader compilation and in the end its the same shader count as before. A bit annoying when you think some secret game finally went further and it was basically just the same shader as last time

[kd-11]

Retest with https://github.com/RPCS3/rpcs3/pull/3827

[kamer1337]

there are still same duplicated shaders in log

[kd-11]

some duplication is fine to some extent, especially if the cost of determining that a shader will be compiled is the same as compiling it in the first place. As long as it stops compiling its ok. The issue was really bad for some games where you could compile over 10k shaders by just standing in one spot for a few minutes.

[prr-123]

The game mentioned in the issue (BLUS30011 - Project 8) does stop compiling now when standing completely still so the hash collision seems to be fixed.

Like kamer1337 said, there are still many duplicates which cause the shader cache to continue growing until it needs too much RAM to run without swapping memory to the hard drive. Maybe some shaders could be cleared from the cache when they have not been used for a while to prevent this? That's probably another issue completely though.

[kd-11]

Just clear the shader cache if it grows too large, at least for the moment until the situation improved. Removing shaders from the cache because they are unused will bring the same situation as having no cache where playing some sections of a game will make others alot slower as a new cache will need to be written. With current code in the PR, most games are now reduced to a far smaller number of shaders, but you need to delete the old cache. I will bump the cache version number to keep old broken shaders from loading.

[scribam]

@prr-123 Does #4505 improve the situation?

[prr-123]

Yes!
Shader count is greatly reduced from my previous test. From about 15000 to ~1000 after playing for a reasonable amount of time!

Excellent job 👍
I think the issue can now be closed