Ppsspp: [Android] Texture hashing method changed since v1.5.4 - Custom Textures don't work

[leopard20]

What happens?
Apparently you've made some changes to the hashing method since v1.5.4, which means custom textures don't work anymore and need to be renamed.
For example, these textures are from The 3rd Birthday.
Saved via v1.5.4:

Saved via v1.5.4-760:

They only differ in the last 8 characters.

What hardware and operating system are you running PPSSPP on? GPU might matter if it's a graphical issue.
Android 7.0
CPU Architecture:Qualcomm Kryo - ARMv8a (64 bit)

[hrydgard]

If you compare the hashes to those from a PC/Windows build, which match?

[leopard20]

All of Windows build textures match those of v1.5.4.
Apparently it's platform-specific.

My PC secs:
Windows 8.1 64 bit
CPU: Intel Core i7 4700HQ 64 bit

[TehPlayer14]

So can't we just make a converter if they are equally offset?

[hrydgard]

Absolutely not, we need to fix whatever bug changed the hashes.

I don't see any relevant change in QuickTexHashNEON... wonder what it could be.

@TehPlayer14 any change you or @leopard20 could try a few builds between 1.5.4 and current master to narrow down the range of where it broke?

[AkiraJkr]

I can confirm the latest PPSSPP build and a few more further older ones, texture replacement didn't work for android on my phone, tested with my Birth by Sleep HD ReMix pack.

[hrydgard]

Right, so what we need is to find out the very last build where it did work. Apparently it does work in 1.5.4, so it's just a matter of trying builds in between...

[leopard20]

@hrydgard
Build 209 works but 484 doesn't. I'll see if I can narrow it down any further.

[leopard20]

This is as close as I could get:
Build 211 works but 246 doesn't.
Some builds between these couldn't even run the game so I got fed up!😩

[unknownbrackets]

That range would imply the arm64jit improvements, I guess? But the texture hash shouldn't use jit, so unless those changes messed up arm64 global state or wrote the memory wrong...

https://github.com/hrydgard/ppsspp/compare/v1.5.4-211-gc948a3df0...v1.5.4-246-geb35e8828

-[Unknown]

[hrydgard]

@leopard20 , one more thing to test - go into settings/tools/developer options and change CPU Core to IR Interpreter or Interpreter. Does that make a difference?

Went through that whole diff and there's nothing that should change the hashes..

[leopard20]

Yep! Changing CPU core to IR interpreter makes custom textures work again. (just exit and start the game)

[hrydgard]

Wow, ok, thanks for confirming. @unknownbrackets seems indeed so, but what state could have changed? I can't find any uninitialized variables in the hashing code, and it doesn't use floating point so it's not rounding mode or something...

[leopard20]

One more thing that I noticed is that build 235 sometimes works and sometimes doesn't (JIT).

[hrydgard]

Does it depend on the game whether it works or not or does it seem completely random?

[leopard20]

Apparently the hashes changes randomly every time I restart PPSSPP. Once it works (no matter how many times I restart the game), but once I restart PPSSPP it doesn't!

[leopard20]

Any idea why this happens? (first time custom textures don't work, but works next time)
20180326_150523.mp4.zip

[leopard20]

Ok that's definitely weird. I also noticed a similar behavior in the latest build. Sometimes the textures don't work but if I close the game and start it again (sometimes a couple of times) it works!

[leopard20]

@hrydgard @unknownbrackets
Please hold off on making any changes for now. Apparently the patterns have been random. I'll test more builds before 211 to see when it happened.

[hrydgard]

OK, thanks for your testing work :)

[leopard20]

@hrydgard
Build 34, has the same patterns!
Total shot in the dark, but could it be because of this commit (build 25):

Vulkan: Add facility to auto-generate mipmap levels. Not yet used. Intended for replacement textures and optional quality improvements later.

[hrydgard]

Seems highly unlikely.

[leopard20]

@hrydgard
Build 19 works too.

[hrydgard]

I sure hope it's not https://github.com/hrydgard/ppsspp/commit/253bba28d85696260eef01c9f2b55161dbd5687b (34 itself) ... that would indicate possibly a compiler bug. Can you try 33, and the other ones in between?

[leopard20]

Seems like it could be 34 because build 30 works!

[hrydgard]

Oh no :(

[leopard20]

@hrydgard Ok good news (I hope!). Build 33 doesn't work either!

Apparently it's not good news!

[leopard20]

https://github.com/hrydgard/ppsspp/compare/v1.5.4-30-g3eda31000...v1.5.4-33-gc268e3781

[hrydgard]

Well it's different but also annoying :) Somehow switching to a different STL implementation caused this, and not a bug in our code - or a bug that manifests itself differently :P Might be tricky to figure out..

[leopard20]

One thing I don't get is why is it such a big deal?! (sorry if this question seems stupid! 😉 but I really don't get it)

[hrydgard]

The annoying thing is that it broke when we updated external libraries. If the problem had started with a change in our code, we would know exactly where to start looking, but in this case, it's much harder...

[leopard20]

No I meant why this entire issue is a big deal! Because textures render correctly in the game and for the custom texture issue we can always rename them.

[hrydgard]

Programmers get uneasy when code misbehaves! Also, having to rename textures specifically for android - and only sometimes! - is not a good user experience.

[TehPlayer14]

Perhaps there another way of replacing textures that wouldn't cause so much trouble at cost of computing power

[leopard20]

@hrydgard
By the way, IR Interpretor doesn't work either (it worked because of the same random pattern). But I guess you already figured it out.

[hrydgard]

Oh! Well that's important info. That means it might not have anything to do with the JIT.

[leopard20]

@hrydgard
Apparently your latest commit has already fixed this issue (I didn't bother testing it because you said it won't help this issue). So far it looks good!
But given the highly irregular pattern I'll run the test for another 100 times and report back.

[leopard20]

It certainly fixes The 3rd Birthday.
But for some games the textures are saved with different names.

[leopard20]

This is from Flatout:
v1.5.4:

v1.5.4-773:

[leopard20]

It looks like the only game that's fixed is the 3rd Birthday.

[leopard20]

@hrydgard
Seems like the random pattern also occurs on Windows! This is really weird!
If it was because of Android STL, then this wouldn't happen on Windows right?!

[hrydgard]

Definitely not. This is quite mysterious...

[unknownbrackets]

What hash mode are you using in these inis? Does this happen even without an ini, or only when there is an ini?

Does anything change if you force the size of a particular texture (use its address) to a certain w/h via hashrange?

On Windows, when this happens, are you sure they're not just different textures? Remember that PSP games specify textures using powers of 2. If a texture contains garbage at the end (this is frequent), then it may appear using multiple hashes - PPSSPP can't be sure what area of the texture will really be used.

-[Unknown]

[leopard20]

If by inis you're referring to the configs, I tried deleting my configs but didn't make any difference.

I don't understand what the rest means! Just tell me what to do (in simpler words!) and I'll do it

[unknownbrackets]

When you create replacements, you can optionally create a textures.ini file in that directory and change settings for replacements. Are you using that, or is all of this without any such ini files?

Actually, it looks like the reduceHash_ thing was unintialized in 1.5.4, so it was probably always on in older versions. So the hashes in v1.5.4 were somewhat random or otherwise wrong.

-[Unknown]

[leopard20]

Nope, not using inis.

What's really strange is that The 3rd Birthday works but other games don't.

[unknownbrackets]

Just to confirm, the current situation is:

1. Hashes different for some games still between v1.5.4 and latest git.
2. Comparing latest git to latest git, every time it uses the same hashes. It just uses different hashes than v1.5.4.
3. 3rd Birthday is still using the same hashes as v1.5.4.

Is that correct? Or is 2 in the list above wrong - is it still a different hash every time you restart the latest git?

Can you compare replacement packs created with PPSSPP v1.4.2 (NOT v1.5.4) with latest git? And latest git with latest git?

The issue is, a bug was introduced in v1.5.4. We've fixed it, but that bug caused some randomness. While latest git made that randomness "more random" or at least "differently random", it was still always a bug.

So, if I'm right and that was the bug - then it means replacements created with v1.3.x, v1.4.x, or latest git should all be compatible. However, replacements created with v1.5.x won't always work in latest git (or even on different platforms.)

Just to note - The 3rd Birthday probably just got the "full house" on that randomness and was lucky. So it by accident behaves the same as other versions in v1.5.4 because of this luck.

-[Unknown]

[leopard20]

@unknownbrackets

1. Yes
2. Yes
3. Yes

I also compared v1.4.2 and the latest git. Turns out you were completely right (they are both compatible). It was v1.5.4 that created different hashes.
But to clear one thing up, the randomness was aggravated after v1.5.4-34 (it created different hashes every time you restarted a game) and later fixed by @hrydgard 's commit (a few comments above)

Anyway, I guess you can close this then. (if you think things are good)

[hrydgard]

Alright, yeah. Unfortunate with the bug in 1.5.4, but not much we can do about it now.