Orbit-db: Head hash didn't match the contents errors

[aphelionz]

Several users are reporting this warning

console.warn node_modules/orbit-db-store/src/Store.js:258
    "WARNING! Head hash didn't match the contents

While this seems alarming, it's much more likely a bug in OrbitDB than anything else. We will investigate.

[decypher93]

Hello! Have you found the reason of this warning?

[phillmac]

Try the latest version, i.e. v0.21.3. This seems to be fixed.

[RichardLitt]

@aphelionz Should we close this?

[aphelionz]

Affirmative

[chmanie]

We are on version 0.21.3 and are still seeing this (a lot). @thiagodelgado111 was working on a fix, maybe he knows more.

[decypher93]

not sure though that it's the case but do you have only one db per instance?
I think i had this warning because i created few bases per one instance.

[oed]

I've narrowed it down to this:
It happens with entries I receive from another node on the network.
The reason it happens is because of the keys being in the wrong order. Here is the output from the two hashes that don't match:

Object.keys(a)
(8) ["hash", "id", "payload", "next", "v", "clock", "sig", "key"]
Object.keys(b)
(8) ["hash", "id", "payload", "next", "v", "clock", "key", "sig"]

For some reason it seems like the sig and key property in the object received from another node are swapped. The objects are the same otherwise.
If I only load the db locally I don't see this issue.

There might of course be other causes for the issue, but atleast this one is confirmed.

I'm on orbitdb 0.21.4.

[chmanie]

The order of Object.keys is not deterministic. If there's a part of the code that relies on that, we should change that. There are libraries like https://github.com/substack/json-stable-stringify that could be used, if necessary.

Are you able to point me to the code in question, @oed?

[oed]

I found the issue here: https://github.com/orbitdb/orbit-db-store/blob/master/src/Store.js#L268
By simply logging the two hashes and doing ipfs.dag.get on both of them and comparing.
Still not sure where this issue arise. I also don't know which of the entries are incorrect. The already saved one or the one received over network.
Would be interesting to see if you have the exact same issue @chmanie

[ptoner]

I've been seeing this a lot lately too. Always when things are being replicated remotely. At this point it seems like it almost always happens. It also seems like it stops the replicated event from firing for some reason.

[aphelionz]

@ptoner is this happening on a v0.21.3 version or higher? also what version of js-ipfs are you using?

[ptoner]

Good question. Just noticed I've been playing around with the latest stuff.

orbit 0.22.0-rc4
js-ipfs 0.38.0-pre.1

I had upgraded to the new ipfs because there was mention of some pubsub improvements. But that might be messing with it. I'll drop IPFS back to the latest stable one.

[aphelionz]

It's likely they switched the bases around. My guess is that if you do what @oed did you'd see zdpu... hashes trying to match with bafy... hashes

[ptoner]

Seems to both be z ones.

[ptoner]

I've had other issues related to serialization that were unrelated to orbit. This might also be. I'll check the underlying objects and see what the difference is.

[ptoner]

I downgraded to js-ipfs 0.37.1 and it stopped happening.

[aphelionz]

Ok, cool. Closing since we're already planning on supporting future versions of js-ipfs

[oed]

Ehm, why was this closed? We are still experiencing this issue with [email protected]. Don't think this is related to the ipfs version actually.

[aphelionz]

This was happening before because JSON.stringify wasn't deterministic, so I thought this had been fixed. There's likely another culprit not being serialized correctly, or something similar. Retrograding seems to have helped @ptoner which is why I closed it.

Reopened :)