Patchwork: Fork on a feed.

Created on 25 Mar 2016  路  50Comments  路  Source: ssbc/patchwork

from here

Which is also causing #321 for me on missing posts.

picture wanderer

Most helpful comment

Wow. Great job, mystery team.

picture pfrazee on 25 Mar 2016
🎉2 😄1

All 50 comments

Did the script I posted resolve it for you?

pfrazee picture pfrazee on 25 Mar 2016

No, I didn't see the script you posted!

wanderer picture wanderer on 25 Mar 2016

I think @du5t must of commented on that thread. If he did, it seems like I miss messages that come after du5t comment.

wanderer picture wanderer on 25 Mar 2016

He did, but it should just show up as a gap in the thread, not as the end of the thread. Can you post a screenshot of what you see at the end of that thread?

pfrazee picture pfrazee on 25 Mar 2016

Here you go @pfraze 1458926246 Do you see my last message on the feed?

wanderer picture wanderer on 25 Mar 2016

Hmm. This is worrisome. I havent received those latest messages from you, and you haven't received my latest either.

Can you run sbot getLatest @hxGxqPrplLjRG2vtjQL87abX4QKqeLgCwQpS730nNwE=.ed25519 and show me the output? That will show my last message.

pfrazee picture pfrazee on 25 Mar 2016

I get

{ "key": "%jPMw5cHTbzrVPhnQ2Yfx9Yul8RAvD24lL7AkDg+OZwk=.sha256", "value": { "previous": "%PVn5WP2eL64WDm2WJ+gTi9/A+goiKifXBg8uXx/wrgA=.sha256", "author": "@hxGxqPrplLjRG2vtjQL87abX4QKqeLgCwQpS730nNwE=.ed25519", "sequence": 3332, "timestamp": 1458606875992, "hash": "sha256", "content": { "type": "post", "text": "His log has forked, somehow, for you. Here's what I get at seq 2888 for @dust:\n\n```js\n{\n \"key\": \"%lFluepOmDxEUcZWlLfz0rHU61xLQYxknAEd6z4un8P8=.sha256\",\n \"value\": {\n \"previous\": \"%VBfEJjeNUlxLuK0eyRzVha3TLu5PPWLwsvGgnmAdPas=.sha256\",\n \"author\": \"@/02iw6SFEPIHl8nMkYSwcCgRWxiG6VP547Wcp1NW8Bo=.ed25519\",\n \"sequence\": 2888,\n \"timestamp\": 1457679971682,\n \"hash\": \"sha256\",\n \"content\": {\n \"type\": \"post\",\n \"text\": \"oh no\\n\\nhttps://medium.com/making-instapaper/bookmarklets-are-dead-d470d4bbb626\\n\\n> The ultimate catch-22 of the new Content Security Policy wording is that it鈥檚 intended to benefit the users, by providing additional security from hypothetical malicious add-ons on websites that enforce a Content Security Policy. In the end the bookmarklet has been relegated obsolete by the change, a casualty of one clause in one section of one web specification, and end-users and developers are the ones who will mourn its demise. The path to hell is paved with good intentions.\\n\\n> I鈥檇 probably try to do more about it, but I鈥檓 too busy rewriting Instapaper鈥檚 bookmarklet into extensions for every major browser.\\n\\nhttps://www.youtube.com/watch?v=n5diMImYIIA\",\n \"root\": \"%VBfEJjeNUlxLuK0eyRzVha3TLu5PPWLwsvGgnmAdPas=.sha256\",\n \"branch\": \"%VBfEJjeNUlxLuK0eyRzVha3TLu5PPWLwsvGgnmAdPas=.sha256\",\n \"channel\": \"javascript\"\n },\n \"signature\": \"Pv+LWJumKE8nIOfsZxgMcg/EcR/tZeJShmiVIGizERuiAMzwzTTjg78r+InmJopJwMogEG7/W3FLTnH/EOzLCg==.sig.ed25519\"\n }\n}\n```\n\nExact same content, but the key is different.", "root": "%OmgwEVkNh4OV4v+z7Mfyt8pRjTHPJchOM+gBpDniYDc=.sha256", "branch": "%mHtDDrXxpS5AEu3TWYaF1nrfe3h3lg5aEowDoTWQCMY=.sha256", "mentions": [ { "link": "@/02iw6SFEPIHl8nMkYSwcCgRWxiG6VP547Wcp1NW8Bo=.ed25519", "name": "dust" }, { "link": "%lFluepOmDxEUcZWlLfz0rHU61xLQYxknAEd6z4un8P8=.sha256" }, { "link": "%VBfEJjeNUlxLuK0eyRzVha3TLu5PPWLwsvGgnmAdPas=.sha256" } ], "channel": "patchwork-dev" }, "signature": "qzrcndNQRVWNwW58MP/gJeWIfUiNv69Mi7Ah9DGc67hbtCTPwYTfXaJ+Xt3vkPj6SGo0fhB9qmZPVhb11k4UBQ==.sig.ed25519" } }

wanderer picture wanderer on 25 Mar 2016

Ok, that's correct. I was concerned you might be getting more incorrect hashes.

@dominictarr is it possible that the replication failure on dust's feed is causing @wanderer's entire gossip flow to halt? Would one feed's failure abort the entire connection?

@wanderer I'll give you the script here, to fix the issue with dust. Then we'll see if you catch up across the board.

https://gist.github.com/pfraze/c679c4b601c78e194302

Stop patchwork/sbot, back up your ssb directory, put this script in your scuttlebot directory, and then run

node delete-msg.js "%vyyVmWprqjKPtcjw9nB1w0pmld+m41rQp/4pvM5yNw4=.sha256"

If everything looks correct, add --do-it. If you see any errors prior to do-it (which you might) copy them here for me.

pfrazee picture pfrazee on 25 Mar 2016

I just read your last comment in that screenshot, and that's an interesting theory. @dominictarr make sure you see that.

pfrazee picture pfrazee on 25 Mar 2016

hmm the script doesn't seem to work. It looks like it deleted Du5t messages, but then when it resynced the message with "key": "%vyyVmWprqjKPtcjw9nB1w0pmld+m41rQp/4pvM5yNw4=.sha256" cames back. And none of dust's message after that sync.

wanderer picture wanderer on 25 Mar 2016

No kidding. That's very interesting. Is it possible you're still using the JS version of chloride?

pfrazee picture pfrazee on 25 Mar 2016

nope but im using node -v v6.0.0-pre. I got chloride to compile successfully with it.

wanderer picture wanderer on 25 Mar 2016

This suggests there's some codepath or condition which causes the hash-miscalculation to happen consistently. If you look at the message that you received after delete, does it look 100% the same as before?

pfrazee picture pfrazee on 25 Mar 2016

yep it looks exactly the same

{
  "key": "%vyyVmWprqjKPtcjw9nB1w0pmld+m41rQp/4pvM5yNw4=.sha256",
  "value": {
    "previous": "%VBfEJjeNUlxLuK0eyRzVha3TLu5PPWLwsvGgnmAdPas=.sha256",
    "author": "@/02iw6SFEPIHl8nMkYSwcCgRWxiG6VP547Wcp1NW8Bo=.ed25519",
    "sequence": 2888,
    "timestamp": 1457679971682,
    "hash": "sha256",
    "content": {
      "type": "post",
      "text": "oh no\n\nhttps://medium.com/making-instapaper/bookmarklets-are-dead-d470d4bbb626\n\n> The ultimate catch-22 of the new Content Security Policy wording is that it鈥檚 intended to benefit the users, by providing additional security from hypothetical malicious add-ons on websites that enforce a Content Security Policy. In the end the bookmarklet has been relegated obsolete by the change, a casualty of one clause in one section of one web specification, and end-users and developers are the ones who will mourn its demise. The path to hell is paved with good intentions.\n\n> I鈥檇 probably try to do more about it, but I鈥檓 too busy rewriting Instapaper鈥檚 bookmarklet into extensions for every major browser.\n\nhttps://www.youtube.com/watch?v=n5diMImYIIA",
      "root": "%VBfEJjeNUlxLuK0eyRzVha3TLu5PPWLwsvGgnmAdPas=.sha256",
      "branch": "%VBfEJjeNUlxLuK0eyRzVha3TLu5PPWLwsvGgnmAdPas=.sha256",
      "channel": "javascript"
    },
    "signature": "Pv+LWJumKE8nIOfsZxgMcg/EcR/tZeJShmiVIGizERuiAMzwzTTjg78r+InmJopJwMogEG7/W3FLTnH/EOzLCg==.sig.ed25519"
  }
}
wanderer picture wanderer on 25 Mar 2016

@pfraze what do you get for message %VBfEJjeNUlxLuK0eyRzVha3TLu5PPWLwsvGgnmAdPas=.sha256 ?

wanderer picture wanderer on 25 Mar 2016 
{
  "previous": "%cNJgO+1R4ci/jgTup4LLACoaKZRtYtsO7BzRCDJh6Gg=.sha256",
  "author": "@/02iw6SFEPIHl8nMkYSwcCgRWxiG6VP547Wcp1NW8Bo=.ed25519",
  "sequence": 2887,
  "timestamp": 1457676449950,
  "hash": "sha256",
  "content": {
    "type": "post",
    "text": "has this [CSP browser garbage from 2013](https://github.com/blog/1477-content-security-policy) been fixed yet?\n\n> Bookmarklets\n\n> As made clear by the CSP spec, browser bookmarklets shouldn't be affected by CSP.\n\n>> Enforcing a CSP policy should not interfere with the operation of user-supplied scripts such as third-party user-agent add-ons and JavaScript bookmarklets.\nhttp://www.w3.org/TR/CSP/#processing-model\n\n>> Whenever the user agent would execute script contained in a javascript URI, instead the user agent must not execute the script. (The user agent should execute script contained in \"bookmarklets\" even when enforcing this restriction.)\nhttp://www.w3.org/TR/CSP/#script-src\n\n> But, none of the browsers get this correct. All cause CSP violations and prevent the bookmarklet from functioning.\n\n> Though its highly discouraged, you can disable CSP in Firefox as a temporary workaround. Open up about:config and set security.csp.enable to false.\n\nlike literally a cdn-centralised script will get to run code over a local bookmarklet.",
    "channel": "javascript"
  },
  "signature": "54rc8soP8jVtGc5p7wq2B0ljuZFaQV3e6qSg+68wPHJDITYKfS0nZL5WIu2yf+4SDogmXxFK28pjsO+sxRREDw==.sig.ed25519"
}
pfrazee picture pfrazee on 25 Mar 2016

ok i get the same thing for 2887. So must just be 2888 that is messing up

wanderer picture wanderer on 25 Mar 2016

Try running this script in your sbot directory, and tell me what the output is: https://gist.github.com/pfraze/0ebacc23a0d5a1c22867

pfrazee picture pfrazee on 25 Mar 2016 
hash variations tester
null_radix's hash: %vyyVmWprqjKPtcjw9nB1w0pmld+m41rQp/4pvM5yNw4=.sha256
expected hash: %lFluepOmDxEUcZWlLfz0rHU61xLQYxknAEd6z4un8P8=.sha256

gave the crazy hash!

wow! crazy!

wanderer picture wanderer on 25 Mar 2016

Mind blown. Can you modify the script to output the JSON, with this?

console.log(defaultEncode(theMessage))

Then reply with the output?

pfrazee picture pfrazee on 25 Mar 2016

Also, what version of ssb-keys do you have?

pfrazee picture pfrazee on 25 Mar 2016

Also, what version of ssb-keys do you have?

"version": "5.0.0"

Then reply with the output?

{
  "previous": "%VBfEJjeNUlxLuK0eyRzVha3TLu5PPWLwsvGgnmAdPas=.sha256",
  "author": "@/02iw6SFEPIHl8nMkYSwcCgRWxiG6VP547Wcp1NW8Bo=.ed25519",
  "sequence": 2888,
  "timestamp": 1457679971682,
  "hash": "sha256",
  "content": {
    "type": "post",
    "text": "oh no\n\nhttps://medium.com/making-instapaper/bookmarklets-are-dead-d470d4bbb626\n\n> The ultimate catch-22 of the new Content Security Policy wording is that it鈥檚 intended to benefit the users, by providing additional security from hypothetical malicious add-ons on websites that enforce a Content Security Policy. In the end the bookmarklet has been relegated obsolete by the change, a casualty of one clause
in one section of one web specification, and end-users and developers are the ones who will mourn its demise. The path to hell is paved with good intentions.\n\n> I鈥檇 probably try to do more about it, but I鈥檓 too busy rewriting Instapaper鈥檚 bookmarklet into extensions for every major browser.\n\nhttps://www.youtube.com/watch?v=n5diMImYIIA",
    "root": "%VBfEJjeNUlxLuK0eyRzVha3TLu5PPWLwsvGgnmAdPas=.sha256",
    "branch": "%VBfEJjeNUlxLuK0eyRzVha3TLu5PPWLwsvGgnmAdPas=.sha256",
    "channel": "javascript"
  },
  "signature": "Pv+LWJumKE8nIOfsZxgMcg/EcR/tZeJShmiVIGizERuiAMzwzTTjg78r+InmJopJwMogEG7/W3FLTnH/EOzLCg==.sig.ed25519"
}
hash variations tester
null_radix's hash: %vyyVmWprqjKPtcjw9nB1w0pmld+m41rQp/4pvM5yNw4=.sha256
expected hash: %lFluepOmDxEUcZWlLfz0rHU61xLQYxknAEd6z4un8P8=.sha256

gave the crazy hash!
wanderer picture wanderer on 25 Mar 2016

The mystery is deep here. That's the same JSON output I get. @dominictarr I gotta pass the ball to you.

pfrazee picture pfrazee on 25 Mar 2016

I also get the same result when i replace ssb-keys whit

var crypto = require('crypto')
var hash = function (data, enc) {
  crypto.createHash('sha256').update(data, enc).digest('base64')
}
wanderer picture wanderer on 25 Mar 2016

is it possible you're getting different newlines in your JSON output? \r\n vs \n, something like that?

pfrazee picture pfrazee on 25 Mar 2016

won't matter here

wanderer picture wanderer on 25 Mar 2016

It should, shouldnt it? We're hashing the JSON output, so if there's different newlines then the hash would differ

pfrazee picture pfrazee on 25 Mar 2016

@pfraze what do you get if you replace hash with

var crypto = require('crypto')
var hash = function (data, enc) {
  return crypto.createHash('sha256').update(new Buffer(data), 'binary').digest('base64')
}

edit: return!

wanderer picture wanderer on 25 Mar 2016

That gave me your hash.

pfrazee picture pfrazee on 25 Mar 2016

I found the difference. Your example above gives the "crazy" hash. This gives the expected hash:

var hash = function (data) {
  return crypto.createHash('sha256').update(data).digest('base64')+'.sha256'
}

It changes if data is a buffer, or a string.

pfrazee picture pfrazee on 25 Mar 2016

got it! you need to set the encoding to binary and I get the right hash :)

wanderer picture wanderer on 25 Mar 2016

You mentioned you're on node 6. I wonder if there's a behavior that got changed, there, that relates to this.

pfrazee picture pfrazee on 25 Mar 2016

yeah probably

wanderer picture wanderer on 25 Mar 2016

Wow. Great job, mystery team.

pfrazee picture pfrazee on 25 Mar 2016
🎉2 😄1

holy shit, so what changed in node 6?

dominictarr picture dominictarr on 26 Mar 2016

hang on, how do you get node@6? only see v5.9 in https://github.com/nodejs/node we should post an issue there asap. the same arguments should produce the same hash in any version change.

maybe it's this?
https://github.com/nodejs/node/commit/b010c8716498dca398e61c388859fea92296feb3

dominictarr picture dominictarr on 26 Mar 2016

Okay so this is actually a quite a big problem, especially going forward.
Basically, it looks like hash.update was interpreting strings incorrectly!
and it's changed, but that means that we have lots of old messages that are actually wrong!

I made a quick script to count how many messages have the wrong hashes:

{
  "@xR8CGZw3tKBZW8834EMiXd+U99i76jQdsChn7Z7y1bU=.ed25519": 3,
  "@p13zSAiOpguI9nsawkGijsnMfWmFd5rlUNpzekEE+vI=.ed25519": 18,
  "@f/6sQ6d2CMxRUhLpspgGIulDxDCwYD7DzFzPNr7u5AU=.ed25519": 12,
  "@vt8uK0++cpFioCCBeB3p3jdx4RIdQYJOL/imN1Hv0Wk=.ed25519": 4,
  "@XbgAjnHAIESiABEGuJOAojN4zkZcPaF34/YRbpOHyiI=.ed25519": 4,
  "@BIbVppzlrNiRJogxDYz3glUS7G4s4D4NiXiPEAEzxdE=.ed25519": 3,
  "@dnr1swLSAgf36g+FzGjNLgmytj2IIyDaYeKZ7F5GdzY=.ed25519": 5,
  "@/RM1Id8j05uitIt6iwMpiivnCqHcbcC1IHyi5FrvLLQ=.ed25519": 2,
  "@hxGxqPrplLjRG2vtjQL87abX4QKqeLgCwQpS730nNwE=.ed25519": 17,
  "@Tc/1cgullpQYaQk/JKNwDAsfniVADcU9lulwYXrGsos=.ed25519": 2,
  "@0vEsc7TAnwrHEYRQodQzA0TGMBxGDIh2zUW0IzxjAzE=.ed25519": 2,
  "@2DmoKgZU8XUQnhwp47txg8BUonXxU6gtkT1H+BM3ias=.ed25519": 4,
  "@eO4/h8L6WQi4grlB3BfhfQdv/fTFGHzqyG0qCPtLaJQ=.ed25519": 2,
  "@/02iw6SFEPIHl8nMkYSwcCgRWxiG6VP547Wcp1NW8Bo=.ed25519": 57,
  "@xZtIjBbXhioJEqsq3cRonwdtgil144i1Vv9CzL8GsJo=.ed25519": 2,
  "@yBk5GQVZDmV5Q5H3uhkusiyJDVApGQQ6hSlBjZMR65Y=.ed25519": 2,
  "@dx86zKQsDjLKsuO8QXfgJCd5j2upHUildv+lFqgmzAc=.ed25519": 2,
  "@x4rsayTzoXVlxcSeeQ49A6PzSQa41MRySB33z1LjZr0=.ed25519": 7,
  "@ye+QM09iPcDJD6YvQYjoQc7sLF/IFhmNbEqgdzQo3lQ=.ed25519": 5,
  "@xfOofVMkuo/IV95TssFeYQ9Iuh69TbszryOMARERUpo=.ed25519": 2,
  "@MCKev/sFgaFwgQHIdXMIZvwCu+pnNA485Nisgr2lUpk=.ed25519": 2,
  "@aZle6I74lhH2ztmxiBL6rICbAL6Z+ahVu5S9fM3Ezxw=.ed25519": 2,
  "@uikkwUQU4dcd/ZrHU7JstnkTgncxQB2A8PDLHV9wDAs=.ed25519": 9,
  "@GLH9VPzvvU2KcnnUu2n5oxOqaTUtzw+Rk6fd/Kb9Si0=.ed25519": 2,
  "@U1+BrfWhx8x3RSCJuRDAaKCnPYsQdQRyGMKncWl1Y1o=.ed25519": 3,
  "@OweKXqiKl49z9pzp764z0xvzOfHqTacYxuLubLzrCgQ=.ed25519": 143,
  "@KFTtT5upVH1s/cQQ1lFC100E6x0HfV792BN04z+HSXg=.ed25519": 12,
  "@X9K+gcwfWZAii2LqfGvVpicoStZXMd4XMMtoyFvkVO8=.ed25519": 78,
  "@zZsc8cvgfZzE85QkwPbf9Zio4cHVWL2R/zAD1zVgh34=.ed25519": 147,
  "@iL6NzQoOLFP18pCpprkbY80DMtiG4JFFtVSVUaoGsOQ=.ed25519": 3,
  "@lzMHSo63Do77Fg4qdDrcIk6yJ+HWQs6YH+fD217S6+s=.ed25519": 8,
  "@6ilZq3kN0F+dXFHAPjAwMm87JEb/VdB+LC9eIMW3sa0=.ed25519": 3,
  "@YXkE3TikkY4GFMX3lzXUllRkNTbj5E+604AkaO1xbz8=.ed25519": 3,
  "@EMovhfIrFk4NihAKnRNhrfRaqIhBv1Wj8pTxJNgvCCY=.ed25519": 4,
  "@hGd3sjghbrkVaFEMUrXV11m2oyKLfLqJQcadvapH02w=.ed25519": 3,
  "@6VZZAQiR8pScq9M6ExRoKzXOmRF+8aV8v7xIDfFrhDE=.ed25519": 2
}
23086 543 0.02298023615049304

bottom 2 numbers are good hashes, bad hashes, proportion of bad / total.
so messages are 2% rotten.

dominictarr picture dominictarr on 26 Mar 2016

oh, and above, is the count of messages which are bad, by author.
The quick fix, is for ssb-keys to create a buffer, with new Buffer(string, 'binary')
but that means that future implementations will need to replicate this bug,
so really, we need to update this some how - with only the legacy messages being treated like this!

dominictarr picture dominictarr on 26 Mar 2016

also, TODO: check whether this affects blobs...

dominictarr picture dominictarr on 26 Mar 2016

hopefully not, because they are generally handled as buffers.

dominictarr picture dominictarr on 26 Mar 2016

earlier there was some conversation about handling schematic updates ('migrations' and other such)...i would love to understand how that works. is this going to be the issue for working that out, or another one?

du5t picture du5t on 26 Mar 2016

There are 36 feeds that have messages with this problem.

{
  "@xR8CGZw3tKBZW8834EMiXd+U99i76jQdsChn7Z7y1bU=.ed25519": 3,
  "@p13zSAiOpguI9nsawkGijsnMfWmFd5rlUNpzekEE+vI=.ed25519": 18,
  "@f/6sQ6d2CMxRUhLpspgGIulDxDCwYD7DzFzPNr7u5AU=.ed25519": 12,
  "@vt8uK0++cpFioCCBeB3p3jdx4RIdQYJOL/imN1Hv0Wk=.ed25519": 4,
  "@XbgAjnHAIESiABEGuJOAojN4zkZcPaF34/YRbpOHyiI=.ed25519": 4,
  "@BIbVppzlrNiRJogxDYz3glUS7G4s4D4NiXiPEAEzxdE=.ed25519": 3,
  "@dnr1swLSAgf36g+FzGjNLgmytj2IIyDaYeKZ7F5GdzY=.ed25519": 5,
  "@/RM1Id8j05uitIt6iwMpiivnCqHcbcC1IHyi5FrvLLQ=.ed25519": 2,
  "@hxGxqPrplLjRG2vtjQL87abX4QKqeLgCwQpS730nNwE=.ed25519": 17,
  "@Tc/1cgullpQYaQk/JKNwDAsfniVADcU9lulwYXrGsos=.ed25519": 2,
  "@0vEsc7TAnwrHEYRQodQzA0TGMBxGDIh2zUW0IzxjAzE=.ed25519": 2,
  "@2DmoKgZU8XUQnhwp47txg8BUonXxU6gtkT1H+BM3ias=.ed25519": 4,
  "@eO4/h8L6WQi4grlB3BfhfQdv/fTFGHzqyG0qCPtLaJQ=.ed25519": 2,
  "@/02iw6SFEPIHl8nMkYSwcCgRWxiG6VP547Wcp1NW8Bo=.ed25519": 57,
  "@xZtIjBbXhioJEqsq3cRonwdtgil144i1Vv9CzL8GsJo=.ed25519": 2,
  "@yBk5GQVZDmV5Q5H3uhkusiyJDVApGQQ6hSlBjZMR65Y=.ed25519": 2,
  "@dx86zKQsDjLKsuO8QXfgJCd5j2upHUildv+lFqgmzAc=.ed25519": 2,
  "@x4rsayTzoXVlxcSeeQ49A6PzSQa41MRySB33z1LjZr0=.ed25519": 7,
  "@ye+QM09iPcDJD6YvQYjoQc7sLF/IFhmNbEqgdzQo3lQ=.ed25519": 5,
  "@xfOofVMkuo/IV95TssFeYQ9Iuh69TbszryOMARERUpo=.ed25519": 2,
  "@MCKev/sFgaFwgQHIdXMIZvwCu+pnNA485Nisgr2lUpk=.ed25519": 2,
  "@aZle6I74lhH2ztmxiBL6rICbAL6Z+ahVu5S9fM3Ezxw=.ed25519": 2,
  "@uikkwUQU4dcd/ZrHU7JstnkTgncxQB2A8PDLHV9wDAs=.ed25519": 9,
  "@GLH9VPzvvU2KcnnUu2n5oxOqaTUtzw+Rk6fd/Kb9Si0=.ed25519": 2,
  "@U1+BrfWhx8x3RSCJuRDAaKCnPYsQdQRyGMKncWl1Y1o=.ed25519": 3,
  "@OweKXqiKl49z9pzp764z0xvzOfHqTacYxuLubLzrCgQ=.ed25519": 143,
  "@KFTtT5upVH1s/cQQ1lFC100E6x0HfV792BN04z+HSXg=.ed25519": 12,
  "@X9K+gcwfWZAii2LqfGvVpicoStZXMd4XMMtoyFvkVO8=.ed25519": 78,
  "@zZsc8cvgfZzE85QkwPbf9Zio4cHVWL2R/zAD1zVgh34=.ed25519": 147,
  "@iL6NzQoOLFP18pCpprkbY80DMtiG4JFFtVSVUaoGsOQ=.ed25519": 3,
  "@lzMHSo63Do77Fg4qdDrcIk6yJ+HWQs6YH+fD217S6+s=.ed25519": 8,
  "@6ilZq3kN0F+dXFHAPjAwMm87JEb/VdB+LC9eIMW3sa0=.ed25519": 3,
  "@YXkE3TikkY4GFMX3lzXUllRkNTbj5E+604AkaO1xbz8=.ed25519": 3,
  "@EMovhfIrFk4NihAKnRNhrfRaqIhBv1Wj8pTxJNgvCCY=.ed25519": 4,
  "@hGd3sjghbrkVaFEMUrXV11m2oyKLfLqJQcadvapH02w=.ed25519": 3,
  "@6VZZAQiR8pScq9M6ExRoKzXOmRF+8aV8v7xIDfFrhDE=.ed25519": 2
}

@du5t yeah, we have talked about this before, but havn't had to actually do it.
about a year ago we rewrote all the crypto, but decided that it was easier to just make everyone start over, but now there is a lot more content and I feel it would be a better challenge to move forwards, backwards compatibly.

That said, we can easily _treat_ this problem, by just preserving the current schema hashing objects as binary strings new Buffer(JSON.stringify(msg, null, 2), 'binary')) That is ugly, but works for now.

This one is extra hard, because it's right in the guts of the system. the hashes or encoding are the hardest thing to change.

dominictarr picture dominictarr on 26 Mar 2016

We have the algorithm identifier on all references to handle protocol upgrades, but it wasnt designed to handle bugs in a protocol. We can use it if we have to, but it would be something like .sha256-2, or perhaps include the correct encoding: .sha256-utf8.

pfrazee picture pfrazee on 26 Mar 2016

@dominictarr is that a listing of messages that reference incorrect hashes in their content?

pfrazee picture pfrazee on 26 Mar 2016

That said, we can easily treat this problem, by just preserving the current schema hashing objects as binary strings new Buffer(JSON.stringify(msg, null, 2), 'binary')) That is ugly, but works for now.

how about a migration tool that updates all the bad hashes?

wanderer picture wanderer on 26 Mar 2016

Not possible. Because each user must sign their messages, and messages are referenced by hash, a rebuild would require each user in the network to republish their feeds in concert with each other.

pfrazee picture pfrazee on 26 Mar 2016

yeah maybe .sha2-utf8 but also maybe we should just consider the reference an (approximately) opaque identifier, because turns out it also includes encoding (implicitly) and other things, like JSON.stringify(msg, null, 2). now we did put a "hash": "sha256" property on each message, which is great, because then people upgrading can update that too, and then implementations will know to use the new hash... and maybe we have something like, implementation just rejects messages with legacy encoding after some "flag day", so eventually future clients don't have to worry about old encodings. (because millions of people use the recent one, but hundreds used the old one)

dominictarr picture dominictarr on 27 Mar 2016

Can you go into more detail on your alternative idea to sha2-utf8? Im not following all the pieces.

It is true there's a lot of implicit information to the encoding. Perhaps we should use a madeup identifier other than the hashing function, like .hash1. Then in the future, if something like this happened, we'd just make the update and use .hash2.

pfrazee picture pfrazee on 27 Mar 2016

I think you got it. it would sha256-utf8 (or something) and then hash would be set to the same thing, so that readers know that message is meant to be interpreted with that encoding as the primary algorithm, and also, that they should _refer_ to that message with via that encoding. We'd just have the peers reject messages they don't know (but probably let the user know, somehow)

This way it should be pretty easy to introduce a new hash or encoding.

dominictarr picture dominictarr on 27 Mar 2016

Looks like this was sorted out. Closing!

mmckegg picture mmckegg on 30 Oct 2016
Was this page helpful?
0 / 5 - 0 ratings

Related issues

mmckegg picture mmckegg  路  4Comments
skippednote picture skippednote  路  6Comments
Gaffen picture Gaffen  路  4Comments
geovanisouza92 picture geovanisouza92  路  3Comments
molstrangler picture molstrangler  路  7Comments