Grin: discrepancies between UTXO set on two different nodes
Differences between the UTXO sets on two different nodes.
Both nodes at same height.
Both nodes showing identical numbers of inputs and outputs based on traversing the blocks in the chain.
But the UTXO set on each node is different.
Node1 - 4 spent outputs still showing up in UTXO set.
Node2 - 6 spent outputs still showing up in UTXO set, 10 unspent outputs missing from UTXO set.
Node 1
======
Spent outputs still in UTXO set
-------------------------------
{ 'commit': '08d973a9ac3a792ed0329f926abbbc091d0922ada0e77a2d3c86443182936d41db',
'height': 4092,
'height_spent': 4093,
'output_type': 'Transaction'}
[ { 'commit': '08d973a9ac3a792ed0329f926abbbc091d0922ada0e77a2d3c86443182936d41db',
'height': 4092}]
{ 'commit': '089f09f378755d60bbb51290c3c8d94d36fae3059e5db470534f0513a939bc7a57',
'height': 4069,
'height_spent': 4070,
'output_type': 'Transaction'}
[ { 'commit': '089f09f378755d60bbb51290c3c8d94d36fae3059e5db470534f0513a939bc7a57',
'height': 4069}]
{ 'commit': '082fd6bcfb345f1513c0933fa5948a26d20aea14ebd3d27ff0bce4ff08b636e937',
'height': 4067,
'height_spent': 4068,
'output_type': 'Transaction'}
[ { 'commit': '082fd6bcfb345f1513c0933fa5948a26d20aea14ebd3d27ff0bce4ff08b636e937',
'height': 4067}]
{ 'commit': '08781f222444deb6a672972b543b05bd27f927a2938e08789f744b5034dc24b914',
'height': 4041,
'height_spent': 4042,
'output_type': 'Transaction'}
[ { 'commit': '08781f222444deb6a672972b543b05bd27f927a2938e08789f744b5034dc24b914',
'height': 4041}]
--------------------
Inputs: 2010
Outputs: 6450
*** Duplicate (sets) of inputs: 0
*** Duplicate (sets) of outputs: 0
*** Spent yet still in UTXO set: 4
*** Unspent but missing from UTXO set: 0
--------------------
Blocks where these outputs were created -
4041, 4067, 4069, 4092,
Node 2
======
Spent outputs still in UTXO set
-------------------------------
{ 'commit': '09eb11e2fa1a55a40803ca3249b82cb1ec3b30060505324acf53ba460d2a573497',
'height': 2773,
'height_spent': 2774,
'output_type': 'Transaction'}
[ { 'commit': '09eb11e2fa1a55a40803ca3249b82cb1ec3b30060505324acf53ba460d2a573497',
'height': 2773}]
{ 'commit': '09a7f914e85520128d62fd9a347c7ae5eb4beaa0d097baadae5df62d2019cce590',
'height': 2773,
'height_spent': 2774,
'output_type': 'Transaction'}
[ { 'commit': '09a7f914e85520128d62fd9a347c7ae5eb4beaa0d097baadae5df62d2019cce590',
'height': 2773}]
{ 'commit': '08781f222444deb6a672972b543b05bd27f927a2938e08789f744b5034dc24b914',
'height': 4041,
'height_spent': 4042,
'output_type': 'Transaction'}
[ { 'commit': '08781f222444deb6a672972b543b05bd27f927a2938e08789f744b5034dc24b914',
'height': 4041}]
{ 'commit': '08d973a9ac3a792ed0329f926abbbc091d0922ada0e77a2d3c86443182936d41db',
'height': 4092,
'height_spent': 4093,
'output_type': 'Transaction'}
[ { 'commit': '08d973a9ac3a792ed0329f926abbbc091d0922ada0e77a2d3c86443182936d41db',
'height': 4092}]
{ 'commit': '082fd6bcfb345f1513c0933fa5948a26d20aea14ebd3d27ff0bce4ff08b636e937',
'height': 4067,
'height_spent': 4068,
'output_type': 'Transaction'}
[ { 'commit': '082fd6bcfb345f1513c0933fa5948a26d20aea14ebd3d27ff0bce4ff08b636e937',
'height': 4067}]
{ 'commit': '089f09f378755d60bbb51290c3c8d94d36fae3059e5db470534f0513a939bc7a57',
'height': 4069,
'height_spent': 4070,
'output_type': 'Transaction'}
[ { 'commit': '089f09f378755d60bbb51290c3c8d94d36fae3059e5db470534f0513a939bc7a57',
'height': 4069}]
Unspent outputs missing from the UTXO set
-----------------------------------------
{ 'commit': '085294a0d4602312771a71104d3f8a44d6b908e183de97186256ffa5ffde4f8496',
'height': 4030,
'output_type': 'Transaction'}
None
{ 'commit': '09243b6a0bfd68dde202141f3f1a0224d08aa6002935d67894efcaa62fa73d14b4',
'height': 4049,
'output_type': 'Transaction'}
None
{ 'commit': '087bcfc4015fe8193d8d48686b3ae9faec2d21cda7339428590795f8b002390e42',
'height': 4049,
'output_type': 'Transaction'}
None
{ 'commit': '08b5dd6ed3724e583993b909bca15f68448344432b6d004a017ce034115adcb21b',
'height': 4030,
'output_type': 'Transaction'}
None
{ 'commit': '08ee8630d2ec7d17323a8e883bdb26975c497a630d0f6930790e9e8dc4ef639c51',
'height': 4049,
'output_type': 'Transaction'}
None
{ 'commit': '08b655391d112a115fa41726b787e432b2bb7bcc37498225122ed360699bdc42b9',
'height': 4031,
'output_type': 'Transaction'}
None
{ 'commit': '08327b670d59f7c0370557aeef0f51db39bea90d2b86db79c85566fbb16fd3d64f',
'height': 4030,
'output_type': 'Transaction'}
None
{ 'commit': '09a268c6b494fb2803639b8fcc9bffca65b376800706bbd77488c07daa76e8f3fa',
'height': 4049,
'output_type': 'Transaction'}
None
{ 'commit': '099f2d69b5d21467414643f0ec438c8526373473aa01ec1a75ffde3012f9f7e0ee',
'height': 4049,
'output_type': 'Transaction'}
None
{ 'commit': '092afd52da0df10923e05de9a2c1df52b5882e9af803837470d046a6783a6d2bac',
'height': 4049,
'output_type': 'Transaction'}
None
--------------------
Inputs: 2010
Outputs: 6450
*** Duplicate (sets) of inputs: 0
*** Duplicate (sets) of outputs: 0
*** Spent yet still in UTXO set: 6
*** Unspent but missing from UTXO set: 10
--------------------
Blocks where these outputs were created -
2773, 4030, 4031, 4041, 4049, 4067, 4069, 4092,
antiochp
All 18 comments
Can you detail how you're getting those spent outputs that are still in the UTXOs?
ignopeverell
on 31 Dec 2017
The script is here - https://github.com/antiochp/grin-recon
It traverses the block chain via the blocks api and tracks input/output commitments for each block.
For every output, if there is corresponding input then we treat it as spent.
We then look for all outputs via the utxo api to see if these outputs are in the utxo set.
So these discrepancies are where we have -
- an output from a block with a corresponding input spending it but an entry still in the utxo set
- an output with no corresponding input, but no entry in the utxo set
So for example we can see 08781f222444deb6a672972b543b05bd27f927a2938e08789f744b5034dc24b914 is an output from block 4041 and an input to the next block 4042 -
curl http://127.0.0.1:13413/v1/blocks/4041
{
"header": {
"hash": "edcb4ebe14f3f779d4ccadd12834d9242fbff7d3b47c40bb31c86bea8d59fadb",
"version": 1,
"height": 4041,
"previous": "522d09df2e55f59549264f75028b82dbabdf5a63ed90cbdea209dd717cb610b3",
"timestamp": "2017-12-28T15:42:40Z",
"utxo_root": "4cca56d6e101b895864215bd79520a11e50decb388aeb74f7663d478b937e99c",
"range_proof_root": "6f248827d646d58083c4abde0ee0b1b46b489378c4d87d8b9572242881f5e8dc",
"kernel_root": "fc6a90dc27a6dc9f03d190f15b8a2d2f45521e67e86d3cdcad2e51019f004dd2",
"nonce": 6460632661550439187,
"difficulty": 540,
"total_difficulty": 12933190
},
"inputs": [
"090aee3f4f77dcccb35ebc4a0aee26552ce98eb1cf7f25ec347248ba40a09cdf54",
"0897aba9c442af7518ee23c68c22404bd0d7c09c92849e6c8aa77a08407aae38c2",
"08ed755b0466de64bffcc0ebab31cd607ada84e5c524d1c369b1cfe7c2920ac5df"
],
"outputs": [
{
"output_type": "Transaction",
"commit": "0811d3cb1cb4c455fbb22c1f4608753533213fcea78b4119011ec68d5c63751a8a",
"switch_commit_hash": "0525795ac81a8ea6819e89ea3dc44fc2b0f2d693",
"height": 4041,
"lock_height": 0,
"spent": true,
"proof_hash": "3ea474bbca6444e258d5ef7bdee0226b74cd8a081cced7de94a0e7bded2f4a5d"
},
{
"output_type": "Transaction",
"commit": "08045ff0bd4fcc3e1b19a32b283ac3a338f417250bd9ec1c7877b7998b66881183",
"switch_commit_hash": "0611f1f8aff63e560388a52e7023109bb20913b4",
"height": 4041,
"lock_height": 0,
"spent": true,
"proof_hash": "27fda7d074b917a149ed641c97247bdc3934ef3902229ec5feba945bda824f79"
},
{
"output_type": "Transaction",
"commit": "08974bcf0bdcabdb56f10959e35cab6917b0419a7c951398ef67c68fb660f4765f",
"switch_commit_hash": "c241ec4da2688c8c4c8ed5bbf1742bdd26c21e97",
"height": 4041,
"lock_height": 0,
"spent": true,
"proof_hash": "bd1a672fcaada93e994f594307bfd1bd772d4a69144168d1d506edb9652afc5c"
},
{
"output_type": "Transaction",
"commit": "08229464209d9ba73374a194f0f44df24dda29aab62eb4246dc65ca5268bfab7bc",
"switch_commit_hash": "32c03837c9bf250dfca06e6c2c357d0b55490232",
"height": 4041,
"lock_height": 0,
"spent": true,
"proof_hash": "8cc980412cc3bc74e4f00b3c23e80528c8be527f4bcfc992fdb739733e9b291a"
},
{
"output_type": "Coinbase",
"commit": "08d1cd012d454d14f606987ab58ad7d951394ca15aa6127a9dd4274589491d3c01",
"switch_commit_hash": "0364713351a8d44cd818f9d9675a63e7d12e42bf",
"height": 4041,
"lock_height": 5041,
"spent": true,
"proof_hash": "f65c6a14c27ffbbcb9a51c8ddd510e68805b3414c9b1396b911561e1580de149"
},
{
"output_type": "Transaction",
"commit": "089c63852e08c6797bcb5a7734c0f62ebca565a5c74d86d299e24dbfe54e900c3b",
"switch_commit_hash": "0486a10f4d3f1df9891c89a6f7b580cc0140b9b7",
"height": 4041,
"lock_height": 0,
"spent": true,
"proof_hash": "25f78b8529183d6b3345aaa57fca1eff2efeb59db56a542e94743dc52dd8a5d7"
},
{
"output_type": "Transaction",
"commit": "08781f222444deb6a672972b543b05bd27f927a2938e08789f744b5034dc24b914",
"switch_commit_hash": "15fa80cb9815b3c71e6ca0592d92c6012b12d58c",
"height": 4041,
"lock_height": 0,
"spent": true,
"proof_hash": "b341875dae54c73845b9906bfec962e42354dfdaca0c1988965a6c3d97f415f4"
}
],
"kernels": [
{
"features": "",
"fee": 8000000,
"lock_height": 4039,
"excess": "091db539091c120e7eb0feb21a2d3fd890b2d8dd50c7a3716d7fdfae8622f8dd55",
"excess_sig": "30450221008d94b2c7b366a72e0c2ca4e2bb37ae463236dc333e10428a8737fb6c8ae5b137022079ec4476f6196a0dd7a8cbb3802d7531ca70f22120035f97302f47c073c5d29e"
},
{
"features": "",
"fee": 8000000,
"lock_height": 4039,
"excess": "08e11b75bc9ac1ca8510bd7c9cb37ac7177a2b418c2838ad9414368f2815aaf334",
"excess_sig": "30440220580d16bc1a154f89e1efccbf8863b4fa4819b72c5f40992792ad95f4609cd11d02206f6a9287b550c739f8cb295c835ffe33d43f272f10cb899120c4d1df8cc67544"
},
{
"features": "COINBASE_KERNEL",
"fee": 0,
"lock_height": 0,
"excess": "0930265090f5242e025471e1827322bfef4d73a803bbb205c7fc8fb1fdcb98d7e9",
"excess_sig": "30440220677e086a18269df726e3ee092869130b3e561a4aa659bc114ce72533b72f35480220193323c92464ecf332d86759501b94ee6593df00d8028c97c7636b65657d7aad"
},
{
"features": "",
"fee": 8000000,
"lock_height": 4039,
"excess": "08db6ef75dee909d8b4cd5c07a34a7634dfe23aa2ea7f970e3af58062b446ab361",
"excess_sig": "304402203aef760d7906aa5448c836f9005b14c99f6db015ef64bbf6ddb4a802f69dd1ba02207e303faa1ce9c0d54cd77ff822fe1fcad1003b1cb0c6099f4a80d8cdb9e1c7b3"
}
]
}
------------------------------------------------------------
curl http://127.0.0.1:13413/v1/blocks/4042
{
"header": {
"hash": "20eeb00bb8ee8010fc9ade307270cf272d73400487a6c104e8d038aa981924ae",
"version": 1,
"height": 4042,
"previous": "edcb4ebe14f3f779d4ccadd12834d9242fbff7d3b47c40bb31c86bea8d59fadb",
"timestamp": "2017-12-28T15:44:46Z",
"utxo_root": "c167a975305a5ef7928f6e144ee1a50a70b1c8160042f07ef4ab29640afd907f",
"range_proof_root": "c70c745ae2ce81460fb6cab55436418a91fd45e4719eecbe23d2439864eceddd",
"kernel_root": "d018328db62e99ca5cf70bfe40a14d36ba671a6b3cb2d9cb8c81ca4d76014a31",
"nonce": 13328075940776718580,
"difficulty": 544,
"total_difficulty": 12933876
},
"inputs": [
"08974bcf0bdcabdb56f10959e35cab6917b0419a7c951398ef67c68fb660f4765f",
"08a94fd0bddec0d713f24ae02d82be0ef8f4ed7e8ab65d0096317adedb1a5966ba",
"08781f222444deb6a672972b543b05bd27f927a2938e08789f744b5034dc24b914",
"099d4178e99d37e92282e35ba6cd3c95b0c6cbdbf7fddf5469604051dcc4829e0a",
"08a8c9f6dc3666cd486fa0011da40d94f2e6652d6a32e7121b46087d54b1fd5425",
"089c63852e08c6797bcb5a7734c0f62ebca565a5c74d86d299e24dbfe54e900c3b"
],
"outputs": [
{
"output_type": "Transaction",
"commit": "08c4d5f74b74ab29db67bb6d660f015ee57f0f509eef8aa3932ac716ab6e34ad82",
"switch_commit_hash": "1a182bdcfd345cd94be1c3141327d94d295c0de8",
"height": 4042,
"lock_height": 0,
"spent": true,
"proof_hash": "c1947aaa2613694af2056ce5930c4caf3ed38e2d29e68ae2ac0aad7e370c74a5"
},
{
"output_type": "Transaction",
"commit": "08bae25fb3e58624054d9560c61e40124f6e5a5aee0b0e407233cd11e4a150861d",
"switch_commit_hash": "94bd27947af66b48492147deea237bcd8808688f",
"height": 4042,
"lock_height": 0,
"spent": true,
"proof_hash": "cf62fb4ef0cbd2ba86066262f7fffd360e6d2092ab3b9557fec5c297d31276db"
},
{
"output_type": "Transaction",
"commit": "09ed25980a656639ba837655bf4efb04a7f0fadcfccbeb26ec2fbd7d28affccab6",
"switch_commit_hash": "d9b64c03b6c4a1fff90b2cd1c2aab26d7de021b9",
"height": 4042,
"lock_height": 0,
"spent": true,
"proof_hash": "daf848cf6ae851c9f864d3f60aaa049181db46f7e5ebb3b02eb03abf4b105054"
},
{
"output_type": "Coinbase",
"commit": "08d239587f18c3606417cc2a214746f6baaf6f95d1524592079668ae363c54056c",
"switch_commit_hash": "0cab89d3e9f29aacd71bfccd3bb7339e8075a5b3",
"height": 4042,
"lock_height": 5042,
"spent": true,
"proof_hash": "abd76ccf8986b9d36c6a67947b79ea22611dfa5b56d1f7d6e09aae70f4d2cf49"
},
{
"output_type": "Transaction",
"commit": "08b1a6adfafc17e1aba3605fee776d1b84a3f57148fb20fe444eace49254855ac9",
"switch_commit_hash": "7a45d4fffb59b210140c7e1abecbe0a70321224e",
"height": 4042,
"lock_height": 0,
"spent": true,
"proof_hash": "739ac80294932eb80086c6f2715f84597426b2627c6c5f7bbe53a2d806ec6800"
},
{
"output_type": "Transaction",
"commit": "09d2ccdb05280570337a2633b5edef71099210830fe4b00fa63b0f2955afbc9bda",
"switch_commit_hash": "42d2c1e72398aeaeec81443c47f988f2e4c8baf3",
"height": 4042,
"lock_height": 0,
"spent": true,
"proof_hash": "40d1d0d31f20ae1c504816e5358f2267a0c7c1bfdeaa51e840e2736022d1c69c"
},
{
"output_type": "Transaction",
"commit": "0828bfafe8a8c3b6ca248f1c8307be6278f7b08b70d76c84cc032f579712bf6589",
"switch_commit_hash": "fe676cd9691b88f868db923bf042f249767bd70a",
"height": 4042,
"lock_height": 0,
"spent": true,
"proof_hash": "736def670dd48b62e5cabb58be4c4260bb362db37928c0e6d25a8cd556d53d62"
},
{
"output_type": "Transaction",
"commit": "086118f9d2348985648cfe33112a1e800951dbff42c3d80c2868745745509c7bb9",
"switch_commit_hash": "d06192a44c7d877c40422581ffd0a9d0ce436065",
"height": 4042,
"lock_height": 0,
"spent": true,
"proof_hash": "563bafca0fa8ad51598433d93d0265d07ded22a3a4968d52de2d3800dfc8876e"
},
{
"output_type": "Transaction",
"commit": "083fd38d4c95ef6199c5f7ca3ec49b6e28c7e9c2483fe717c20cb18f3089bbd6da",
"switch_commit_hash": "2bbecf0138c1c00d360fb66aabe9a87e2bfd8883",
"height": 4042,
"lock_height": 0,
"spent": true,
"proof_hash": "0fe3f74775bedc1ce4205b219f2d2575bef73d80ccd34156f2594e42acc650e9"
},
{
"output_type": "Transaction",
"commit": "09da32ca0664083a31b4840cfa2bc4303e852f2b408983b5d8ba1da25b9c9da6e9",
"switch_commit_hash": "9f85336ad71e2578e7366f3e954232a850368cd4",
"height": 4042,
"lock_height": 0,
"spent": true,
"proof_hash": "4dfa15cdfcf66171d544fa41f813813c3637023a1c383e67ed7d0fc509b5f04f"
},
{
"output_type": "Transaction",
"commit": "08bee227b967a2725f5eb4cb9c206d25b1ba4d1178fd0828bcf99565614df6cb23",
"switch_commit_hash": "0495449875a8af172184ab95dc8408757bafdeb6",
"height": 4042,
"lock_height": 0,
"spent": true,
"proof_hash": "9dad2d7616b18297cf7197fea0483a7d21f34cefdec92ec5c9dc43d9c0e8722c"
},
{
"output_type": "Transaction",
"commit": "0813981bad1e849d5371ce5a6cb8178a28b9979064c314cfcd0303291f1f3cbb7f",
"switch_commit_hash": "1703523ff45d1c8bbeec6b45c434ad9a1525ebe1",
"height": 4042,
"lock_height": 0,
"spent": true,
"proof_hash": "4b0a4f8bf0ebfbc8a3d8019e01deab428e99e1f694cbc7e99470fd3867d65d82"
},
{
"output_type": "Transaction",
"commit": "0872e351b72aa61682aef71e31e6e48692ca489cb8e76e66c1ad64ecb593b542ac",
"switch_commit_hash": "1831727103bbfad740bf1c0ed3f84b419d93d0d5",
"height": 4042,
"lock_height": 0,
"spent": true,
"proof_hash": "ccbc8b1dcf456c0c61d252b644b23edd21365d7041f8253a4ff0bb190b7ffa07"
}
],
"kernels": [
{
"features": "",
"fee": 8000000,
"lock_height": 4041,
"excess": "08539e24f76543f2b654a06973f6f11e9de99ce1ace4976f1029a6f39519a304c7",
"excess_sig": "3045022100d36766d812fe9f754c345513af31dbf17783a2290a82030f6a111346fba67b91022074e2b21823ac3e416271f6b2ade9a395c2801e26c5f9da5f03b353340719ba9a"
},
{
"features": "",
"fee": 8000000,
"lock_height": 4041,
"excess": "097d38e16a2a6dce892867b58f6529d145383fd8d29ca8f16c900b8671fdd547e4",
"excess_sig": "3045022100f5b2c7ef954bb3c0ac3404a1163a987d8f9d64e7b44f6b34053063e78c743bf1022072cc625db2209bbbb0dba8e906b34b9710a941f264c1f802268b5428b2f100f3"
},
{
"features": "",
"fee": 8000000,
"lock_height": 4041,
"excess": "08f86ed23aa9f28b8b9b66945894b216b95bb352d0f6012e7e41183bcae12555cf",
"excess_sig": "3044022046e85075956d21608311f345a2f6a536c6240e30e52c603f26607810c3a4e9b602207481f0dd76186950a4fb0cf54ab12479a2b43673eaab65e7a8ccb50b5e4a8c78"
},
{
"features": "COINBASE_KERNEL",
"fee": 0,
"lock_height": 0,
"excess": "0965cd7909d9df2bef138eaf34d52d13c1258614dfc63e64ef247cee8629156b00",
"excess_sig": "30450221009cbf768a91d298f6022311dc76589497f15d694c856b4978ff0a5cdd1218805f02207d4f6ba6ebaef7ba895f5247ae76a92ac7079747d50541b7d178b60671ab2e3a"
},
{
"features": "",
"fee": 8000000,
"lock_height": 4040,
"excess": "095a849eed110cb125be0ebd53eb39ac324a6d3de1667402e4122e391463e7bad0",
"excess_sig": "304402204589372210a7a755e2325439e0a398ea9dc603defee81c3479aea92c7b1b8c3302203ec514e38a67ecd7760cf82bcb2392234980e68375205da49d8b2fed64ee1802"
},
{
"features": "",
"fee": 8000000,
"lock_height": 4040,
"excess": "0963aeaee87ef7505ece0ad11736c4593f5bc76e286913a11fbd1b94bfc194e306",
"excess_sig": "3045022100e20092d26f8d760f40de80148d5ca25487b022f7743b963a483a64ed87ae2c70022030550bc387e42e804dc5649e1a42a83902bc7a624dfcfe6562d8049c9e5e11e9"
},
{
"features": "",
"fee": 8000000,
"lock_height": 4041,
"excess": "091ec6c2d7e1c3e0f4ce462f20537caf97ba7f41b4d0c1d50f4d6eb4ab59d66dc9",
"excess_sig": "3044022023f2551266018226a4adc6ac8004b781529b2ccbf860b61c6678c68704a4459402200f9b02932ad0f994ef6b429aaaef940c8af46723fd0a447c053cfc979c3aff24"
}
]
}
But if we hit the utxo api for this commitment we also see it in there as a UTXO, which is incorrect based on blocks 4041 and 4042 above -
curl http://127.0.0.1:13413/v1/chain/utxos/byids\?id\=08781f222444deb6a672972b543b05bd27f927a2938e08789f744b5034dc24b914
[
{
"output_type": "Transaction",
"commit":
[8,120,31,34,36,68,222,182,166,114,151,43,84,59,5,189,39,249,39,162,147,142,8,120,159,116,75,80,52,220,36,185,20],
"switch_commit_hash": null,
"proof": null,
"height": 4041,
"lock_height": 0
}
]
Some circumstantial evidence - I think I saw something similar happen when spending a large number of coinbase outputs (to consolidate them) from a mining node.
The block was then rollbacked and re-orged.
These outputs should have become spendable again - but they disappeared from the wallet (even after restoring the wallet on this node). This feels like the rm_log still maintains these outputs even after rollbacking - we're treating them as spent even when they are not.
Edit:
But rm_log looks consistent between these 2 nodes. So must be a difference between the 2 sets of pos indices in rocksdb? -
ls -al .grin/sumtrees/utxo/ | head
total 8480
drwxr-xr-x 2 root root 12288 Jan 1 20:21 .
drwxr-xr-x 5 root root 4096 Dec 27 21:31 ..
-rw-r--r-- 1 root root 1438385 Jan 1 20:43 pmmr_dat.bin
-rw-r--r-- 1 root root 60864 Jan 1 20:43 pmmr_rm_log.bin
ls -al .grin/sumtrees/utxo | head
total 5216
drwxr-xr-x 6 staff staff 192 Jan 1 15:21 .
drwxr-xr-x 5 staff staff 160 Dec 29 18:12 ..
-rw-r--r-- 1 staff staff 1438385 Jan 1 15:43 pmmr_dat.bin
-rw-r--r-- 1 staff staff 60864 Jan 1 15:43 pmmr_rm_log.bin
Added additional logging here - #567 so we have more visibility into the is_unspent logic via api.
# API call returning empty results for this utxo -
curl http://127.0.0.1:13413/v1/chain/utxos/byids\?id\=09e145a22b24032f6badf02b9e336e920cf142264f20bda3aa476252690ea0123a
[]
# But we do see this in the logs -
Jan 02 14:37:25.940 DEBG chain_store: get_output_pos: Commitment(09e145a22b24032f6badf02b9e336e920cf142264f20bda3aa476252690ea0123a), Ok(21512)
Jan 02 14:37:25.940 DEBG sumtree: is_unspent: Commitment(09e145a22b24032f6badf02b9e336e920cf142264f20bda3aa476252690ea0123a), pos: 21512, match: true
Jan 02 14:37:25.940 DEBG chain: get_unspent: Commitment(09e145a22b24032f6badf02b9e336e920cf142264f20bda3aa476252690ea0123a) found and not spent
The found and not spent in the last line indicates this is a valid UTXO (as expected).
We found the pos and the hash matched when we looked up the entry in the pmmr via that pos.
So -
- We are finding the utxo and it is unspent
- But - the api then makes an additional lookup to get the block_header for the utxo
- And this is what I suspect we are failing on.
let header = self.chain
.get_block_header_by_output_commit(&commit)
.map_err(|_| Error::NotFound)?;
Still investigating.
antiochp
on 2 Jan 2018
Ok I think I know what's going on here -
When we process a block on a fork -
- rewind the sumtrees
- process all necessary blocks on the fork up to and including the new one
- processing these blocks causes the "header_by_output_commit" index to be updated for all outputs on the fork
- we then discard the pmmr changes (if fork does not increase total work)
- but we do not discard the changes to the "header_by_output_commit" index
- any outputs that exist on both the forks will be incorrect at this point
Intuitively - if we discard a sumtree exension, we also need to re-apply all blocks to get back onto the main chain in a consistent state. And I don't believe we are doing this (but may be wrong in my understanding here).
@ignopeverell thoughts on this?
When we save a block we insert into these indices -
- BLOCK_PREFIX
- BLOCK_HEADER_PREFIX
- OUTPUT_COMMIT_PREFIX
- HEADER_BY_OUTPUT_PREFIX
BLOCK_PREFIX and BLOCK_HEADER_PREFIX are fine - these are indexed by the block hash itself (and multiple forks can co-exist in the index).
OUTPUT_COMMIT_PREFIX I believe is fine - the output itself will be identical even across multiple forks.
The HEADER_BY_OUTPUT_PREFIX is not fine - this is specific to a particular fork at any given time and will need to be rewritten after discarding a sumtree extension for a fork.
Additionally (still not sure of the implications here) -
get_block_header_by_output_commit() relies on is_on_current_chain() so will not work on anything but the main chain (the height indices are not updated unless we are on the longest chain).
So if we are processing a block on a fork then we cannot get block_headers by output commits reliably.
antiochp
on 2 Jan 2018
The more I think about this the more I'm wondering if we need to get rid of the get_block_header_by_output_commit index entirely and lookup block_headers on the chain itself.
This is going to be considerably slower but it will be consistent (main chain or fork).
Right now we're making too many assumptions about there only being one chain/fork in existence when reading/writing this index.
Then think about optimizing this once we know it is working.
antiochp
on 2 Jan 2018
Experimenting in #568.
Attempting to do this without the index is prohibitively slow.
So I think we need to alter the "header by output commit" index to allow it to reference multiple blocks (where each block is on a different fork).
This basically boils down to the following requirements -
For a given commitment, for a particular fork/chain -
- is the output spent/unspent (may be different across forks/chains)
- the block height it originated from (may be different across forks/chains)
Processing a block on a fork - we cannot rely on indices being accurate/consistent with fork state.
We have an entire pmmr implementation dedicated to determining spent/unspent state of a given output based on its commitment.
Do we need something similar to maintain block heights, or even block hashes for flexibility (where we can rollback and reapply different forks etc.)?
antiochp
on 2 Jan 2018
To be fair, the PMMR implementation does a bit more than just indexing, but that's beside the point :-) I do think that the less indexes we have, the happier we will be, especially when they're reorg sensitive. So ideally, we'd just get rid of it.
As far as I can tell, the only use of the "header by output commit" index is to validate coinbase maturity. Is that corect? If so, how about reviving #215?
ignopeverell
on 3 Jan 2018
Yeah maybe "dedicated" was too strong... :-)
And yes - I think if we can solve the coinbase maturity then we can safely get rid of the index.
(I'm cringing looking at some of the code and the assumptions I was making around coinbase maturity a couple of months ago... guess this is the only way to learn though...)
One thing that I'm realizing now though looking at #215 -
1) Today we can uniquely identify an output based solely on the commitment (even if an output appears on multiple forks this is ok as the output itself is identical across forks).
2) #215 changes this assumption as we bake the lock height into the switch commit - a version of the output on one fork may actually be different to one on another fork.
A commitment aaaa could refer to -
- output o1 at height h1 on fork f1 , _and_
- output o2 at height h2 on fork f2
And actually (1) is not true in all cases as we do potentially allow outputs/commitments to be reused as long as they are not in the UTXO set at the same time (see #544).
So if we go down the path of #215 to make outputs self-contained (lock height in switch commit) then the commit->output index itself will no longer be sufficient.
We're going to need some way to handle "duplicate" outputs - where they would need to be identified by (commitment, switch_commit_hash) or something similar.
This may be right direction to go - but we are moving the problem around.
Either an output (identified by a commitment) can exist in multiple blocks, _or_ we have multiple outputs (all referenced by the same commitment).
Working on rebasing #215 against master today so we can see where this gets us.
antiochp
on 3 Jan 2018
Thought about that for a bit, and I think that by adding the switch commit in the MMR hash we've gone down a slippery slope. The MMR is meant to track the overall state of the chain, which is mostly its accounting, and offer strong guarantees of summability and unicity. We sort of piggy-back it a little bit for the range proof and kernels but that's mostly for convenience. Anyway I don't think the switch commit fits in there.
I don't recall exactly why the switch commit made it into the MMR hash, maybe @yeastplume would. It doesn't solve malleability anyway because the transaction can still be changed before it makes it into a block, #207 is a better fix for that. So I think we should revert that decision to keep strong guarantees to uniquely identify a commitment on a given fork. I was planning on revisiting this anyway as it also makes more granular pruning and fast sync trickier.
TL; DR: instead of making the identifier more complicated, let's also remove the switch commit from any identifier. Does that make sense?
ignopeverell
on 4 Jan 2018
I think what you are saying makes sense. I need to play around with the code a bit to fully understand though.
I'm not clear on how this "uniquely identify a commitment/output on a particular fork" meshes with the potential for different "versions" of an output across different forks.
antiochp
on 4 Jan 2018
See #576 for a simplification to remove the switch commit from the PMMR hash.
antiochp
on 4 Jan 2018
Switch commit hash in MMR tree was an attempt to solve transaction malleability, which we later determined wasn't proper fix leading to #207 (which hopefully can also be inserted into bullet proofs). And no, it absolutely shouldn't be relied upon for anything crucial, given their primary purpose is to provide a doomsday backup (which they may not even be doing properly, given recent discussions on the mailing list.)
yeastplume
on 4 Jan 2018
Ok - the more I think about this the less I like the idea of trying to put the lock_height in the output.
This breaks the commitment->output identity and I think losing this guarantee of identity introduces a lot of complexity. We no longer know _which_ output we're talking about given a commitment (and we need to go off and look at heights etc on the current chain to determine which one it is).
We lose the invariant that an output is the _same_ output, regardless of the fork/chain it is on, it becomes dependent on the specific fork it exists on.
But - if we keep the commitment->output identity then we need to introduce the possibility of different output->height (aka output->block) mappings as outputs can be produced in different blocks, at different heights across forks.
I'm leaning toward multiple output->height (vs. multiple commitment->output) but we need a way to handle maintaining these height mappings in the presence of forks.
And if heights/lock_heights cannot be maintained on the inputs/outputs themselves (to preserve their commitment identities) then we need to maintain the full chain worth of heights somewhere else.
So going back to an earlier thought - could we treat height/lock_height as "witness" data and maintain it in the witness pmmr (aka the rangeproof one)?
This would let us maintain it outside the outputs themselves, and allow us to rewind and regenerate the height info on a fork currently being processed so we can correctly determine coinbase maturity under various conditions.
@ignopeverell @yeastplume thoughts on this?
Maybe I'm overthinking this though -
the lock_height issue is only an issue for coinbase outputs, so if we can _guarantee_ a coinbase output/commitment will never get reused (even across forks) then we won't ever see the situation where a given commitment maps to potentially multiple outputs.
I just don't immediately see how we can guarantee this unless we kept a record of all forked blocks in addition to the full main chain.
antiochp
on 4 Jan 2018
I'm not sure I follow your first paragraph. How does putting the lock height in the output breaks the commitment->output identity? I'm assuming the lock height doesn't get included in computing the hash.
ignopeverell
on 5 Jan 2018
It opens up the possibility that node A could store one version of an output (say with lock_height 1) and another node B could store a different version of the "same" output (with lock_height 2) in rocksdb for a given commitment.
A call to get_output_by_commit() would not return a consistent view of the same output across nodes.
This is dependent on which fork each node sees first and writes blocks and outputs to the db for.
We see this currently for the block_header_by_commit index as it is not fork aware and the changes proposed just push these issues down to get_output_by_commit which would not be fork aware either.
Does that make sense?
antiochp
on 6 Jan 2018
Very useful chat about this on gitter.
Closing #580.
Continuing to make progress on #215 (but maintaining switch commit hashes in output_pmmr).
tl;dr -
- maintain switch_commit_hash in output_pmmr (in addition to existing commitment)
- input will provide (to spend output) -
- lock_height
- height
- merkle proof of output inclusion in originating block
- preimage of the switch commit hash
- wallet will need to maintain merkle proofs for outputs
- this will allow us to get rid of "header by output commit" _and_ "output by commit" indices
antiochp
on 7 Jan 2018
Closing - resolved (hopefully) in #615.
antiochp
on 17 Jan 2018
Related issues
ignopeverell
路
3Comments
garyyu
路
6Comments
lehnberg
路
5Comments
antiochp
路
6Comments
antiochp
路
3Comments