Steem: [0.14.0-shared-db] Stuck at block 3705110-3705111

Created on 31 Oct 2016  路  11Comments  路  Source: steemit/steem

From cli_wallet: "head_block_num": 3705110

From the node:

914760ms th_a       fork_database.cpp:43          push_block           ] Pushing block to fork database that failed to link: 00388918d33ec4d70c779d2b59efc9d3cd6e024d, 3705112
914760ms th_a       fork_database.cpp:44          push_block           ] Head: 3705111, 00388917fe8c0184c499e115d9ee3bf0cf6825bb
914761ms th_a       application.cpp:481           handle_block         ] Error when pushing block:
4080000 unlinkable_block_exception: unlinkable block
block does not link to known chain
    {}
    th_a  fork_database.cpp:64 _push_block

    {"new_block":{"previous":"00388917fe8c0184c499e115d9ee3bf0cf6825bb","timestamp":"2016-08-01T14:56:03","witness":"joseph","transaction_merkle_root":"7aa84a094d40da3fbad391725ccc9e1a6f10d8d9","extensions":[],"witness_signature":"20648fd6d05f28208f2db6358a42b20d80d67743f43fb14023f0d8ad096c7be3217be2b7b37138d5c18c347e1bd63a97af1e65bed33617d3a77d794efeb20dc2f7","transactions":[{"ref_block_num":35062,"ref_block_prefix":3135183822,"expiration":"2016-08-01T14:56:12","operations":[["vote",{"voter":"prufarchy","author":"helikopterben","permlink":"peak-inside-a-usd50-000-trading-portfolio-and-see-near-real-time-thoughts-and-actions-as-trades-occur-over-time-08-01-16","weight":10000}]],"extensions":[],"signatures":["2021d887cbbce7cafa83559d3d1d843b357908c8f11b818bbf91f49b45ad786de07d5d04835147370dd7fb933625202c50ba4e3ae6b74210e9aed6897fef6b748f"]},{"ref_block_num":35022,"ref_block_prefix":967163360,"expiration":"2016-08-01T14:56:15","operations":[["vote",{"voter":"stranger27","author":"sokoloffa","permlink":"can-you-solve-this-task","weight":10000}]],"extensions":[],"signatures":["1f312d2e35a3843857e7ddd7423f5a9b87c4ec87070de6121ba1db29a2924061112f7c5bb78cb9b116f2c281ea1fbe655ab2e5bda45d1c4524b82be9af0e20c501"]},{"ref_block_num":35085,"ref_block_prefix":4182913395,"expiration":"2016-08-01T14:56:09","operations":[["vote",{"voter":"digi3d","author":"tokyodude","permlink":"alert-of-massive-tokyo-quake-causes-brief-panic-today","weight":10000}]],"extensions":[],"signatures":["20134f9636b64bab3be2cbeb2b403e6d5c627eb1cac41c30ec2d6eaa65368003102eb9d70f48b124865e6aefd46e7b362b126df4f0cc5de400cfd2fd0bf96488db"]},{"ref_block_num":35084,"ref_block_prefix":4238056159,"expiration":"2016-08-01T14:56:12","operations":[["vote",{"voter":"grosu20","author":"athenadruid","permlink":"arianrhod","weight":10000}]],"extensions":[],"signatures":["20478d92d8ba6c2e9891f4fec9397547fe078ed81eecbe4c1bcbc4b9f31379d0f30e02d5fa77abfa864115caebbc33f470a0ab07a4cca2f5abae49c9e28866273f"]},{"ref_block_num":35067,"ref_block_prefix":2222533022,"expiration":"2016-08-01T14:56:15","operations":[["vote",{"voter":"kalimor","author":"domino","permlink":"re-angielb-my-first-minimalist-art-20160720t194251352z","weight":10000}]],"extensions":[],"signatures":["2021daf244ef902c087618d6f1e2e8fd54ff0972fc2d65684c9bd0809bb19f856220e1b8b153e7e3f8862db3d0416dbc52202352cc3fde43de0fb30274b3f6ef01"]},{"ref_block_num":35046,"ref_block_prefix":1043799504,"expiration":"2016-08-01T14:56:15","operations":[["vote",{"voter":"aleksandraz","author":"deanliu","permlink":"leslie-chow-from-hangover-is-in-a-hotel-room-next-door-to-me-live-from-taipei-taiwan","weight":10000}]],"extensions":[],"signatures":["201f722bb0bf4f9f28eba78117e0922bbbc2d5088aa24123f8048bf59e9a42c3362d8d23102a7e9ca10bf92b34c3a0b438f2e4740f2464150b158e3f1c717b968d"]},{"ref_block_num":35079,"ref_block_prefix":1911931486,"expiration":"2016-08-01T14:56:12","operations":[["vote",{"voter":"erikkartmen","author":"zyntra","permlink":"o-hai-steemit-molecular-mixologist-and-snowboard-instructor-just-checked-in","weight":10000}]],"extensions":[],"signatures":["206b1db069efd216fcedfef46394c735098d7650c2fcad3df90e1e9642ddfdd5f951e7e61758e58399c5fa5f9bc7df5d96f73acf640aa54a27aef310164d0dbb2f"]}]}}
    th_a  database.cpp:542 operator()

The two Heads do not match...

The first error I see on the node is about tx_missing_active_auth:

914758ms th_a       database.cpp:2917             _apply_block         ] 3010000 tx_missing_active_auth: missing required active authority
Missing Active Authority jesus2
    {"id":"jesus2","auth":{"weight_threshold":1,"account_auths":[],"key_auths":[["STM8UsRn6HbmRXkQSTtLRsUmXzb1HEvad5xU6AVY2XTugxYdd8s6P",1]]},"owner":{"weight_threshold":1,"account_auths":[],"key_auths":[["STM82yPKqrrrUEhuVYNEHEEqRWU16b4uyRmW6MXPDEzZgpJey9rsB",1]]}}
    th_a  transaction.cpp:157 verify_authority

    {"ops":[["account_update",{"account":"jesus2","posting":{"weight_threshold":1,"account_auths":[["STM6GVCS7JeftT27",1]],"key_auths":[["STM7xqXUoCzjB7BXvzE7hCjVcLpgpEVMy3ALy3ygCFX3opkmuhDNd",1]]},"memo_key":"STM8Mf3UvSfnoGtBsxi3SjaHYWuUxYgJUJKHWeMsJcLycH6jYMrM5","json_metadata":""}]],"sigs":["STM6vJVmV8maCscDU12e1wf8NGJX6TgKUjUVBih1VTimyF26xj9Qf"]}
    th_a  transaction.cpp:172 verify_authority

    {"*this":{"ref_block_num":35093,"ref_block_prefix":3914984601,"expiration":"2016-08-01T14:56:24","operations":[["account_update",{"account":"jesus2","posting":{"weight_threshold":1,"account_auths":[["STM6GVCS7JeftT27",1]],"key_auths":[["STM7xqXUoCzjB7BXvzE7hCjVcLpgpEVMy3ALy3ygCFX3opkmuhDNd",1]]},"memo_key":"STM8Mf3UvSfnoGtBsxi3SjaHYWuUxYgJUJKHWeMsJcLycH6jYMrM5","json_metadata":""}]],"extensions":[],"signatures":["201123be78b9ee9560e60b90b04712e87e89655c9b6419e32b9b0e715fa651ea8078b09182f2ea09bf0e00b80c3a4dea7a081ddf69972a288fbfc0d30c5ad68a14"]}}
    th_a  transaction.cpp:285 verify_authority

    {"trx":{"ref_block_num":35093,"ref_block_prefix":3914984601,"expiration":"2016-08-01T14:56:24","operations":[["account_update",{"account":"jesus2","posting":{"weight_threshold":1,"account_auths":[["STM6GVCS7JeftT27",1]],"key_auths":[["STM7xqXUoCzjB7BXvzE7hCjVcLpgpEVMy3ALy3ygCFX3opkmuhDNd",1]]},"memo_key":"STM8Mf3UvSfnoGtBsxi3SjaHYWuUxYgJUJKHWeMsJcLycH6jYMrM5","json_metadata":""}]],"extensions":[],"signatures":["201123be78b9ee9560e60b90b04712e87e89655c9b6419e32b9b0e715fa651ea8078b09182f2ea09bf0e00b80c3a4dea7a081ddf69972a288fbfc0d30c5ad68a14"]}}
    th_a  database.cpp:3119 _apply_transaction
914758ms th_a       database.cpp:2917             _apply_block         ] next_block.block_num(): 3705111 
914759ms th_a       database.cpp:622              _push_block          ] Failed to push new block:
3010000 tx_missing_active_auth: missing required active authority
Missing Active Authority jesus2
    {"id":"jesus2","auth":{"weight_threshold":1,"account_auths":[],"key_auths":[["STM8UsRn6HbmRXkQSTtLRsUmXzb1HEvad5xU6AVY2XTugxYdd8s6P",1]]},"owner":{"weight_threshold":1,"account_auths":[],"key_auths":[["STM82yPKqrrrUEhuVYNEHEEqRWU16b4uyRmW6MXPDEzZgpJey9rsB",1]]}}
    th_a  transaction.cpp:157 verify_authority

    {"ops":[["account_update",{"account":"jesus2","posting":{"weight_threshold":1,"account_auths":[["STM6GVCS7JeftT27",1]],"key_auths":[["STM7xqXUoCzjB7BXvzE7hCjVcLpgpEVMy3ALy3ygCFX3opkmuhDNd",1]]},"memo_key":"STM8Mf3UvSfnoGtBsxi3SjaHYWuUxYgJUJKHWeMsJcLycH6jYMrM5","json_metadata":""}]],"sigs":["STM6vJVmV8maCscDU12e1wf8NGJX6TgKUjUVBih1VTimyF26xj9Qf"]}
    th_a  transaction.cpp:172 verify_authority

    {"*this":{"ref_block_num":35093,"ref_block_prefix":3914984601,"expiration":"2016-08-01T14:56:24","operations":[["account_update",{"account":"jesus2","posting":{"weight_threshold":1,"account_auths":[["STM6GVCS7JeftT27",1]],"key_auths":[["STM7xqXUoCzjB7BXvzE7hCjVcLpgpEVMy3ALy3ygCFX3opkmuhDNd",1]]},"memo_key":"STM8Mf3UvSfnoGtBsxi3SjaHYWuUxYgJUJKHWeMsJcLycH6jYMrM5","json_metadata":""}]],"extensions":[],"signatures":["201123be78b9ee9560e60b90b04712e87e89655c9b6419e32b9b0e715fa651ea8078b09182f2ea09bf0e00b80c3a4dea7a081ddf69972a288fbfc0d30c5ad68a14"]}}
    th_a  transaction.cpp:285 verify_authority

    {"trx":{"ref_block_num":35093,"ref_block_prefix":3914984601,"expiration":"2016-08-01T14:56:24","operations":[["account_update",{"account":"jesus2","posting":{"weight_threshold":1,"account_auths":[["STM6GVCS7JeftT27",1]],"key_auths":[["STM7xqXUoCzjB7BXvzE7hCjVcLpgpEVMy3ALy3ygCFX3opkmuhDNd",1]]},"memo_key":"STM8Mf3UvSfnoGtBsxi3SjaHYWuUxYgJUJKHWeMsJcLycH6jYMrM5","json_metadata":""}]],"extensions":[],"signatures":["201123be78b9ee9560e60b90b04712e87e89655c9b6419e32b9b0e715fa651ea8078b09182f2ea09bf0e00b80c3a4dea7a081ddf69972a288fbfc0d30c5ad68a14"]}}
    th_a  database.cpp:3119 _apply_transaction
rethrow
    {"next_block.block_num()":3705111}
    th_a  database.cpp:2917 _apply_block

    {"next_block":{"previous":"00388916318d804f9371fe551732d707617c69e4","timestamp":"2016-08-01T14:56:00","witness":"witness.svk","transaction_merkle_root":"e6ad5676bf6cb1a393376fd4d15f43e9e76a535f","extensions":[],"witness_signature":"205094b547050ea59b5419792efbc633bdaac77f88e98c5927edb645bc642e7ee55254d4a9f51b77587d74eeb894ec96b2e6405a2d19fac2ea02b61dcec25f0b3f","transactions":[{"ref_block_num":35093,"ref_block_prefix":3914984601,"expiration":"2016-08-01T14:56:24","operations":[["account_update",{"account":"jesus2","posting":{"weight_threshold":1,"account_auths":[["STM6GVCS7JeftT27",1]],"key_auths":[["STM7xqXUoCzjB7BXvzE7hCjVcLpgpEVMy3ALy3ygCFX3opkmuhDNd",1]]},"memo_key":"STM8Mf3UvSfnoGtBsxi3SjaHYWuUxYgJUJKHWeMsJcLycH6jYMrM5","json_metadata":""}]],"extensions":[],"signatures":["201123be78b9ee9560e60b90b04712e87e89655c9b6419e32b9b0e715fa651ea8078b09182f2ea09bf0e00b80c3a4dea7a081ddf69972a288fbfc0d30c5ad68a14"]},{"ref_block_num":35067,"ref_block_prefix":2222533022,"expiration":"2016-08-01T14:56:12","operations":[["vote",{"voter":"kalimor","author":"domino","permlink":"re-lehard-steemit-dlya-chainikov-5-prostykh-shagov-ot-novichka-do-steemian-80-lvl-50-poleznykh-ssylok-2-redakciya-20160721t084441610z","weight":10000}]],"extensions":[],"signatures":["2030862252eb295c804e11e7ba40a9c02b9cfe8178522bc9ef22b8134b999da0c139d447752aefb29ee6f7caf310f2a052071a19fe9db07c4d2dbff69dbf7e31d4"]}]}}
    th_a  database.cpp:2802 apply_block
914759ms th_a       application.cpp:488           handle_block         ] Error when pushing block:
3010000 tx_missing_active_auth: missing required active authority
Missing Active Authority jesus2
    {"id":"jesus2","auth":{"weight_threshold":1,"account_auths":[],"key_auths":[["STM8UsRn6HbmRXkQSTtLRsUmXzb1HEvad5xU6AVY2XTugxYdd8s6P",1]]},"owner":{"weight_threshold":1,"account_auths":[],"key_auths":[["STM82yPKqrrrUEhuVYNEHEEqRWU16b4uyRmW6MXPDEzZgpJey9rsB",1]]}}
    th_a  transaction.cpp:157 verify_authority

    {"ops":[["account_update",{"account":"jesus2","posting":{"weight_threshold":1,"account_auths":[["STM6GVCS7JeftT27",1]],"key_auths":[["STM7xqXUoCzjB7BXvzE7hCjVcLpgpEVMy3ALy3ygCFX3opkmuhDNd",1]]},"memo_key":"STM8Mf3UvSfnoGtBsxi3SjaHYWuUxYgJUJKHWeMsJcLycH6jYMrM5","json_metadata":""}]],"sigs":["STM6vJVmV8maCscDU12e1wf8NGJX6TgKUjUVBih1VTimyF26xj9Qf"]}
    th_a  transaction.cpp:172 verify_authority

    {"*this":{"ref_block_num":35093,"ref_block_prefix":3914984601,"expiration":"2016-08-01T14:56:24","operations":[["account_update",{"account":"jesus2","posting":{"weight_threshold":1,"account_auths":[["STM6GVCS7JeftT27",1]],"key_auths":[["STM7xqXUoCzjB7BXvzE7hCjVcLpgpEVMy3ALy3ygCFX3opkmuhDNd",1]]},"memo_key":"STM8Mf3UvSfnoGtBsxi3SjaHYWuUxYgJUJKHWeMsJcLycH6jYMrM5","json_metadata":""}]],"extensions":[],"signatures":["201123be78b9ee9560e60b90b04712e87e89655c9b6419e32b9b0e715fa651ea8078b09182f2ea09bf0e00b80c3a4dea7a081ddf69972a288fbfc0d30c5ad68a14"]}}
    th_a  transaction.cpp:285 verify_authority

    {"trx":{"ref_block_num":35093,"ref_block_prefix":3914984601,"expiration":"2016-08-01T14:56:24","operations":[["account_update",{"account":"jesus2","posting":{"weight_threshold":1,"account_auths":[["STM6GVCS7JeftT27",1]],"key_auths":[["STM7xqXUoCzjB7BXvzE7hCjVcLpgpEVMy3ALy3ygCFX3opkmuhDNd",1]]},"memo_key":"STM8Mf3UvSfnoGtBsxi3SjaHYWuUxYgJUJKHWeMsJcLycH6jYMrM5","json_metadata":""}]],"extensions":[],"signatures":["201123be78b9ee9560e60b90b04712e87e89655c9b6419e32b9b0e715fa651ea8078b09182f2ea09bf0e00b80c3a4dea7a081ddf69972a288fbfc0d30c5ad68a14"]}}
    th_a  database.cpp:3119 _apply_transaction
rethrow
    {"next_block.block_num()":3705111}
    th_a  database.cpp:2917 _apply_block

    {"next_block":{"previous":"00388916318d804f9371fe551732d707617c69e4","timestamp":"2016-08-01T14:56:00","witness":"witness.svk","transaction_merkle_root":"e6ad5676bf6cb1a393376fd4d15f43e9e76a535f","extensions":[],"witness_signature":"205094b547050ea59b5419792efbc633bdaac77f88e98c5927edb645bc642e7ee55254d4a9f51b77587d74eeb894ec96b2e6405a2d19fac2ea02b61dcec25f0b3f","transactions":[{"ref_block_num":35093,"ref_block_prefix":3914984601,"expiration":"2016-08-01T14:56:24","operations":[["account_update",{"account":"jesus2","posting":{"weight_threshold":1,"account_auths":[["STM6GVCS7JeftT27",1]],"key_auths":[["STM7xqXUoCzjB7BXvzE7hCjVcLpgpEVMy3ALy3ygCFX3opkmuhDNd",1]]},"memo_key":"STM8Mf3UvSfnoGtBsxi3SjaHYWuUxYgJUJKHWeMsJcLycH6jYMrM5","json_metadata":""}]],"extensions":[],"signatures":["201123be78b9ee9560e60b90b04712e87e89655c9b6419e32b9b0e715fa651ea8078b09182f2ea09bf0e00b80c3a4dea7a081ddf69972a288fbfc0d30c5ad68a14"]},{"ref_block_num":35067,"ref_block_prefix":2222533022,"expiration":"2016-08-01T14:56:12","operations":[["vote",{"voter":"kalimor","author":"domino","permlink":"re-lehard-steemit-dlya-chainikov-5-prostykh-shagov-ot-novichka-do-steemian-80-lvl-50-poleznykh-ssylok-2-redakciya-20160721t084441610z","weight":10000}]],"extensions":[],"signatures":["2030862252eb295c804e11e7ba40a9c02b9cfe8178522bc9ef22b8134b999da0c139d447752aefb29ee6f7caf310f2a052071a19fe9db07c4d2dbff69dbf7e31d4"]}]}}
    th_a  database.cpp:2802 apply_block

    {"new_block":{"previous":"00388916318d804f9371fe551732d707617c69e4","timestamp":"2016-08-01T14:56:00","witness":"witness.svk","transaction_merkle_root":"e6ad5676bf6cb1a393376fd4d15f43e9e76a535f","extensions":[],"witness_signature":"205094b547050ea59b5419792efbc633bdaac77f88e98c5927edb645bc642e7ee55254d4a9f51b77587d74eeb894ec96b2e6405a2d19fac2ea02b61dcec25f0b3f","transactions":[{"ref_block_num":35093,"ref_block_prefix":3914984601,"expiration":"2016-08-01T14:56:24","operations":[["account_update",{"account":"jesus2","posting":{"weight_threshold":1,"account_auths":[["STM6GVCS7JeftT27",1]],"key_auths":[["STM7xqXUoCzjB7BXvzE7hCjVcLpgpEVMy3ALy3ygCFX3opkmuhDNd",1]]},"memo_key":"STM8Mf3UvSfnoGtBsxi3SjaHYWuUxYgJUJKHWeMsJcLycH6jYMrM5","json_metadata":""}]],"extensions":[],"signatures":["201123be78b9ee9560e60b90b04712e87e89655c9b6419e32b9b0e715fa651ea8078b09182f2ea09bf0e00b80c3a4dea7a081ddf69972a288fbfc0d30c5ad68a14"]},{"ref_block_num":35067,"ref_block_prefix":2222533022,"expiration":"2016-08-01T14:56:12","operations":[["vote",{"voter":"kalimor","author":"domino","permlink":"re-lehard-steemit-dlya-chainikov-5-prostykh-shagov-ot-novichka-do-steemian-80-lvl-50-poleznykh-ssylok-2-redakciya-20160721t084441610z","weight":10000}]],"extensions":[],"signatures":["2030862252eb295c804e11e7ba40a9c02b9cfe8178522bc9ef22b8134b999da0c139d447752aefb29ee6f7caf310f2a052071a19fe9db07c4d2dbff69dbf7e31d4"]}]}}
    th_a  database.cpp:542 operator()

All 11 comments

Relevant tx from account history:

90    3705111    45ba8e5eca6cd89a2f6f4c42af7bfe4f5f1b51d5 account_update       {"account":"jesus2","posting":{"weight_threshold":1,"account_auths":[["STM6GVCS7JeftT274UDKSFD2PGFMsJUxWMmQyTDCDnUouWkcReRFo",1]],"key_auths":[["STM7xqXUoCzjB7BXvzE7hCjVcLpgpEVMy3ALy3ygCFX3opkmuhDNd",1]]},"memo_key":"STM8Mf3UvSfnoGtBsxi3SjaHYWuUxYgJUJKHWeMsJcLycH6jYMrM5","json_metadata":""}

It seems jesus2 is one of the accounts which has a tx triggering account_auths length issue #465

We have not had any problems with this internally.

In order to get around this issue we added shared_db_merkle.hpp which added a pairing of block num to valid merkle checksums for shared-db.

This block is one that is accounted for.

{"trx":{"ref_block_num":35093,"ref_block_prefix":3914984601,"expiration":"2016-08-01T14:56:24","operations":[["account_update",{"account":"jesus2","posting":{"weight_threshold":1,"account_auths":[["STM6GVCS7JeftT27",1]],"key_auths":[["STM7xqXUoCzjB7BXvzE7hCjVcLpgpEVMy3ALy3ygCFX3opkmuhDNd",1]]},"memo_key":"STM8Mf3UvSfnoGtBsxi3SjaHYWuUxYgJUJKHWeMsJcLycH6jYMrM5","json_metadata":""}]],"extensions":[],"signatures":["201123be78b9ee9560e60b90b04712e87e89655c9b6419e32b9b0e715fa651ea8078b09182f2ea09bf0e00b80c3a4dea7a081ddf69972a288fbfc0d30c5ad68a14"]}}

The account_auths field showing in the json is obviously wrong. The account name "STM6GVCS7JeftT274UDKSFD2PGFMsJUxWMmQyTDCDnUouWkcReRFo" got truncated to "STM6GVCS7JeftT27". Conversion from string to account_name_type (aka fixed_string(16)) happened with this new code: https://github.com/steemit/steem/blob/0.14.0-shared-db/libraries/protocol/include/steemit/protocol/authority.hpp#L27 . As a result, the transaction content is changed after re-serialization, so the signature in the transaction become invalid. Perhaps we need a white list to handle these modified transactions.

Yes, we are aware of what is causing the problem. Specifically, authorities aren't correctly validating account names. The transaction itself should be invalid but is not because of a lack of restraints in the validation logic.

We don't do any signature validation on reindex. Under the current skip flag configuration, the merkle check is all that should need an exception table.

The error had to do with the fact that block producing nodes validate signatures on resync and non producing nodes do not. Our testing has been on nodes that do not produce blocks. Setting a checkpoint past block 5435426 should fix the problem. We can implement an exception table later to handle this problem, but for now it should get you up and running.

Thank you for testing this branch.

We don't do any signature validation on reindex

But how about resync?

//Update: this question is replied in the new comment ("block producing nodes validate signatures on resync and non producing nodes do not"). For anyone who have interest, related code is here: https://github.com/steemit/steem/blame/0.14.0-shared-db/libraries/app/application.cpp#L465

            bool result = _chain_db->push_block(blk_msg.block, (_is_block_producer | _force_validate) ? database::skip_nothing : database::skip_transaction_signatures);

By the way I wonder why there is | but not ||, although likely it's practically correct.

Testing a fix now.

I have a blockchain folder stuck at block 3705110 just to test this issue.

This commit does not seem to fix it.

Are you running with any extra configuration options? I did a full resync with --force-validate that was failing, made the change, and now it works.

Hold on, I may have broke the change.

Fixed.

Was this page helpful?
0 / 5 - 0 ratings