Solana: thread 'solana-repair-service' panicked at 'assertion failed: self.storage.read().unwrap().0.get(&remove_slot).is_none()', runtime/src/accounts_db.rs:1163:9

Created on 13 Jun 2020  路  12Comments  路  Source: solana-labs/solana

Problem

Thread 'solana-repair-service' panicked and killed this TdS validator.
Log and stack trace below/

Proposed Solution

TBD

[2020-06-13T09:11:29.859984901Z INFO  solana_metrics::metrics] datapoint: cluster_info_stats entrypoint=0i entrypoint2=0i push_vote_read=20i vote_process_push=19i get_votes=4401i get_accounts
_hash=0i all_tvu_peers=0i tvu_peers=9020i new_push_requests_num=2164i
[2020-06-13T09:11:29.860002264Z INFO  solana_metrics::metrics] datapoint: cluster_info_stats2 retransmit_peers=2184i repair_peers=32297i new_push_requests=2828i new_push_requests2=2i purge=85
334i process_pull_resp=9476i filter_pull_resp=15009i process_pull_resp_count=19i pull_response_fail_insert=996i pull_response_fail_timeout=310i pull_response_success=158i process_pull_resp_ti
meout=310i push_response_count=0i
[2020-06-13T09:11:29.860006432Z INFO  solana_metrics::metrics] datapoint: cluster_info_stats3 process_pull_resp_len=1613i process_pull_requests=17493i generate_pull_responses=174231i process_
prune=0i process_push_message=7263i prune_received_cache=622i epoch_slots_lookup=15i epoch_slots_push=20i push_message=0i new_pull_requests=301396i mark_pull_request=3i
[2020-06-13T09:11:29.860024145Z INFO  solana_metrics::metrics] datapoint: cluster_info_stats4 skip_push_message_shred_version=0i skip_pull_response_shred_version=0i skip_pull_shred_version=0i
 push_message_count=585i push_message_value_count=730i new_pull_requests_count=640i
[2020-06-13T09:11:29.867053185Z INFO  solana_core::replay_stage] bank frozen: 19768796
[2020-06-13T09:11:29.868035967Z INFO  solana_metrics::metrics] datapoint: replay-slot-stats slot=19768796i fetch_entries_time=10050i fetch_entries_fail_time=0i entry_verification_time=152i re
play_time=11052i replay_total_elapsed=179112i total_entries=79i total_shreds=69i
[2020-06-13T09:11:29.868047429Z INFO  solana_metrics::metrics] datapoint: collect_rent_eagerly accounts=0i
[2020-06-13T09:11:29.877029680Z INFO  solana_runtime::bank] bank frozen: 19768796 hash: CrSzxfGmnRrJX5zxLHkBGaABre53mDjA2HgfRHjb5WhL accounts_delta: 32zFwCt9K1TxFJBzqka2SPfrh9LZ8GgspBq2TsfKfm
qK signature_count: 22 last_blockhash: ED3bAE5TtaeeS4hvEfd7dmdUawU6SsRbZUREWhSPZJ2m capitalization: 3060532303688703776
[2020-06-13T09:11:29.877049087Z INFO  solana_runtime::bank] accounts hash slot: 19768796 stats: BankHashStats { num_updated_accounts: 22, num_removed_accounts: 0, num_lamports_stored: 4870081
8596820686, total_data_len: 161495, num_executable_accounts: 0 }
[2020-06-13T09:11:29.888970038Z INFO  solana_core::replay_stage] 9QxCLckBiJc783jnMvXZubK4wH86Eqqvashtrwvcsgkv slot_weight: 19768796 45651262031200697246592518 4776983652706108101297387489250
19768795
[2020-06-13T09:11:29.889017116Z INFO  solana_core::replay_stage] voting: 19768796 45651262031200697246592518
[2020-06-13T09:11:29.889022867Z INFO  solana_core::replay_stage] LEADER CHANGE at slot: 19768796 leader: 87VQhN7dUfS9wacre7vqRm561bNUk5PwUB8xmroc2yEw
[2020-06-13T09:11:29.890218311Z INFO  solana_core::replay_stage] new root 19768745
[2020-06-13T09:11:29.890696282Z INFO  solana_core::replay_stage] vote bank: Some(19768796) reset bank: 19768796
[2020-06-13T09:11:29.895156509Z INFO  solana_metrics::metrics] datapoint: tower-observed slot=19768782i root=19768725i
[2020-06-13T09:11:29.895167760Z INFO  solana_metrics::metrics] datapoint: bank_weight slot=19768796i weight="25C307DFA0C0AE2A58AE06"
[2020-06-13T09:11:29.895170996Z INFO  solana_metrics::metrics] datapoint: tower-vote latest=19768796i root=19768745i
[2020-06-13T09:11:29.895175345Z INFO  solana_metrics::metrics] datapoint: blocks_produced num_blocks_on_fork=105960i num_dropped_blocks_on_fork=24943i
[2020-06-13T09:11:29.895177889Z INFO  solana_metrics::metrics] datapoint: block-commitment-cache aggregate-commitment-ms=2i
[2020-06-13T09:11:29.901981755Z INFO  solana_core::poh_recorder] reset poh from: D7JgFqf3Y8y2hqJUxSBQBbUTBKJz5cYC3gv8XtuzXdu1,1265202990,19768795 to: ED3bAE5TtaeeS4hvEfd7dmdUawU6SsRbZUREWhSPZ
J2m,19768796
[2020-06-13T09:11:29.902007563Z INFO  solana_core::replay_stage] 9QxCLckBiJc783jnMvXZubK4wH86Eqqvashtrwvcsgkv reset PoH to tick 1265203008 (within slot 19768796). My next leader slot is 19770
080
[2020-06-13T09:11:29.950724985Z INFO  solana_core::replay_stage] new fork:19768801 parent:19768796 root:19768745
[2020-06-13T09:11:29.953532597Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=19768801i block_height=15767009i
[2020-06-13T09:11:29.999120510Z INFO  solana_metrics::metrics] datapoint: poh-service ticks=128i hashes=1602699i elapsed_ms=683i
[2020-06-13T09:11:30.122774997Z INFO  solana_core::cluster_info]
    IP Address        |Age(ms)| Node identifier                              | Version       |Gossip| TPU  |TPUfwd| TVU  |TVUfwd|Repair|ServeR| RPC  |PubSub|ShredVer
    ------------------+-------+----------------------------------------------+---------------+------+------+------+------+------+------+------+------+------+--------
    95.217.197.103  me|  2304 | 9QxCLckBiJc783jnMvXZubK4wH86Eqqvashtrwvcsgkv |1.2.1 devbuild | 9000 | 9003 | 9004 | 9001 | 9002 | 9006 | 9007 | 8899 | 8900 | 56096
    116.202.109.229   |  3414 | 9dCMmPpfNuWKyZ2D1iRstMCc1rhr2DbHVFrZ9wFncQjp |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    62.171.184.84     |   281 | 7arfejY2YxX9QrmzHrhu3rG3HofjMqKtfBzQLf8s3Wop |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.217.153.227    |   281 | 5NH47Zk9NAzfbtqNpUtn8CQgNZeZE88aa2NRpfe7DyTD |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    159.65.83.2       |   281 | A1voPbfnmCq8UBNQTBKnZ3Xbhs2x4cS2Gx2b2wJtqCh1 |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.217.128.244    |   278 | D23NCAVxinE53BTemguZCheAqCdMGfNTUzWdoWvq4Xj  |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    62.171.182.213    |  1045 | 6ZEbKFxTjEKGC9HUqzy9z4ccJ8Aq3ktPKEzHGDosQJo4 |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    159.69.244.44     |   278 | URnkWZGiuB7jXbfCSuNSwir1qkn7sXjiSPeLPaXys7b  |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.216.137.174    |  3404 | 74U9LiSPv2gb8bBZSh5uVNf89W4wZ8zs9B8EvRVVwr87 |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    5.12.185.208      |   226 | DFb6qaAkd5DTnFVYLDjzJNfsUPygP8GHHebN1CBv25cf |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.216.36.34      |   368 | 8nvD1CUE48WdcmRdvbyWcM5LdJKRTNP3tXT6Qp2CSND5 |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    54.229.26.131     |   281 | H4Dgb3KyCuYWKT8yKtp8qbY49cvaqZcisa2GDnroFsv6 |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | none | none | 56096
    95.216.96.141     |  3404 | CiY5RjWPs1XyegKyBLcG7Ue7YMf98eiEnmvqnSuSKbob |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    195.201.160.171   |   226 | 9gmnbM2GUVXiTfCg1Pj3ZTJdqyKdS81kjBWwwnZbS4MR |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | none | none | 56096
    5.189.150.170     |   278 | 84BoHfEzq1FccYEMZPeuTwc68DQC7LS6Bm5dRMsKypA5 |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    94.130.15.229     |   278 | 8onJp7KyshoMcxVm5CemhPgGvA1hdSnHbcjLCvJidV8o |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    113.88.136.167    |   278 | 8xFQP5mPt9Nzr5wMeUsH4azMM3zyGqrUn3LeopKSwptX |1.2.1 devbuild | 19003| 19006| 19007| 19004| 19005| 19009| 19010| 19001| 19002| 56096
    95.217.145.161    |   229 | 123vij84ecQEKUvQ7gYMKxKwKF6PbYSzCzzURYA4xULY |1.2.1 devbuild | 18021| 18024| 18025| 18022| 18023| 18027| 18028| none | none | 56096
    95.217.114.150    |   278 | 2ibbdJtxwzzzhK3zc7FR3cfea2ATHwCJ8ybcG7WzKtBd |1.1.17 3f60fe62| 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | none | none | 56096
    95.217.203.156    |   281 | 2xFjhfxTKGVvGDXLwroqGiKNEF3KCSFaCRVLHfpsiPgd |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    144.91.83.172     |   368 | AkkJv1meyo2Ax2XTXEXWpvHTh4F8a68Lja5dx3TaX47K |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    195.201.109.247   |   278 | CsaAGRau3ZvyMQvJ9CWSqbqeVv9zw2Am8FhnL9sr6jTk |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    116.202.233.120   |  3943 | 9qpfxCUAPyaYPchHgRGXmNDDhPiNmJR39Lfx4A49Uh1P |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.217.45.22      |  3444 | 5TkrtJfHoX85sti8xSVvfggVV9SDvhjYjiXe9PqMJVN9 |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.217.144.107    |   281 | 3W4fe5WTAS4iPzBhjGP8a1LHBTx8vbscqThXT1THqEGC |1.1.17 3f60fe62| 9001 | 9004 | 9005 | 9002 | 9003 | 9007 | 9008 | 8899 | 8900 | 56096
    95.217.122.169    |   281 | 5H3sMCaSJdN2k1hyuFTzq2BrZHUq7CinTa82hJS6EDTf |1.1.17 3f60fe62| 10001| 10004| 10005| 10002| 10003| 10007| 10008| 8899 | 8900 | 56096
    83.85.228.65      |  1812 | 33LfdA2yKS6m7E8pSanrKTKYMhpYHEGaSWtNNB5s7xnm |1.2.1 ba36308d | 8001 | 8003 | 8004 | 8000 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    88.198.66.56      |   284 | 9oKrJ9iiEnCC7bewcRFbcdo4LKL2PhUEqcu8gH2eDbVM |1.1.17 3f60fe62| 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    144.76.172.165    |  3943 | HwFvyMbGLkiTUaT66cfL1FnJ26c9VqtpqAg4UbWSXtdq |1.1.17 3f60fe62| 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    136.243.8.205     |   278 | 8dHEsm9aLBt7q6zu3ESfRXkS2eCwkbbzzynfd2QxDzms |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | none | none | 56096
    95.217.196.211    |   278 | 4gMboaRFTTxQ6iPoH3NmxLw6Ux3SEAGkQjfrBT1suDZd |1.1.16 d5d5e879| 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    213.133.99.209    |   368 | 731Lnc3mbXpquV8FmFnTL8EE36uoZRiDMUKXehwZq8x2 |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.111.244.71     |   226 | 4fBQr617DmhjekLFckh2JkGWNboKQbpRchNrXwDQdjSv |1.1.17 3f60fe62| 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | none | none | 56096
    122.11.202.58     |  5282 | 5KZRD6hDCZd1dgM9rJLweFFRkHu3qAgBhHgQ96Wz1VSn |1.1.17 3f60fe62| 8000 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 9001 | 9002 | 56096
    95.217.192.39     |   284 | 5rxRt2GVpSUFJTqQ5E4urqJCDbcBPakb46t6URyxQ5Za |1.1.17 3f60fe62| 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | none | none | 56096
    88.99.6.218       |   281 | BWiftESMUsve87rkjU7HsaA7fkiJRAbv3xZLQrmKZtnz |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    95.217.121.184    |  2682 | 7dEjSFnrm66CJ7Aj5mC1hsYmMzmGgWPr6iZNhcvANZ1w |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    192.145.45.35     |  2867 | 8ZjS3d1bQihC3p5voM8by2xi5PoBNxzTJtaQ9rvxUbbB |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    159.69.58.7       |   229 | EBxhSfAWW2Cfouvj1k242W6U8krZVAxJS47SG8UKb4ch |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    213.239.200.148   |   226 | 4B25C4dkCx8wALoXz1mCXsTY3UbefBYbBXCyCpx1Yyf6 |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    174.112.70.64     |   281 | fish2tcDFRiAM1cNKqF7b96WzQmmDRjsf5XD214MhpV  |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    88.198.66.152     |   281 | 7sANjVRV4JRst8nAJTTLDfTgEGWTCbw8cEoxkBxSgxWk |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    95.217.33.217     |   278 | 2X5JSTLN9m2wm3ejCxfWRNMieuC2VMtaMWSoqLPbC4Pq |1.1.17 3f60fe62| 8001 | 8003 | 8004 | 8000 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    207.180.255.2     |   189 | hQBS6cu8RHkXcCzE6N8mQxhgrtbNy4kivoRjTMzF2cA  |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    161.97.68.14      |  4034 | DVnKs7XAL9au7cWrTT335gZ3agJVwrqeSVnSWANo1SJG |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    152.32.186.68     |   368 | xViooFwobBZE416moj2yFV6RJwGVaBDbCQJRfCyz2FS  |1.1.17 3f60fe62| 8009 | 8012 | 8013 | 8010 | 8011 | 8015 | 8016 | 9999 | 10000| 56096
    148.251.66.119    |   278 | HeeEbBAkuLzqxsFLcbKUfWmeNizywy2uzAfvRg63LFT2 |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.217.193.34     |  3404 | A9XUvhm5yKVs9Z3tYdyiAYRx9mNr2rqnv2VkY8D1N4uZ |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | none | none | 56096
    35.237.144.155    |  3404 | 2DvsPbbKrBaJm7SbdVvRjZL1NGCU3MwciGCoCw42fTMu |1.2.1 ba36308d | 11000| 11003| 11004| 11001| 11002| 11006| 11007| none | none | 56096
    148.251.19.197    |   616 | 3JeLfM9V7CkK1D8W8LFirU65AZsWRo127PumwNc3dFQi |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    173.249.58.103    |   226 | 57DPUrAncC4BUY7KBqRMCQUt4eQeMaJWpmLQwsL35ojZ |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    116.203.110.213   |  3943 | Fb5cEcYNgPXKJoEmvPvsU2ENYRVePQtExqgf77AnVX54 |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    108.49.70.159     |   229 | 9uEazQxpRTyYX1hHMMySDBWad7zb54K9PkHKeZemK2m7 |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | none | none | 56096
    93.104.210.89     |   278 | DKnZytVA5wKbNPYW1pvPpoE5YeSsxu12KJFa95gBAGm7 |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.217.157.187    |   278 | 4WufhXsUhPc7cdHXYxxDrYZVVLKa9jCDGC4ccfmuBvu2 |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    85.190.150.169    |   226 | Ft5fbkqNa76vnsjYNwjDZUXoTWpP7VYm3mtsaQckQADN |1.2.1 baadf00d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.217.193.110    |   226 | Dx4bMuKpGaxAnd53QYDyKhD45PjuFLx16mrgoRK36STf |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    95.217.193.109    |  2682 | Aom2EwxRjtcCZBDwqvaZiEZDPgmw4AsPQGVrsLa2srCg |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    95.217.36.145     |   281 | AiWqv1dqsbvkUMec7G4DmM88ka7SaoqDPkn5U2iuvqnn |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | none | none | 56096
    95.111.238.43     |  3414 | BHR2K2tpc1fowNyUf4PfAumc2tfaT2SpvQVqmmpuN6tF |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    86.125.41.159     |  3943 | ErbvzZx2Ss9GxizKyDviybhZPu8noHv4AM5vuzTh1ij6 |1.1.17 3f60fe62| 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | none | none | 56096
    86.120.124.27     |   281 | DadnDZbFH5BHHRHD7TaobaSQ7QATXgvWegHUcZ7ZGzmW |1.1.17 devbuild| 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    89.42.234.74      |  3404 | 6PwxMMGLFnAf9sjMHfVr15z9fntjYTNPxJ7gFhkFxkXi |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    116.202.196.147   |   278 | A5hMwgm8QfooAuCMw9Rw2S9vXbBwCknFMhhUwKKHvYeJ |1.2.1 ba36308d | 6111 | 6114 | 6115 | 6112 | 6113 | 6117 | 6118 | 5511 | 5512 | 56096
    89.33.8.237       |   226 | F7FgS6rrWckgC5X4cP5WtRRp3U1u12nnuTRXbWYaKn1u |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | none | none | 56096
    95.216.243.4      |   226 | GFFuGhyHAr2fjH1DL42m9EWpAWXXdZ7R6PyzuMzDodLy |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    216.24.140.155    |   278 | 5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on |1.1.17 3f60fe62| 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    195.171.215.154   |   278 | GuKn8nEJwUPjBfxpwyq2MXU2JNrSpj7gqnKptCZeEk7j |1.2.1 devbuild | 8820 | 8823 | 8824 | 8821 | 8822 | 8826 | 8827 | 8804 | 8805 | 56096
    91.90.148.195     |   440 | 55nmQ8gdWpNW5tLPoBPsqDkLm1W24cmY5DbMMXZKSP8U |1.1.17 3f60fe62| 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | none | none | 56096
    94.130.25.240     |  3404 | Fv4zJ7RvV8gEYxEtLjnGZAX1qxjqRh56DzBgqvFEVjjM |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    88.99.224.138     |   284 | DzxNmWD99qvkPdDR94ojXhUrpzT8VdqB5ktYX7fZr4gc |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.217.121.146    |   281 | 4g5gX1mmFGGragqYQ1AsRpB8ZJvwCoUKVT5LtKTDrNSp |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    207.180.195.178   |   229 | 2tZoLFgcbeW8Howq8QMRnExvuwHFUeEnx9ZhHq2qX77E |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    14.52.147.51      |   278 | HEbMY624UhDGm1Qhy6neKSyi3bQjQ2RidSTyt7ARK8RW |1.2.1 ba36308d | 30000| 30003| 30004| 30001| 30002| 30006| 30007| 30899| 30900| 56096
    51.89.194.50      |  3414 | 5vxoRv2P12q4K4cWPCJkvPjg6jYnuCYxzF3juJZJiwba |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    209.222.97.92     |   281 | HM2hzFLTd5TAhejGFjaXAm8LLjdmnj7bqQrzpRTaawdo |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.217.59.238     |  3951 | 2GAdxV8QafdRnkTwy9AuX8HvVcNME6JqK2yANaDunhXp |1.1.17 3f60fe62| 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | none | none | 56096
    51.79.83.74       |  1268 | 47UuTGPAQZX2HnVcfxKk8b1BtA4rRTduVaHnvxzQe6AJ |1.1.16 d5d5e879| 8001 | 8003 | 8004 | 8000 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.217.36.182     |  1264 | 4Nh8T1d4YBZHEuQNRmFbLXPT5HbWicqPxGeKZ5SdAr4i |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | none | none | 56096
    88.198.39.210     |   229 | 4Bx5bzjmPrU1g74AHfYpTMXvspBt8GnvZVQW3ba9z4Af |1.1.17 3f60fe62| 9666 | 9669 | 9670 | 9667 | 9668 | 9672 | 9673 | 8899 | 8900 | 56096
    89.42.234.59      |   281 | B4xFUYq2TDmz6PsiCj29vFyvmYvqTRAtnhQ3uSmxQVFd |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.217.181.253    |   284 | DaB3ZwVtGLzSjazk5STQEu3MkJR2nkK3tDdCPAvx9QpM |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.217.192.222    |   284 | B6ZramCQhQWcq4Vxo3H5y1MYvo37wRZiAwKk88qaYNiF |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    3.82.158.154      |  3404 | B52Da5MCyTcyVJEsR9RUnbf715YuBAJMxCEEPzyZXgvY |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.217.121.152    |  2867 | 4dWYFeMhh2Q6bqXdV7CCd4mJC81im2k6CXCBKVPShXjT |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    188.40.100.189    |   281 | C4N1bMSzbfDwHGMitxyufNZPaAkNYx8vJxHRnHWrptAT |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    78.46.124.140     |   284 | 5vaCfp7UEpW5qdJYyVH4m93oMzzyzTqXdbr7xLGobY8q |1.2.1 ba36308d | 8001 | 8003 | 8004 | 8000 | 8002 | 8006 | 8007 | 19001| 19002| 56096
    138.201.135.79    |   368 | D2NjDkcv8Y1dWGdtWAKPT4em2D3sYzM8AzMTpCG1RVf7 |1.1.16 d5d5e879| 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    95.168.167.81     |  3404 | 5NorYZBbtgbouD3toX3761ZGbaYTWrNSDNci4G4zV8eo |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    94.130.54.247     |   281 | AbF88hkkpZ28VaT3vYn4xu5CeNC8G6Dq9cc8ciRR4fY5 |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    95.217.144.105    |   226 | CPPVEbGFbX3XAThetvfveCE1vYLWUwwJGT7DxkPAWb8D |1.1.17 3f60fe62| 9001 | 9004 | 9005 | 9002 | 9003 | 9007 | 9008 | 8899 | 8900 | 56096
    95.217.208.140    |  3414 | 2JT1SRSm61vvHKErY2PCnHUtMsumoh69jrC7bojd9f1x |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | none | none | 56096
    194.60.196.133    |   284 | 4XWxphAh1Ji9p3dYMNRNtW3sbmr5Z1cvsGyJXJx5Jvfy |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    83.54.21.63       |  3444 | 5dLMRyPWx6rdPGZpZ7uuZiqry96dUT5yz48u62Gzugi6 |1.1.17 3f60fe62| 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    159.69.119.87     |   281 | FzAv1TFpCyR65GrxeqBwnEzNVXEeUMPV5rKZGQhPR7mq |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    88.198.63.78      |  1262 | EWg9NTC5s7Pa9FktUk6dX8xRYkvJ952peH1z1iznd4nV |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    144.91.110.116    |   278 | H8kdiUSyvHbxshcFmRqWTB1HZkQHKcQcagQ56TzLe8ib |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.217.192.210    |  2687 | B8T2dbvYaM4gJfpLLbuBScuWKFhnG6KspRK72v5D5uoK |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    116.203.29.53     |   284 | 6qJPxxgZHCQKBvbGC9zCsuuPtHMMLszVCoiCvEhVULyJ |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    136.243.65.35     |  3404 | 8PTjAikKoAybKXcEPnDSoy8wSNNikUBJ1iKawJKQwXnB |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    144.91.94.30      |    12 | 2ZZkgKcBfp4tW8qCLj2yjxRYh9CuvEVJWb6e2KKS91Mj |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    165.227.100.142   |   868 | 71bhKKL89U3dNHzuZVZ7KarqV6XtHEgjXjvJTsguD11B |1.2.1 ba36308d | 8001 | 8003 | 8004 | 8000 | 8002 | 8006 | 8007 | none | none | 56096
    13.115.229.115    |   226 | AkVMbJq8pqKEe87uFaxjmt35tX2cNhUJTJwv13iioHu7 |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    38.135.195.85     |  1021 | 2jrM8c8ZhpX9CovseJ2sRsrU9yzsFrP7d3gCi5ESU5Rs |1.2.1 ba36308d | 8001 | 9054 | 9055 | 9052 | 9053 | 9057 | 9058 | 8899 | 8900 | 56096
    116.202.225.7     |  3943 | 8oRw7qpj6XgLGXYCDuNoTMCqoJnDd6A8LTpNyqApSfkA |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    217.69.11.199     |   284 | 2yDwZer11v2TTj86WeHzRDpE4HJVbyJ3fJ8H4AkUtWTc |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.217.193.177    |   226 | CUdHsUm5eaCZRctJo4HayvvYeh3AE3wCrNYXvMtVniBc |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    178.63.94.21      |  3414 | 2Pik6jn6yLQVi8jmwvZCibTygPWvhh3pXoGJrGT3eVGf |1.1.17 3f60fe62| 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | none | none | 56096
    116.202.224.31    |  3404 | GbtVg3D6bNFSjem21vrJBJTpUniwwEtmvs8mQkX5XS1V |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    46.4.19.106       |   284 | 9Q8xe8KgzVf2tKwdXgjNaYdJwvChihmjhcHdae7c4jPb |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    88.198.24.93      |  3943 | D52Q6Ap8RVMw1EvJYTdEABP6M5SPg98aToMcqw7KVLD9 |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8800 | 8801 | 56096
    88.99.186.187     |   226 | 3i7sS5McrJ7EzU8nbdA5rcXT9kNiSxLxhwyfuxbsDvBj |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.217.212.93     |   284 | HgmPwzNcY85HfrN3bYiqaypb6Nmf7ayaZEaivGY37913 |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    161.97.68.37      |   368 | VCRrRTgSjDLHvo6UQKXy8VQbNVG2ioHNUEyS7oB7u3X  |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.216.241.93     |  2686 | ChLMXZ4KXsMpa8W1VymMxim1vdPSK5a1jDwfMbm7cycT |1.2.1 ba36308d | 8001 | 11002| 11003| 11000| 11001| 11005| 11006| none | none | 56096
    95.216.158.252    |   281 | Hu7DW7BoXXuKbwaFJaAMEXpBv8pqBJPhfThMD96WHiJR |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.217.195.236    |   368 | 3FhfNWGiqDsy4xAXiS74WUb5GLfK7FVnn6kxt3CYLgvr |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    161.97.68.191     |   284 | BunxTHgkSEyHHMikCe9ofDB5dsgcXKN6nqKC5WQsd1op |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    144.91.126.158    |  3404 | AicKhNhJmkdqafRDjKLPgVqLzXLzJ8pS6aVrYrRkq1iq |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    173.231.58.226    |   284 | ChorusM5BVgnAKbg9PF15285LkqeCoZWK2p9s35T7J2A |1.1.17 devbuild| 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    213.239.200.147   |  2680 | 7pKVxV6BtnZ2aQMjpHXJSWLWmk33ivjRQ4kCPyESseLJ |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    150.238.34.227    |   368 | 77uXenX1Y9T2D1pcnHnYsYiwTTHbnzkyrKX5fQFMGVCR |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    152.32.185.183    |   278 | 836riBS2E6qfxjnrTkQdzD1JkFAoQDyUjTmzi38Gg84w |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.217.203.95     |  3943 | DnwgcbSGsr4RxDgoKpUTC5NdfadCKSXmZ8fDy9Nk1kSd |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    78.47.100.250     |  3404 | 87VQhN7dUfS9wacre7vqRm561bNUk5PwUB8xmroc2yEw |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    46.160.240.132    |   278 | CZWpCTN4rCWer8fm5ZqFdx82CDiCJjZLKZ5Ti2gdmchQ |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | none | none | 56096
    161.97.68.65      |  3404 | C3hD8Q7dLoodUm6E6LTWR3XqJgcRrvrVaMscwMBV6vaV |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    152.32.185.183    |   278 | 7mmRxJNttYcJVNsJmiLjTHYmNDt32EpzUSgLDoBsTKKK |1.1.17 3f60fe62| 8009 | 8012 | 8013 | 8010 | 8011 | 8015 | 8016 | 9999 | 10000| 56096
    95.217.122.207    |  2868 | BLfpk2WoF8RnerCqjxHe7qFJjpEE427riMkf2fS6vUB6 |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | 8899 | 8900 | 56096
    193.105.188.243   |  3434 | D71JRzjPpHipt8NAWnWb3yZoXezbkGXqSf7TVCir6wvT |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    90.188.226.112    |    40 | HGHMEEHCfbVFjqB69Hu9oNW6SviukB8jUheEhYVZJKe2 |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.217.193.36     |  3404 | 3ANJb42D3pkVtntgT6VtW2cD3icGVyoHi2NGwtXYHQAs |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | none | none | 56096
    95.216.46.162     |  3404 | Db2V7nPHc4sPHne87nYXPGn8Kv8rMsiWCAjgAXmpqcpC |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.217.196.126    |  1015 | 7X3csFXUN2AZph83GC2FZCpkCTZXfVWssaJ72cpwG96w |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.217.196.110    |   229 | 8aZ5CJf9qYnQtT2XYuDKASj2pCiPwhWoNsc2rBqc9s1n |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    136.24.223.18     |  2867 | GcwgiTexA2xbj8GCjDs1YWLcTPqtcb3qZa5WtZSWzNQr |1.2.2 devbuild | 15001| 15004| 15005| 15002| 15003| 15007| 15008| 15400| 15401| 56096
    95.217.119.153    |   229 | 8RsYRsi6f3hiK4EhyLS22Cy5KkrNbuidVYmsaYR1Xx78 |1.2.1 ba36308d | 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    185.144.83.136    |   278 | 8diJdQj3y4QbkjfnXr95SXoktiJ1ad965ZkeFsmutfyz |1.2.1 ba36308d | 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | none | none | 56096
    15.188.56.237     | 50659941 | VzZqDXSDF8KbfAQUvizj52NTjRg9cfXCAZ1cUJt5614  |1.1.13 4d181442| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    145.239.5.228     | 30099143 | D2PNC1USZ1XZ7mZPgdLjAvbDnaWBx59WKKM3CxjAqyu9 |1.1.17 3f60fe62| 8000 | none | none | none | none | none | none | none | none | 56096
    116.202.196.217   |  3414 | BSF2yD9mqzaixDaLEraF1en82EWaXx7wbaCqSuKppqG5 |1.2.1 ba36308d | 6111 | 6114 | 6115 | 6112 | 6113 | 6117 | 6118 | 5511 | 5512 | 56096
    93.81.246.101     |   281 | 8t6UUXRkQTBpanRoMjxNxio1baXXkEdeLniCVJGMdzLJ |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    46.4.116.196      | 50659731 | 6PojfJo3DkFNHsybGRQr3E7DmtUPvnnUdeFFgN2MBkKE |1.3.0 devbuild | 8001 | none | none | none | none | none | none | none | none | 56096
    145.239.5.228     | 45797446 | 2nTdhg2ZyWjRtHHBvKDBaNmtpXDsMXTmszgChpkXVaeU |1.2.1 ba36308d | 8000 | none | none | none | none | none | none | none | none | 0
    145.239.5.228     | 37199459 | EAMwsb5gBgeAiuxyU9yjN8V7veo1zA87JcsCMhATixqp |1.1.17 3f60fe62| 8000 | none | none | none | none | none | none | none | none | 56096
    152.32.186.68     |   226 | 234u57PuEif5LkTBwS7rHzu1XF5VWg79ddLLDkYBh44Q |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    62.171.154.199    | 50657799 | 3w6hQh7Ndx93eqbaEMLyR3BwqtRxT2XVumavvU93mcRk |1.1.16 d5d5e879| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    95.216.21.42      |   673 | 8NndwQsrH4f6xF6DW1tt7ESMEJpKz346AGqURKMXcNhT |1.1.17 3f60fe62| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | 8899 | 8900 | 56096
    211.212.16.31     | 4173950 | HfxzFiP19ymtxHagP4Zpga2zVo6ZgivpK6VkBKDowHRr |1.2.1 ba36308d | 8001 | none | none | none | none | none | none | none | none | 56096
    95.111.248.181    |  3434 | CV3F19YAhoW7DpfHQ5W9t2Zomb9h21NRi8k6hCA36Sk6 |1.1.17 3f60fe62| 8001 | 8004 | 8005 | 8002 | 8003 | 8007 | 8008 | none | none | 56096
    35.203.170.30     |  6328 | 3n9siYMs9TmVb6KBrCWfEruxeLULzxR39yzDqvpwiS4Y |       -       | 8001 | none | none | none | none | none | none | none | none | 56096
    18.234.189.242    |   278 | B5xrAZJNBGbGUoU52enKqTJxZKt7ahAbkvhJh7RzXm47 |1.1.16 devbuild| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | none | none | 56096
    3.235.248.19      |  4034 | 2nk5vxMvNs1R1UjQjDwWEkyBuqSBGpMk5umwpiZxKViL |1.1.17 devbuild| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | none | none | 56096
    148.251.19.197    |   229 | GDvAGACHCgdXUPKPjz2JE6Bkve265idrkM3cz6PdYfmn |1.2.1 ba36308d | 8009 | none | none | none | none | none | none | none | none | 0
    54.175.106.224    |  3951 | 5WvzBKPriX2qHX7t3Ex8jN5EfAj7rsnPxMmFzf3J63f6 |1.1.16 devbuild| 8000 | 8003 | 8004 | 8001 | 8002 | 8006 | 8007 | none | none | 56096
    Nodes: 148
    Spies: 7
    Nodes with different shred version: 104
[2020-06-13T09:11:30.142999850Z INFO  solana_metrics::metrics] datapoint: shred_fetch_tvu_forwards index_overrun=0i shred_count=542i slot_bad_deserialize=0i index_bad_deserialize=0i index_out_of_bounds=0i slot_out_of_range=0i duplicate_shred=0i
[2020-06-13T09:11:30.322664956Z INFO  solana_metrics::counter] COUNTER:{"name": "poh_recorder-tick_lock_contention", "counts": 8882362, "samples": 10126000,  "now": 1592039490322, "events": 1}
[2020-06-13T09:11:30.322696586Z INFO  solana_metrics::counter] COUNTER:{"name": "poh_recorder-tick_overhead", "counts": 3040134, "samples": 10126000,  "now": 1592039490322, "events": 0}
[2020-06-13T09:11:30.500992922Z INFO  solana_metrics::metrics] datapoint: shred_fetch_repair index_overrun=0i shred_count=7579i slot_bad_deserialize=0i index_bad_deserialize=0i index_out_of_bounds=0i slot_out_of_range=7559i duplicate_shred=0i
[2020-06-13T09:11:30.520997731Z INFO  solana_core::replay_stage] bank frozen: 19768801
[2020-06-13T09:11:30.521002620Z INFO  solana_metrics::metrics] datapoint: replay-slot-stats slot=19768801i fetch_entries_time=8807i fetch_entries_fail_time=0i entry_verification_time=473i replay_time=5461i replay_total_elapsed=563967i total_entries=320i total_shreds=66i
[2020-06-13T09:11:30.521040111Z INFO  solana_metrics::metrics] datapoint: collect_rent_eagerly accounts=0i
[2020-06-13T09:11:30.523363380Z INFO  solana_runtime::bank] bank frozen: 19768801 hash: HSMrYXchCHg99JB7oev4BwzBYLbJTFqkrd1GfyraxCSX accounts_delta: Bb5ePCX8TNaF1rhndKS6AFjo9uicx8hQHuUPLGn6FYcv signature_count: 0 last_blockhash: Fc4bb3ePu9iT6AF6v75xJQ4UjNWkq4SHLjJ7KooZ3FGj capitalization: 3060532303688703776
[2020-06-13T09:11:30.523377567Z INFO  solana_runtime::bank] accounts hash slot: 19768801 stats: BankHashStats { num_updated_accounts: 5, num_removed_accounts: 0, num_lamports_stored: 5, total_data_len: 157641, num_executable_accounts: 0 }
[2020-06-13T09:11:30.542980648Z INFO  solana_metrics::metrics] datapoint: tower-observed slot=19768782i root=19768725i
[2020-06-13T09:11:30.544764140Z INFO  solana_core::replay_stage] 9QxCLckBiJc783jnMvXZubK4wH86Eqqvashtrwvcsgkv slot_weight: 19768801 42856577643961673982519826 4777026509283752062971370009076 19768796
[2020-06-13T09:11:30.544813203Z INFO  solana_core::replay_stage] voting: 19768801 42856577643961673982519826
[2020-06-13T09:11:30.544818683Z INFO  solana_core::replay_stage] LEADER CHANGE at slot: 19768801 leader: Ft5fbkqNa76vnsjYNwjDZUXoTWpP7VYm3mtsaQckQADN
[2020-06-13T09:11:30.546940733Z INFO  solana_metrics::metrics] datapoint: bank_weight slot=19768801i weight="23733BBAB647442415AA12"
[2020-06-13T09:11:30.546955300Z INFO  solana_metrics::metrics] datapoint: tower-vote latest=19768801i root=19768745i
[2020-06-13T09:11:30.548993863Z INFO  solana_metrics::metrics] datapoint: block-commitment-cache aggregate-commitment-ms=2i
[2020-06-13T09:11:30.551072992Z INFO  solana_core::replay_stage] vote bank: Some(19768801) reset bank: 19768801
[2020-06-13T09:11:30.551099673Z INFO  solana_metrics::metrics] datapoint: blocks_produced num_blocks_on_fork=105961i num_dropped_blocks_on_fork=24947i
[2020-06-13T09:11:30.553071059Z INFO  solana_core::poh_recorder] reset poh from: 2gnaZWGekPyFpqCs9WCdMTjCHo35YtfLc9HA4uRSsnrg,1265203092,19768796 to: Fc4bb3ePu9iT6AF6v75xJQ4UjNWkq4SHLjJ7KooZ3FGj,19768801
[2020-06-13T09:11:30.553094763Z INFO  solana_core::replay_stage] 9QxCLckBiJc783jnMvXZubK4wH86Eqqvashtrwvcsgkv reset PoH to tick 1265203328 (within slot 19768801). My next leader slot is 19770080
[2020-06-13T09:11:30.619147331Z INFO  solana_core::replay_stage] new fork:19768802 parent:19768801 root:19768745
[2020-06-13T09:11:30.621006967Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=19768802i block_height=15767010i
[2020-06-13T09:11:30.836843776Z INFO  solana_metrics::counter] COUNTER:{"name": "bank-process_transactions-txs", "counts": 8481583, "samples": 10726000,  "now": 1592039490836, "events": 1}
[2020-06-13T09:11:30.836858504Z INFO  solana_metrics::counter] COUNTER:{"name": "bank-process_transactions-sigs", "counts": 14027439, "samples": 10726000,  "now": 1592039490836, "events": 2}
[2020-06-13T09:11:30.848948052Z INFO  solana_metrics::counter] COUNTER:{"name": "vote-native", "counts": 10726001, "samples": 10726000,  "now": 1592039490848, "events": 1}
[2020-06-13T09:11:30.895987661Z INFO  solana_metrics::metrics] datapoint: poh-service ticks=128i hashes=1610654i elapsed_ms=887i
[2020-06-13T09:11:30.988082910Z INFO  solana_metrics::metrics] datapoint: cluster_info_stats entrypoint=0i entrypoint2=0i push_vote_read=43i vote_process_push=28i get_votes=5316i get_accounts_hash=0i all_tvu_peers=0i tvu_peers=25118i new_push_requests_num=1595i
[2020-06-13T09:11:30.988110461Z INFO  solana_metrics::metrics] datapoint: cluster_info_stats2 retransmit_peers=596i repair_peers=170765i new_push_requests=2165i new_push_requests2=0i purge=118430i process_pull_resp=16861i filter_pull_resp=60815i process_pull_resp_count=64i pull_response_fail_insert=3293i pull_response_fail_timeout=411i pull_response_success=263i process_pull_resp_timeout=411i push_response_count=0i
[2020-06-13T09:11:30.988114369Z INFO  solana_metrics::metrics] datapoint: cluster_info_stats3 process_pull_resp_len=4317i process_pull_requests=7118i generate_pull_responses=49804i process_prune=0i process_push_message=5979i prune_received_cache=455i epoch_slots_lookup=7i epoch_slots_push=18i push_message=0i new_pull_requests=275413i mark_pull_request=2i
[2020-06-13T09:11:30.988117625Z INFO  solana_metrics::metrics] datapoint: cluster_info_stats4 skip_push_message_shred_version=0i skip_pull_response_shred_version=0i skip_pull_shred_version=0i push_message_count=475i push_message_value_count=576i new_pull_requests_count=640i
[2020-06-13T09:11:31.090505204Z WARN  solana_ledger::blockstore_processor] Too few entry ticks found in slot: 19768797
[2020-06-13T09:11:31.090524360Z WARN  solana_ledger::blockstore_processor] InvalidTickCount, slot: 19768797, entry len: 8, tick_height: 1265203057, last entry: FVkBQu925JD6sN1wAeQaiv1owoaf5FQw7DVfPLLjWHa, last_blockhash: 4nw6y5okUk2fzG5CXTVEL6DpjnstofoCgLfsghhgGv9P, shred_index: 9, slot_full: true
[2020-06-13T09:11:31.090533077Z WARN  solana_core::replay_stage] Fatal replay error in slot: 19768797, err: InvalidBlock(InvalidTickCount)
[2020-06-13T09:11:31.093121696Z INFO  solana_metrics::metrics] datapoint: replay-stage-mark_dead_slot error="error: InvalidBlock(InvalidTickCount)" slot=19768797i
[2020-06-13T09:11:31.118047240Z INFO  solana_metrics::metrics] datapoint: recv-window-insert-shreds num_shreds=1275i total_elapsed=794508i insert_lock_elapsed=1i insert_shreds_elapsed=648844i shred_recovery_elapsed=111431i chaining_elapsed=3818i commit_working_sets_elapsed=1310i write_batch_elapsed=26151i num_inserted=590i num_recovered=297i
[2020-06-13T09:11:31.126006403Z INFO  solana_core::retransmit_stage] retransmit: packets_by_slot: {
        19768806: 66,
        19768807: 120,
        19768808: 395,
        19768809: 119,
        19768810: 168,
        19768811: 167,
    }
[2020-06-13T09:11:31.126030088Z INFO  solana_core::retransmit_stage] retransmit: packets_by_source: {
        "108.49.70.159:8005": 10,
        "116.202.109.229:8005": 3,
        "116.202.196.147:6116": 10,
        "116.202.196.217:6116": 9,
        "116.202.225.7:8006": 12,
        "13.115.229.115:8006": 7,
        "136.243.65.35:8005": 10,
        "136.243.8.205:8005": 21,
        "144.91.126.158:8006": 9,
        "144.91.83.172:8006": 5,
        "150.238.34.227:8005": 6,
        "152.32.185.183:8005": 1,
        "152.32.185.183:8014": 3,
        "159.69.119.87:8005": 1,
        "159.69.58.7:8005": 25,
        "165.227.100.142:8005": 10,
        "173.231.58.226:8006": 89,
        "173.249.58.103:8006": 7,
        "178.63.94.21:8006": 10,
        "185.144.83.136:8006": 8,
        "188.40.100.189:8005": 35,
        "192.145.45.35:8005": 3,
        "193.105.188.243:8005": 7,
        "194.60.196.133:8006": 8,
        "195.171.215.154:8825": 46,
        "195.171.215.154:8828": 1,
        "195.201.109.247:8005": 11,
        "195.201.160.171:8006": 25,
        "207.180.195.178:8005": 12,
        "207.180.255.2:8006": 10,
        "217.69.11.199:8005": 1,
        "3.82.158.154:8005": 8,
        "38.135.195.85:9056": 5,
        "46.160.240.132:8005": 9,
        "46.4.19.106:8005": 10,
        "51.89.194.50:8005": 6,
        "54.229.26.131:8006": 3,
        "83.85.228.65:8005": 9,
        "85.190.150.169:8005": 118,
        "86.125.41.159:4058": 1,
        "88.198.39.210:9671": 4,
        "88.198.63.78:8005": 2,
        "88.198.66.56:8006": 7,
        "88.99.186.187:8005": 6,
        "88.99.6.218:8006": 22,
        "89.33.8.237:8006": 8,
        "89.42.234.59:8005": 11,
        "89.42.234.74:8005": 1,
        "90.188.226.112:3483": 1,
        "91.90.148.195:8006": 1,
        "93.104.210.89:8005": 2,
        "94.130.15.229:8005": 21,
        "94.130.25.240:8005": 2,
        "94.130.54.247:8006": 32,
        "95.111.244.71:8006": 9,
        "95.111.248.181:8006": 16,
        "95.216.21.42:8005": 10,
        "95.216.36.34:8005": 9,
        "95.216.46.162:8005": 36,
        "95.216.96.141:8006": 16,
        "95.217.114.150:8006": 9,
        "95.217.121.152:8006": 16,
        "95.217.121.184:8006": 19,
        "95.217.122.207:8006": 13,
        "95.217.128.244:8005": 2,
        "95.217.145.161:18026": 8,
        "95.217.181.253:8005": 9,
        "95.217.192.210:8006": 10,
        "95.217.192.222:8005": 14,
        "95.217.192.39:8006": 10,
        "95.217.193.109:8006": 13,
        "95.217.193.110:8006": 9,
        "95.217.193.34:8005": 27,
        "95.217.193.36:8005": 31,
        "95.217.195.236:8005": 13,
        "95.217.196.110:8005": 11,
        "95.217.196.126:8005": 1,
        "95.217.196.126:8008": 5,
        "95.217.208.140:8006": 5,
        "95.217.59.238:8006": 10,
    }
[2020-06-13T09:11:31.126133373Z INFO  solana_metrics::metrics] datapoint: retransmit-num_nodes count=0i
[2020-06-13T09:11:31.126138793Z INFO  solana_metrics::metrics] datapoint: retransmit-stage total_time=19547i epoch_fetch=1003i epoch_cache_update=3449i total_batches=369i total_packets=1035i retransmit_total=1774i compute_turbine=8671i repair_total=102i discard_total=14623i
[2020-06-13T09:11:31.156983725Z INFO  solana_metrics::metrics] datapoint: shred_fetch_tvu_forwards index_overrun=0i shred_count=487i slot_bad_deserialize=0i index_bad_deserialize=0i index_out_of_bounds=0i slot_out_of_range=0i duplicate_shred=0i
[2020-06-13T09:11:31.174035467Z WARN  solana_core::repair_service] Cluster completed slot: 19768797, dumping our current version and repairing
thread 'solana-repair-service' panicked at 'assertion failed: self.storage.read().unwrap().0.get(&remove_slot).is_none()', runtime/src/accounts_db.rs:1163:9
stack backtrace:
   0:     0x55a2c2859347 - backtrace::backtrace::libunwind::trace::hc1c4a1d8ad423b97
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/libunwind.rs:86
   1:     0x55a2c2859347 - backtrace::backtrace::trace_unsynchronized::h82274781060cb056
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/mod.rs:66
   2:     0x55a2c2859347 - std::sys_common::backtrace::_print_fmt::h2a45d89b653a4da8
                               at src/libstd/sys_common/backtrace.rs:78
   3:     0x55a2c2859347 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h41a0a93ab85e6aa1
                               at src/libstd/sys_common/backtrace.rs:59
   4:     0x55a2c215a71c - core::fmt::write::hdaea18585065a96d
                               at src/libcore/fmt/mod.rs:1069
   5:     0x55a2c2858994 - std::io::Write::write_fmt::h0cea70c809005252
                               at src/libstd/io/mod.rs:1504
   6:     0x55a2c2858359 - std::sys_common::backtrace::_print::hd95f9978cc145ca4
                               at src/libstd/sys_common/backtrace.rs:62
   7:     0x55a2c2858359 - std::sys_common::backtrace::print::hfb25ca2291be47d0
                               at src/libstd/sys_common/backtrace.rs:49
   8:     0x55a2c2858359 - std::panicking::default_hook::{{closure}}::h44f76cee5dc8591c
                               at src/libstd/panicking.rs:198
   9:     0x55a2c2857be7 - std::panicking::default_hook::h198e1a712910f1e6
                               at src/libstd/panicking.rs:218
  10:     0x55a2c27761a1 - solana_metrics::metrics::set_panic_hook::{{closure}}::{{closure}}::he6215f24da069541
  11:     0x55a2c2857696 - std::panicking::rust_panic_with_hook::hc0b4730bb8013f9d
                               at src/libstd/panicking.rs:515
  12:     0x55a2c2797274 - std::panicking::begin_panic::h4d3b3102d7d8a1a0
  13:     0x55a2c28088e0 - solana_runtime::bank::Bank::remove_unrooted_slot::hb2e151530827d34b
  14:     0x55a2c24e88cc - solana_core::repair_service::RepairService::run::hdf51514f4e0594b2
  15:     0x55a2c24e5524 - std::sys_common::backtrace::__rust_begin_short_backtrace::h570ffaef03a9cebf
  16:     0x55a2c24e53cf - core::ops::function::FnOnce::call_once{{vtable.shim}}::h3d991982debe4b13
  17:     0x55a2c2875185 - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h1f1fac7de7dd9691
                               at /rustc/49cae55760da0a43428eba73abcb659bb70cf2e4/src/liballoc/boxed.rs:1008
  18:     0x55a2c2875185 - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::ha031f48633a6af54
                               at /rustc/49cae55760da0a43428eba73abcb659bb70cf2e4/src/liballoc/boxed.rs:1008
  19:     0x55a2c2875185 - std::sys::unix::thread::Thread::new::thread_start::h241864caf436e182
                               at src/libstd/sys/unix/thread.rs:87
  20:     0x7f86930ff2de - start_thread
  21:     0x7f8692e30133 - __GI___clone
  22:                0x0 - <unknown>
[2020-06-13T09:11:31.212064812Z ERROR solana_metrics::metrics] datapoint: panic program="validator" thread="solana-repair-service" one=1i message="panicked at 'assertion failed: self.storage.read().unwrap().0.get(&remove_slot).is_none()', runtime/src/accounts_db.rs:1163:9" location="runtime/src/accounts_db.rs:1163:9"
[2020-06-13T09:11:31.212134233Z INFO  solana_metrics::metrics] submitting 266 points
picture svenski123
👀1

All 12 comments

https://github.com/solana-labs/solana/blob/ba36308d698b2b047bfd39e1efad65d35daa005b/runtime/src/accounts_db.rs#L1163

mvines picture mvines on 13 Jun 2020

@carllin - this looks like an assert that you added. Can you take a look please

mvines picture mvines on 13 Jun 2020
👍1

@ryoqun, looking into this area, is there anything that has changed that may have affected this code path since we last looked at it?

carllin picture carllin on 13 Jun 2020

Hmm, this looks very odd and scary.. is it ensured for remove_slot to be frozen, btw? Rather dumb guess is that others are writing accounts at the remove_slot...

ryoqun picture ryoqun on 14 Jun 2020

@ryoqun actually it's quite the opposite, remove_slot should be guaranteed to be not frozen because it was marked dead (hence the purge)

So for some context, this is where the remove_slot is passed: https://github.com/solana-labs/solana/blob/master/core/src/repair_service.rs#L382-L391

Until the call to blockstore.clear_unconfirmed_slot(*slot); which clears the dead flag from blockstore, replay_stage should not attempt to replay the slot. So this should mean nothing is touching the accounts for that slot through replay...

carllin picture carllin on 14 Jun 2020

hehe I got it to repo on a 6 node testnet by cutting up ReplayStage to make those InvalidTickCount errors happen more often: https://metrics.solana.com:3000/d/monitor-edge/cluster-telemetry-edge?orgId=2&from=now-15m&to=now&panelId=35&tab=metrics&refresh=10s&var-datasource=Solana%20Metrics%20(read-only)&var-testnet=testnet-dev-carllin&var-hostid=All

Branch: https://github.com/carllin/solana/tree/Debugging

testnet config:

1) net/gce.sh create -p testnet-dev-carllin -n 10 -c 1 -z us-west1-a -d pd-ssd --dedicated
2) net/net.sh restart -c "bench-tps=1=--tx_count 5000 --thread-batch-sleep-ms 400"

carllin picture carllin on 15 Jun 2020
🎉1

@ryoqun I think i see what's happening:

1) repair_service calls remove_unrooted_slot. It finishes the call to handle_reclaims here: https://github.com/solana-labs/solana/blob/master/runtime/src/accounts_db.rs#L1195and adds itself todead_slots`.

2) Another thread calls process_dead_slots and clears the dead_slots here: https://github.com/solana-labs/solana/blob/master/runtime/src/accounts_db.rs#L754

3) repair_service calls process_dead_slots: https://github.com/solana-labs/solana/blob/master/runtime/src/accounts_db.rs#L1199 sees an empty dead_slots, doesn't clear anything

4) Assertion fails

carllin picture carllin on 15 Jun 2020

@carllin Thanks for debugging!!! Yeah. That hypothesis is very likely...

ryoqun picture ryoqun on 15 Jun 2020

So, maybe we should just remove that assert!? Or is it required to purge any data related to remove_slot before repair_service returns from there? In that case, just wait?

ryoqun picture ryoqun on 15 Jun 2020

remove_slot should be guaranteed to be not frozen because it was marked dead (hence the purge)

I see! Thanks for explanation!

ryoqun picture ryoqun on 15 Jun 2020

yeah I think it's important that all data related to remove_slot is returned b/c when repair_service returns, replay of remove_slot can happen again. So I guess we wait?

carllin picture carllin on 15 Jun 2020
👍1

@svenski123 Thanks for reporting in detail! We believe this's fixed on master, to be released in the 1.2.3. :)

ryoqun picture ryoqun on 16 Jun 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

garious picture garious  路  10Comments
mvines picture mvines  路  7Comments
pgarg66 picture pgarg66  路  11Comments
mvines picture mvines  路  3Comments
t-nelson picture t-nelson  路  3Comments