Cardano-wallet: Unable to retrieve stake-pool list

[PegasusPool]

Context

Unable to retrieve stake-pool list for the last couple of hours.
v2020-01-27 has been running for months without issues.

Tried deleting the entire database used by cardano-wallet, the error still occurs after restarting. Upgrading to v2020-03-11 did not resolve the issue.

Steps to Reproduce

Start Cardano-Wallet

cardano-wallet serve --node-port 3101 --port 3102 --database db --genesis-block-hash 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676

Execute

cardano-wallet stake-pool list --port 3102

Expected behavior

Return a list of information about stake pools

Actual behaviour

Response:

Something is terribly wrong with the metrics I collected. I recorded that some blocks were produced by 607bd76be14c0b852be7f6a2795e95bfdebd290649b7b61af3a81bb2fdb18604 but the node doesn't know about this stake pool!

Possible cause

Related to the recent introduction of pool retirement feature in jormungandr?
Talked to the owner of pool 607bd76be14c0b852be7f6a2795e95bfdebd290649b7b61af3a81bb2fdb18604 and confirmed
that this pool was retired in epoch 89.

| Information | - |
| --- | --- |
| Version | v2020-03-11 & v2020-01-27 |
| Platform | Ubuntu 18.04 |
| Installation | Binary from Github Release |

[SebastienGllmt]

Pool retirements were not part of the scope for cardano-wallet Jormungandr support so there is no plan to add this as far as I know.

[KtorZ]

@PegasusPool are you able to build cardano-wallet from sources?
I can't sync with the ITN today it seems. My node is at epoch 1 and progressing VERY slowly. I got a fix which is kinda hard to test with a private blockchain so I was thinking trying that out on the actual ITN directly.

[KtorZ]

@GaborPeto thank you.

Not final fix but at least should fix the current issue if we identified it correctly.

KtorZ/1442/jormungandr-retirement-certificates

[PegasusPool]

@KtorZ

cardano-wallet-jormungandr version
2020.3.11 (git revision: 5e30fb1ebff15daf876163aa5f5206903b40cdc8)

cardano-wallet-jormungandr network information --port 3102
Ok.
{
    ...
    "sync_progress": {
        "status": "ready"
    }
    ...
}

cardano-wallet-jormungandr stake-pool list --port 3102
Something is terribly wrong with the metrics I collected. I recorded that some blocks were produced by 607bd76be14c0b852be7f6a2795e95bfdebd290649b7b61af3a81bb2fdb18604 but the node doesn't know about this stake pool!

[KtorZ]

Okay, I went down on a wrong path. Can you try again :pray: ?

[PegasusPool]

@KtorZ Bingo!
Didn't get the error, and it returned the pools list.

[PegasusPool]

Thank you and @Anviking for the blazing fast fix :pray:

[KtorZ]

Indeed! Thanks @Anviking for reacting over the weekend :)

[PegasusPool]

@KtorZ
Something's wrong.
I tried to retrieve the pool-list again and I got

cardano-wallet-jormungandr stake-pool list --port 3102
I can't list stake pools yet because I need to scan the blockchain for metrics first. I'm at 97.33%

It seems to be stuck on 97.33%, I tried deleting the db and then watched it scan from 0 back to 97.33% then stuck again. Can it be that the remaining 2.67% are some retired pools cw is trying to scan but is not able to?

[KtorZ]

Can it be that the remaining 2.67% are some retired pools cw is trying to scan but is not able to?

@PegasusPool no. Do you any additional logs? In particular from the "pools-engine" and "pools-db" tracers?

This typically happen when your node is too much ahead of the wallet server. Though, the wallet should catch-up. The wallet not catching up could typically be because the underlying worker has died (because it encountered a fatal exception... probably due to the retirement but I can't think of any reasons right now).

[PegasusPool]

Interesting. I keep it running, just checked again and it's now 97.32%

At the time of query I get this in the logs:

[cardano-wallet.api-server:Info:22447] [2020-03-14 19:41:57.38 UTC] [RequestId 30] [GET] /v2/stake-pools
[cardano-wallet.api-server:Warning:22447] [2020-03-14 19:41:59.68 UTC] [RequestId 30] 503 Service Unavailable in 2.294486014s

[KtorZ]

Is there any error log before?

[PegasusPool]

edited

Actually I found this a bit further up, not sure it's related:

Mar 14 19:29:30 bash[18259]: [cardano-wallet.pools-engine:Error:26] [2020-03-14 19:29:30.56 UTC] Recoverable error following the chain: ErrUnexpectedNetworkFailure (Link {_segments = ["api","v0","block","384eb6252faf2d16fc5807cee5b6fd46f082ab47e6a1db017f0518cdfe9247bf"], _queryParams = []}) (DecodeFailure "isolate: the decoder consumed 148 bytes which is less than the expected 214 bytes\nfragmentType 6\ngetFragment\ngetBlock" (Response {responseStatusCode = Status {statusCode = 200, statusMessage = "OK"}, responseHeaders = fromList [("content-length","914"),("content-type","application/octet-stream"),("date","Sat, 14 Mar 2020 19:29:30 GMT")], responseHttpVersion = HTTP/1.1, responseBody = "\STX\182\NUL\STX\NUL\NUL\NUL\218\NUL\NUL\NULY\NUL\NULt.\NUL\EOT\137D\193\148\176\223\217)\223!\140\ETXBE\185A\249\172\152\ETX\197&h\229\231W=\r\129U\GSz\169\DC4\172ewb\212\147\208\186\181\NULmR\225\247\132\SYN\137g\195>\169$jZ\204\180^E(\164\v\156,\224E\171\245\213\246\187_\143\198\169\152\132\229KI7e\244\177?,\138D\197\170@5\168\200\152\226\141h\237c\219P\161\197\EOT#\218u\155\137u\156bGf~F`0\203.\135m3M\229\ETX>Z\STXd\GS_\151\249&+\nP\248.\152\149s\138\CAN$\251!\230\&7\184\208\245\bH\230\253\SI\235\233k\246\&0g0\DLE\195q7;&1\194W\247\157\129@8.\138A\DLE\247\137\242b\210C\SOH\NUL\NUL\NUL\NUL\152cK\148\196\&0\bnE\148\200\ETBf\215\138\249\192\RSI\219\205\159\171>P\160\154\&6Zf\DEL\aYY`\159\201\168\167\170\163(\165\230\143=^Z\153\157?j\136\213\DC2/\232`\208\172\231k\ENQ\SI\147\217\NUL\188\187\166\244\211i\165\ETBB\154\238\223\239\245\222\DLE}\150\174brQ@\SI\201P\SUB@\160\254<\NAK\206\173\&2\139Fz\RS\SOH$\253\188\152+.\199\142\156\172{N\144\&6w[\167\167\178\&3a\134!\251\140y\ENQ\131\&8\154\223\254\203)npS\137r\154\EOT\n\220|\141\n\132\245\&5\254\DLE\154k\128\193\225P\244\233\166aj9\230\&2ThA\RS@K\145.D\243I\200\208\130Y\ETXf\210\DC1@\179\190\149\187\192|8\255\176\ACK\202\174\138\248\201\163(\"\150{\ACK\193\134\150\&9L\ETBEK\RS\232F\vM\166\178\132a\185\238\221\252pOg\166\ACK\NAK\207$\155\ESC[V\166\246[\233W\171\252\164F\180\209\198\ETXm/\243\183$]\227\FS\229\130\225\SI%I\221M0\225\r\229\229\179]\139)\145\217\213\142\239\229\230C7^\226F\158M\241\244\EOT`\150\172\155\&9y\198p~\172\241\215p\178S\212\216\160m\ENQZ>\191\235\219\r\211\207J\212N\169\198\206\233\175{T\145S\DC4\215\200O\ETX&\247C\207\164\213\&7\187\&1E\167\255\198\194)\167\&1\196\255\235\141\160Z\131\220\165^c\nW\224\168\252\190\133\239hQ\176\190^-Q\ENQT\198:\ETXsD\181[\180\187\\\152=\210k\183\t\135e\ESCyg*\160\209K8\176p\195\171\185\184\205\131\191\145Lq\244\154Z!\203\251W\162\153\227\EOTx\206\&3\159\223\242g\135\178\EOT\214\142S\170R\212\142EB\237\244h\129\230\DC1\147\NULD\204\169n\192KX\184\183d\"/\235\NUL\216\NUL\ACK\199\152\248\SI>\EM\216\201\246(\229\215\186\141\237\225.\227\&0\210,\197K#+I=;\154H\137\SO\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\SOH\NUL\255\NUL\NUL\NUL\NUL\NUL\EOT\186\240\173\200\172\132g\201nDN\r\NUL\251\131yW\172\190k2\193.\204\165\205\177c\244\DC2T\240\n\193\STX<R\139\235[\233 \165\v\169\"\248\233\201\216\137\177s\198m\188\142P\195\165d\191\NUL(\248K\t\ENQ\191\232\&0\244\243\138\160\&5\NUL\EOT?\248\196\FS\238\211\228\220\238\219\214h\249m\131a\195\198\166\227\f\SOH\NUL\235+H\232\255\DC1\193\DEL\DC1\162\187\208\SI\ACK\146\243\164\ENQ&\180\212\NAK\175`\224\a\238\181\DEL\200`p\ENQmy\194%$(q\r\137\188\143\144\t\211\128_\177\229\SO\EM s\135\&9\246m\211\221\186~\SOH"}))
Mar 14 19:29:31 bash[18259]: [cardano-wallet.pools-engine:Info:26] [2020-03-14 19:29:31.09 UTC] Applying blocks [0.530 ... 0.1603]
Mar 14 19:29:31 bash[18259]: [cardano-wallet.pools-engine:Info:26] [2020-03-14 19:29:31.09 UTC] Writing stake-distribution for epoch -0
Mar 14 19:29:31 bash[18259]: [cardano-wallet.pools-engine:Error:26] [2020-03-14 19:29:31.12 UTC] Failed to roll forward: ErrMonitorStakePoolsPoolAlreadyExists (ErrPointAlreadyExists (BlockHeader {slotId = SlotId {epochNumber = EpochNo {unEpochNo = 0}, slotNumber = SlotNo {unSlotNo = 530}}, blockHeight = Quantity {getQuantity = 1}, headerHash = Hash {getHash = "\215\159 \130\221\209*\184\129\247xo7n\225\220T\135t\239\164\130>\254\132\224(\253~r\198?"}, parentHeaderHash = Hash {getHash = "\142M*4?=\207\147\&0\173\144\&5\179\232\209h\230r\137\EOT&/,CJO\143\147N\199\182v"}}))
Mar 14 19:29:31 bash[18259]: [cardano-wallet.pools-engine:Notice:26] [2020-03-14 19:29:31.12 UTC] Worker has exited: main action is over.
Mar 14 19:29:34 bash[18259]: [cardano-wallet.api-server:Info:22352] [2020-03-14 19:29:34.49 UTC] [RequestId 21] [GET] /v2/stake-pools
Mar 14 19:29:36 bash[18259]: [cardano-wallet.api-server:Warning:22352] [2020-03-14 19:29:36.89 UTC] [RequestId 21] 503 Service Unavailable in 2.397836992s

[KtorZ]

It is. That's not cool. So, I've split the handling of stake pool retirements out of the branch / PR and I'll investigate this issue a bit later. Now the branch just fixes the error + fixes the performance calculations and ranking in the meantime.

Thanks for your help on testing this!