Cardano-wallet: Missed UTXO

[Fell-x27]

Context

After a big wave of troubles in the network yesterday, some wallets are going to show zero balances. The same for me. I've opened my seq.id.sqlite db with an sqlite explorer and discovered that 'utxo' table is empty. Transactions are ok, checkpoints are ok. Everythnig is fine but UTXO.

Another users got the same by some random factor.

It doesn't recover itself. Also we don't have any endpoint, or CLI command to force utxo re-collecting.

| Information | - |
| --- | --- |
| Version | 6feffaf29488d15fbf663afd4029597116bd8cc5 |
| Platform | Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-29-generic x86_64) |
| Installation | Github binaries |

Steps to Reproduce

1. Launch cardanoWallet over Jorm by --serve.
2. Restore some wallet.
3. Get node stucked by any reason because of "bleeding shelley".
4. Stop the node, don't stop the wallet. Wallet says that "NodeIsUnreachble" as expected.
5. Start the node, wait for bootstrap ending.
6. If you're not lucky, your balance will be zero valued until you re-create the wallet.

If you are lucky, just truncate the "utxo" table by yourself to see the result. It doesn't highlight the reason of a problem but can help to handle an aftermath.

Expected behavior

1.Wallet shouldn't drop utxos if the node is offline or bootstraping.
2.User should have some abilities to force full re-syncing of the given wallet by id.

---

1. Wallet should check if transactions summary doesn't match the utxo.
2. Wallet should check if there are transactions but none of utxo.
3. CLI and REST sould provide an ability for forced full recovering for given wallet.

Actual behavior

1.Wallet drops utxos table in a database by some reason.
2.Wallet doesn't handle or recover it.
3.User can't fix it manually.

Resolution

QA

[Fell-x27]

Maybe there is a some quick and dirty way to fix it with the current version? Some changes in a database. Something with checkpoints? Or transactions?

[KtorZ]

@Fell-x27 Thanks for reporting this. May you please check the following on your wallet:

• In which state are transactions? Are they pending or in_ledger ?
• Are the transactions outgoing or incoming?
• What is your wallet current tip?

Thank you :pray:

(NOTE: I've updated your ticket and moved some of the Resolution and QA section upstream. These sections are for us to complete during the resolution of the issue.)

[Fell-x27]

All transactions are in ledger. There are both of directions also. Wallet is perfectly synced. If I delete and restore wallet, everything is ok.

I have a lot reports from my customers (https://adawallet.io). There are the same reports from Daedalus users in a community channel either.

With the same scenario: wallet wasnt used for 4-5 days, and after that it shows zero balance at a current wallet instance.

[KtorZ]

Are you able to provide some logs capturing the events?

[Fell-x27]

Im afraid, no :(

[Fell-x27]

Anyway forced resync endpoint could be very useful as new feature at least even if the reason of the problem couldnt be solved.

[KtorZ]

Well, not really. It's not _as simple as_ that. The wallet is synced and re-syncing continually. Depending on whether the node changes to different chains or not. You're not the first one reporting this but, most of those who reported this problem were actually simply facing some "expected" chain switches coupled with a slow network connection. Hence my questions about the state of your transactions.

If your wallet ended up in state where there's a discrepency between the balance and the transaction history, that's bad. I've been chasing this quite a lot these days without success. It's quite hard to reproduce.

[Fell-x27]

I understand. Its a hardly stohastic thing. I can try to find a way to reproduce, but cant promise any success.

But maybe there could be an option to reset wallet state as if was just created and launch all related processes?

[asshhssa]

Hello, I have the same problem, the zero balance is displayed to the Daedalus wallet. How can I help in solving this problem?

[KtorZ]

Hi @asshhssa , should you answer the same questions I posted earlier, that would be fantastic: https://github.com/input-output-hk/cardano-wallet/issues/1230#issuecomment-569211208

[asshhssa]

Hi @asshhssa , should you answer the same questions I posted earlier, that would be fantastic: #1230 (comment)
Test wallet Daedalus for Shelley test. There are no transactions, except for the transfer of ADA from the main wallet to the test one. Funds are delegated to the pool, ADA (devedents) come every epoch, I launch a wallet every day and often the balance is zero. The balance is fully confirmed.

[Fell-x27]

@KtorZ is there a way to export any logs from a daedalus to help you?

[KtorZ]

I'll close this one and mark it as a duplicate of #1146

Thanks for reporting @asshhssa @Fell-x27, I hope we've nailed the issue and will close #1146 once we have confidence about this.

[Fell-x27]

Well this night was a kind of forking hell, so balances has been dropped to zero again. I think its related with long bootstrapping, when node starts to write its giving up.

[KtorZ]

@Fell-x27 if you want to give a try to latest master and tell me if you still experience this 0-balance issue, that'd be fantastic.

[KtorZ]

(see: https://hydra.iohk.io/build/1543818/download/1/cardano-wallet-jormungandr-2019.12.23-linux64.tar.gz )

[Fell-x27]

Sure! I have a backup of a broken wallet and would be happy to test it!

[Fell-x27]

So...looks like this version doesnt recognize wallets from a previous one. All sqlite files are there, but

./cardano-wallet wallet list

returns nothing.
Could it be solved? Because for me that means "you should drop whole your userbase".

[KtorZ]

@Fell-x27 how did you started the server ?

Can you send me the first ~10 log lines or so ?

[Fell-x27]

Jorm do you mean?
As usual
jormungandr --genesis-block-hash 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676 --config ./config.yaml --secret --secret ./node-secret.yaml

And cardanowallet:

cardano-wallet serve --listen-address 127.0.0.1 --port 8090 --node-port 8080 --database ./wallets --sync-tolerance 300s --genesis-block-hash 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676

After reverting to prev version all wallets are visible again.

[Fell-x27]

@Fell-x27 how did you started the server ?

Can you send me the first ~10 log lines or so ?

Sure.
wallet_launch.log

[KtorZ]

Okay, looks right (note that --sync-tolerance is optional and default to 300s, same for --port and --listen-address).

What does the logs say about the database? I am expecting something in the form of:

[cardano-wallet:Info:4] [2020-01-07 14:39:04.72 UTC] Using directory: /home/ktorz/.local/share/Daedalus/itn_rewards_v1/
[cardano-wallet:Info:4] [2020-01-07 14:39:04.72 UTC] Using directory: /home/ktorz/.local/share/Daedalus/itn_rewards_v1/wallets
[...]
[cardano-wallet:Info:4] [2020-01-07 14:39:04.72 UTC] Wallet backend server starting...
[cardano-wallet:Info:20] [2020-01-07 14:39:05.73 UTC] Waiting for Jörmungandr to be ready on tcp/39185
[cardano-wallet:Info:22] [2020-01-07 14:40:30.85 UTC] Jörmungandr is ready.
[cardano-wallet:Info:22] [2020-01-07 14:40:30.87 UTC] Found existing wallet: 04244e24b3681a74483ea2e29bcddfb1611eef8e

[Fell-x27]

As you can see, it founds wallets and collect their ids, but...but returns an empty list when I ask.

[KtorZ]

Ho dear. Yes I see... we've missed a database migration. This won't happen once we are done with #1190 . But, that's clearly a bug, and not a very pleasant one.

Tracked as #1251

[KtorZ]

@Fell-x27 one thing is actually weird. You shouldn't actually face this issue between v2019-12-16 and v2019-12-23 ... are you sure that your "working" version of the wallet is v2019-12-16 (6feffaf29488d15fbf663afd4029597116bd8cc5) ?

[Fell-x27]

No, actually my "production version" is

2019.12.13 (git revision: 9280c870d39242125c11683a16d157f4b5ea26ea)

And right now I cant just migrate from. Or can I? Something mid-version as mid-step and then the last one?

[KtorZ]

Right now you can't migrate indeed. We'll have to fix this first.

[Fell-x27]

Ok, thank you!

[KtorZ]

@Fell-x27 Your migration issue will be fixed as part of #1266 .
Also, note that this is now available: https://input-output-hk.github.io/cardano-wallet/api/edge/#operation/forceResync

Cheers :gift_heart:

[Fell-x27]

Just built and tested. Migration was very long actually, but I have many customers=>many wallets.
But it works! My flushed balance is repared now!

And new rating calculation, based on desirability! Thats a real christmas :)

[erikd]

Christmas is still only once a year isn't it? :smile:

[Fell-x27]

Yeah, and it came early for the current one :)

[KtorZ]

It's a distributed Christmas with non negligible latency. We have more gifts though. Happy to hear things are fixed.
Out of curiosity, may I ask how many wallets and how long ? We've done some stress tests up to 100 wallets in parallel, but it's hard to reflect real-life data.

[Fell-x27]

I cant tell on a public a count of wallets of my users for some reasons. But I can tell you that test batch of 50 real "not fresh generated" wallets took about 40 mins on a KVM VPS with 2 xeon-based cores.

And I have some sad news. There is a new one bug :)
Looks like the wallet doesnt want to work with a private non-itn blockchain. Its not a problem for production mode and for local ITN tests, but private network is very important thing for development. I'll check it deeper and will try to reproduce it in a deterministic way today and make an issue.

[KtorZ]

Thanks, that's already some useful order of magnitude :)

Looks like the wallet doesnt want to work with a private non-itn blockchain.

@Fell-x27 this sounds weird since our whole integration test suite run on a private blockchain and this is something we do basically daily. May you describe the issue you're facing?

[Fell-x27]

Sure, later, when I'll be home and reproduce it.

[Fell-x27]

Hmmm. Today I cant reproduce it. But yesterday, when I tried to launch it over my private network, I got crashes and "ErrStartupMissingIncentiveParameters" error.

[KtorZ]

Ah. This is probably just missing configuration settings from your genesis block. On what version of Jörmungandr did you attempt to start the latest version of the backend ?

[Fell-x27]

The blockchain was generated by 0.8.3 or 0.8.4 but the node at the moment was a 0.8.6.
I've started a new private chain, so its working just as planned now.