Cardano-wallet: Fees calculation with not null withdrawal param takes 3-10mins

[Fell-x27]

Context

Splitted from https://github.com/input-output-hk/cardano-wallet/issues/2004

When I'm trying to calculate fees with "withdrawal"!="null", it takes up to 10 minutes. 5 as average.

It doesn't depend on wallet size\UTxO amount\Reward amount. The same result for funded wallet without reward, rewarded without UTxO, and even for zero-balanced one.

Probably, this problem is related to the count of wallets because there are dozens in my case.

The node and wallet are full synced.

| Information | - |
| --- | --- |
| Version | v2020-08-03 |
| Platform | Linux |
| Installation | From binaries |

Steps to Reproduce

1. Run wallet over full-synced node.
2. Try to calculate fees with withdrawal included
3. ...
4. PROFIT!1

Expected behavior

Fees are calculated for a couple seconds or faster.

Actual behavior

It takes much much more time. Up to 10 mins. For the casual request which can't be prefetched and cached by application :(

---

Resolution

Seems likely to be the same cause as #2005.

---

QA

Pretty sure this is exactly the same as #2005, but tested once manually with the 1.18.0 node.

echo '{"payments": [{"address": "addr1qyx5s6kth3ta38qv782yajnacap97rt6jfugck36ku5n6za3veh8xatdfdra9kx57gray20m28t4mhu9l5pqaeemasdsaw4l0s", "amount": {"unit": "lovelace", "quantity": 0}}], "withdrawal": "self"}' | http post localhost:8090/v2/wallets/9d1e1558177c61d19e93aa25a9cbee774fd08e46/payment-fees

And indeed:

• 2020.8.3 (git revision: 40a8b5daa1da77480e4b818c49f2a4977aed4a71) -> 1min31s
• 2020.8.3 (git revision: fece720cb18d9676c4f1f6b0d3a69a37fab93f1c) -> 700ms

[KtorZ]

May you provide the result of the network information command please ?

[Fell-x27]

{                                                                                                                          
    "network_tip": {                                                                                                       
        "epoch_number": 209,                                                                                               
        "slot_number": 210623                                                                                              
    },                                                                                                                     
    "node_tip": {                                                                                                          
        "height": {                                                                                                        
            "quantity": 4522589,                                                                                           
            "unit": "block"                                                                                                
        },                                                                                                                 
        "epoch_number": 209,                                                                                               
        "slot_number": 210620                                                                                              
    },                                                                                                                     
    "sync_progress": {                                                                                                     
        "status": "ready"                                                                                                  
    },                                                                                                                     
    "next_epoch": {                                                                                                        
        "epoch_start_time": "2020-08-08T21:44:51Z",                                                                        
        "epoch_number": 210                                                                                                
    }                                                                                                                      
} 

[KtorZ]

I can't reproduce this locally. Are you able to provide some additional details about your request and setup? In particular:

• May you confirm which git revision you've been using? (the logic behind withdrawals has changed quite a few times since the latest release already..)

• What sort of withdrawal are you trying? Self or, from another wallet?

• What does your payment look like?

[Fell-x27]

May you confirm which git revision you've been using?

The latest release.

The logic behind withdrawals has changed quite a few times since the latest release already..

So, I have to compile the latest master and try again? Ok, will do later.

What sort of withdrawal are you trying? Self or, from another wallet?

Self

What does your payment look like?

I can't show it right now, I dont have an access to my devmachine. But it's a valid.

As I understand - the solution is trying the latest master or some stable commit between the latest release and the master. And confirm that problem is gone or not after.

[KtorZ]

I can indeed reproduce this locally with ~10 wallets in parallel. It takes _a lot_ longer than it should. I think it's an instance of the _philosophers problem_ here basically where all wallets are trying to take a lock for themselves and are all fighting for it. Which is made worse by the fee estimation which runs several hundred times for each wallet in order to get a statistically satisfactory result.

That is quite bad, we've been abusing the network layer a bit too much by sharing it through all wallets.

[Fell-x27]

I can indeed reproduce this locally with ~10 wallets in parallel.

So...In my case the count is much higher.

the fee estimation which runs several hundred times for each wallet
Ok, but:
1) This problem doesn't appear _when no reward withdrawal specified_.
2) Is it necessary to run it for every wallet at one time to estimate fee just for one of them? So what If I had only one wallet? Fee estimation would be more rough? Is that accuracy worth all these resouces? How many people use more than 1-2 wallets to get benefits from it?

Just thoughts..

[Anviking]

I tried measuring n workers of fetching the stake distribution against a mainnet node:

    worker np socket vData q = do
        withNetworkLayer nullTracer np socket vData $ \nl -> do
            forever $ do
                t <- unsafeRunExceptT  $ currentNodeTip nl
                threadDelay $ 500*1000
                let c = Coin 100
                timeit "stakeDistribution" q $ runExceptT $ stakeDistribution nl t c

There seemed to be a nice pattern until it stopped and I got different results for some reason:

Separated to two groups:
  fast if (< 5s)
  slow if (>= 5s) (tends to happen when the tip changes)

1 Worker:
Fast requests: 0.004724s (mean = 0.005471432173s, n = 118)
Slow requests: 22.007696s (mean = 21.820646s, n = 4)

2 Workers
Fast requests: 0.005821s (mean = 0.005917622505s, n = 310)
Slow requests: 28.542826s (mean = 26.225600099998s, n = 10)

3 Workers:
Fast requests: 0.004419s (mean = 0.023137280319s, n = 214)
Slow requests: 36.856492s (mean = 32.421662222221s, n = 9)

4 Workers:
Fast requests: 0.004585s (mean = 0.011072737325s, n = 198)
Slow requests: 103.087305s (mean = 98.844014624998s, n = 8)

5 workers:
Fast requests: 0.005335s (mean = 0.05364810668s, n = 1066)
Slow requests: 26.355188s (mean = 14.915916327259s, n = 55)

???! faster again?

3 workers again:
Fast requests: 0.0044s (mean = 0.012117661693s, n = 411)
Slow requests: 22.558279s (mean = 18.126501727268s, n = 22)

[Anviking]

Btw:
This code https://github.com/input-output-hk/cardano-wallet/blob/a7670ed340427f3f86dba192325eb7abf67a4ca9/lib/core/src/Cardano/Wallet/Api/Server.hs#L1359-L1365
seems to fit nicely with:

This problem doesn't appear when no reward withdrawal specified.

We only fetch the account balance over LSQ with the withdrawal specified, and this is when it is slow.

[Anviking]

Is it necessary to run it for every wallet at one time to estimate fee just for one of them? So what If I had only one wallet? Fee estimation would be more rough? Is that accuracy worth all these resouces? How many people use more than 1-2 wallets to get benefits from it?

Which is made worse by the fee estimation which runs several hundred times for each wallet in order to get a statistically satisfactory result.

No, the fee calculation itself shouldn't be affected by the number of wallets you have.

[piotr-iohk]

lgtm.

Fee estimation with self and 19 wallets fully synced + 10 wallets syncing concurrently takes ~3s.

Tested on testnet and mc4.

[Fell-x27]

@piotr-iohk , yep, because it was solved with #2039 branch that is already merged into master.