Substrate: Transaction Payment endpoint returning incorrect information

[joepetrowski]

I get a mismatch when I try to reconcile an account after making a transaction. Runtime info:

• system.version() = 0.7.32-fa1536c9-x86_64-linux-gnu
• Runtime: Polkadot 1007

Steps to reproduce:

1. Make a transaction
2. Query the address's balance the block before and block after the transaction
3. Query the transaction's fee info
4. balance_after - transfer_value - fee should equal balance_before

Is this an error in the return value of queryInfo or incorrect usage/interpretation of its result?

Example:

Balance at block 81,116 (api.query.balances.account.at(hash, address);)

{'at': {'hash': '0x42771a692550024b93c883e47710956e86841bee57e4f9e8a7f04f5db4959081',
  'height': '81116'},
 'nonce': '6',
 'free': '93998498721632',
 'reserved': '0',
 'miscFrozen': '0',
 'feeFrozen': '0',
 'locks': []}

Block 81,8117 (api.rpc.payment.queryInfo(extrinsic.toHex(), block_hash);)

{'number': '81117',
 'hash': '0x28143e872c02ba739f92a5da424c42e913dda23e11fbf6f046c04ab825fa8e1b',
 'parentHash': '0x42771a692550024b93c883e47710956e86841bee57e4f9e8a7f04f5db4959081',
 'stateRoot': '0x039bdd3aac029cd1083a773a4e3fb27a4b2a79cf6b5f23b2bbab7f73444820fb',
 'extrinsicsRoot': '0xbfa81a0a00dbfa61b25a867fe9497e3f9f2d44355cd2470a8e723c430c600992',
 'logs': [{ ... }],
 'extrinsics': [{'method': 'timestamp.set', ... },
  {'method': 'finalityTracker.finalHint', ... },
  {'method': 'parachains.setHeads', ... },
  {'method': 'balances.transferKeepAlive',
   'signature': {'signature': '0x1f5e322c1adb0d3d7b9941103278918bdc97cfc03c83fb18781a652100fdfe7a3206b7dd8fbdb60628952f5c4f52e82a48f87284c394d0a410c4bd52cb3f7f04',
    'signer': '15wAmQvSSiAK6Z53MT2cQVHXC8Z2et9GojXeVKnGZdRpwPvp'},
   'nonce': '6',
   'args': ['13iSQm7iyDjoTo3HndhCzpQztAxkNpB1SyRkEuucmAShcApQ',
    '1000000000000'],
   'tip': '0',
   'hash': '0xeeb6d3d5e458dd843d3d5076d5f648bc18e457b8ad7c002c19f421add007317d',
   'info': {'weight': '150000000',
    'class': 'Normal',
    'partialFee': '156060973'},
   'events': [{'method': 'balances.Transfer',
     'data': ['15wAmQvSSiAK6Z53MT2cQVHXC8Z2et9GojXeVKnGZdRpwPvp',
      '13iSQm7iyDjoTo3HndhCzpQztAxkNpB1SyRkEuucmAShcApQ',
      '1000000000000']},
    {'method': 'treasury.Deposit', 'data': ['124851124']},
    {'method': 'balances.Deposit',
     'data': ['16CeUh9xzKaM7CBjJQvyDEi6KLmijx6tyqiQ1A7Lz1RNSnNs', '31212781']},
    {'method': 'system.ExtrinsicSuccess',
     'data': [{'weight': '150000000', 'class': 'Normal', 'paysFee': True}]}],
   'success': True}]}

Balance at block 81,118

{'at': {'hash': '0x3266cd2b286858a9180308761852135be406ff03c7f1a53dbb173f450d8022b7',
  'height': '81118'},
 'nonce': '7',
 'free': '92998342657727',
 'reserved': '0',
 'miscFrozen': '0',
 'feeFrozen': '0',
 'locks': []}

Results

pre_balance = 93998498721632 # balance['free']
transfer_value = 1000000000000 # extrinsic['args'][1]
fee = 156060973 # extrinsic['info']['partialFee']
post_balance = 92998342657727 # balance['free']

expected = pre_balance - transfer_value - fee

Actual Balance:   92998342657727
Expected Balance: 92998342660659
              ------------------
Difference:                -2932

This results in a difference of -2932.

Notably, if I replace the fee from queryInfo with the sum of the treasury.Deposit and balances.Deposit events (fee = 124851124 + 31212781), it reconciles.

CC @kianenigma

[athei]

tl;dr

You are using the wrong API.

Long Answer

The queryInfo rpc call is not meant to retrieve the fee that _was_ induced by an extrinsic. It is a tool that can be used to estimate the costs of an extrinsic ahead of dispatch. It only looks at the length in bytes and weight of an extrinsic not at any events that were generated. After all the field is called partialFee. It does not account for:

• Tip as this is not included in the extrinsic but in the signature
• Post Dispatch weight correction
• Fees induced by the logic of the call itself

I think the tip is what you are missing here. I would say that this works as intended.

Determining the actual fee of an extrinsic is not trivial. Sure you can skim through all treasury events but are those really fees? I mean there could be other reasons why this turing complete code would send something to the treasury. And you also want to add other events which might resulted from removing money. Maybe the code even removes money without triggering an event.

What you want is a way to look at an extrinsic and somehow now how much money is removed by executing it. There is no way to do that in a general way other than checking the balance before and after executing it. I think you are running into Rice's theorem here.

That said you can define the bespoke treasure event as transaction payment events and add an API to the JS lib that just sums up those amounts for an extrinsic and use that as the "transaction fee" shown to the user.

[kianenigma]

@athei thanks for helping out with this!

I think the tip is what you are missing here. I would say that this works as intended.

Is it? I as far as I knew all of Joe's examples did not have any tip. And we already noted it off-band.

The queryInfo rpc call is not meant to retrieve the fee that was induced by an extrinsic. It is a tool that can be used to estimate the costs of an extrinsic ahead of dispatch

I both agree and disagree. Yet, this is not complete. and it does not account for all things you named. But, if for instance, we know that a particular dispatch does NOT have any deduction logic within itself, and it does not have weight refund, and ofc there are no tips attached to the transaction, then @joepetrowski's basic assumption must be correct:

balance_after_tx = balance_before_tx - query_info(tx)

and since the tx here is a transfer:

balance_after_tx = balance_before_tx - (query_info(tx) + transfer_amount)

Perhaps you can investigate if the transfer and tranferkeepalice that Joe used meet this situation or not? I don't think either of them do weight refund. But if that is the case, I'd be very happy and indeed that is the reason. Although this will have somewhat bad consequence for exchanges.

[kianenigma]

Actually, I do recall that transfer _must_ deduct some extra money of the receiver account was non-existent at the time, and the query_info() cannot know this. Although I can't yet find it in the code.

[athei]

My statement is of course only true for the general case. Of course you can build such a reconcile logic for a specific dispatch.

However, there is no tip included. So there is either a bug or some logic in the transfer that removes balance (as you said). I will look into that.

[joepetrowski]

Tips

We do indeed account for the tip separately. It is known that this does not factor in tip, but we easily account for that elsewhere. Tips don't need to be considered at all for this issue.

What you want is a way to look at an extrinsic and somehow now how much money is removed by executing it. There is no way to do that in a general way other than checking the balance before and after executing it.

This is wrong. Blockchains must have deterministic logic. The problem statement does not say that you need to know how much is deducted _pre-dispatch_ (in fact the opposite). The amount you take up-front is deterministic and then any changes based on function logic (refunds or further deductions) must be accounted for with an event. That's the purpose of events - to relay information from the chain's logic to the outside world. Account balances shouldn't change without some way of knowing why, either by querying the transaction fee or having the logic emit an event to notify you. As an example, this is why we emit Bonded from the Staking pallet; you don't know pre-dispatch how many tokens will get locked.

Actually, I do recall that transfer must deduct some extra money of the receiver account was non-existent at the time, and the query_info() cannot know this. Although I can't yet find it in the code.

This used to be true but was removed a few months ago. AFAIK no current functions deduct additional fees based on their logic path.

An acceptable solution (not sure if it's possible) would be to add the actual fee to the DispatchInfo struct so that it emits as part of the System.ExtrinsicSuccess or System.ExtrinsicFailed event:

{'method': 'system.ExtrinsicSuccess',
     'data': [{
         'weight': '150000000', 
         'class': 'Normal', 
         'paysFee': True,
         'fee': '156063905' // <-- add this
     }]
}

[athei]

This is wrong. Blockchains must have deterministic logic. The problem statement does not say that you need to know how much is deducted _pre-dispatch_ (in fact the opposite). The amount you take up-front is deterministic and then any changes based on function logic (refunds or further deductions) must be accounted for with an event. That's the purpose of events - to relay information from the chain's logic to the outside world. Account balances shouldn't change without some way of knowing why, either by querying the transaction fee or having the logic emit an event to notify you. As an example, this is why we emit Bonded from the Staking pallet; you don't know pre-dispatch how many tokens will get locked.

But you cannot determine, in the general case, which are the events you should sum up. But in the specific case for balance transfer you should be able to derive that from the events but only because you know what the code does. And you successfully did that.

That said, in the specific case for balance transfer the queryInfo(which is only pre-dispatch logic) must be able to predict the right amount as it is not using any special logic. I have no idea how this difference comes about. I am trying to reproduce it on substrate master.

Regarding adding the fee to the success event: The weight returned there is not even the corrected weight (post dispatch). This might be an oversight. Adding the fee might be helpful. But it won't be the complete fee (no tip, no special logic) as it would only be derived from the weight.

[joepetrowski]

But you cannot determine, in the general case, which are the events you should sum up. But in the specific case for balance transfer you should be able to derive that from the events but only because you know what the code does.

I see what you're saying. Yes, I had thought of adding the treasury deposit and balances deposit events rather than using query_info, but you are correct that it's not a general solution. But what I'm saying is, some method needs to clearly mark that some amount was deducted for fees. So you could add an extra param to those Deposit events like enum Reason { Fee, Donation, Other } so that we can find the events that correspond to the fee.

Regarding adding the fee to the success event: The weight returned there is not even the corrected weight (post dispatch). This might be an oversight. Adding the fee might be helpful. But it won't be the complete fee (no tip, no special logic) as it would only be derived from the weight.

As long as the final fee can be calculated then it's OK. So you might have total pre-dispatch fee in one event, and then another event for 'fee refund'. Then the actual fee equals pre-fee - refund.

[athei]

The bug is in the javascript. When calling api.rpc.payment.queryInfo you must call it at the predecessor block of the one that contains the extrinsic you want to lookup the DispatchInfo for. That is because the state at block n - 1 is used to calculate the costs of an extrinsic in block n. In between the blocks the NextFeeMultiplieris adjusted. That explains the difference.

So my point still stands: You are using the wrong API. However, the correct API does not exist yet but that is another issue. We should have a separate "FeesPayed" event that also takes into consideration the post dispatch weight correction.

[joepetrowski]

This particular issue was solved as long as there is no post-dispatch correction. Closing this in favor of https://github.com/paritytech/substrate/issues/6022