Walletwasabi: CoinJoin transactions stopped being broadcasted (txn-mempool-conflict) yet the backend thinks they were broadcasted

[ghost]

Hi, last successfully broadcasted CoinJoin transaction was 616966f7e7bdbf9a229f3cc8e07e8c4cc512486f04ef15168635519dcd1f56d3 16 hours ago. Later on there was one more (591b7a721a30d482ce9d49f113e06adecc9139aff3f2d975ce8306e9915ffd0d) but that is now marked as "conflicted" in my bitcoind (later became rejected by nodes with bad-txns-inputs-missingorspent).
Every CoinJoin transaction after that up until this moment was seemingly broadcasted by the backend but rejected by other nodes, however the backend incorrectly thinks all these transactions were broadcasted successfully. Because they in fact weren't, the WasabiWallet re-enqueues the coins which are then consequently rejected by the backend claiming they were already spent and then WasabiWallet dequeues them and mark them as "spent". I have accumulated 16 fakely "spent" coins so far.

Last such a transaction is e1aa57fe424506e30adb80edd27661c00d6d7732af76e2093980eaec42f8cf94. When I try to broadcast this (or any such other) transaction myself I get txn-mempool-conflict. But when I try to broadcast any of these transaction via WasabiWallet's transaction broadcaster or directly via the backend API I get "Transaction is successfully broadcasted."

It seems the backend isn't detecting txn-mempool-conflict while broadcasting and thus thinks the CoinJoins are successful while in fact they aren't.

It's rather interesting that there is no mayhem yet. There must be dozens, if not hundreds, of people having their precious coins marked as "spent" and yet so far only this guy noticed something is wrong: https://www.reddit.com/r/WasabiWallet/comments/hwm1dd/my_coins_are_showing_as_spent_when_they_are_no.

[ghost]

So after accumulating 20 coins falsely marked as "spent", 2 hours ago Wasabi finally made a successful CoinJoin 26555ecb9e6949f8e57600c32b7b7a0ee6a3db0c1d0d5c6c48b9104febd83be5, yay.

Then it made what might seem as another CoinJoin (944edd4f347f107c5d91e5d6a5c9185edc9fd6fa3288897d2e3dad977f276e78) 20 minutes ago but this one is being rejected by nodes with bad-txns-inputs-missingorspent. No wonder, some people probably freaked out seeing their coins "spent" and moved them, of which Wasabi backend seems to be oblivious.

[molnard]

Yesterday the server was updated to knots 0.20.0, obviously, something went wrong. I am on the problem. Thanks for the report!

[ghost]

You are on it, cool!

Maybe knots itself is not the problem, it seems as if your knots wasn't connected to other nodes or somehow disconnected from the network consensus, or if you have only the default 8 connections maybe some or all of them are weird or rogue nodes. Either way it might be beneficial to increase the number of connected peers, my nodes are connected to nearly 200 peers simultaneously and everything is peachy.

[molnard]

Seemingly the confirmation on blocks and restarting Wasabi fixing the case of spent coins but the problem will keep coming until we figure out the fix on the backend.

[ghost]

There are still many txn-mempool-conflict rejected transactions in your backend's mempool, maybe you could clear your mempool.

CoinJoin transaction 26555ecb9e6949f8e57600c32b7b7a0ee6a3db0c1d0d5c6c48b9104febd83be5 is the most recent correct one, that was 4 hours ago. All the transactions after that are rejected.

[ghost]

Just now Wasabi produced another two CoinJoin transactions f04d757279967bf6c80af5909d280fb443bbab63a0857628fe74a85c38c19585 and 09e9cd4bbdd409db16e08a7271d3fefcfc4570976e42657c1730f7a0829918cc. The backend thinks they are both ok, while to the whole rest of the world they are bad-txns-inputs-missingorspent rejected. You may as well just stop it, you are only filling your mempool with trash.

//EDIT: Now 8 previously rejected transactions got confirmed and the 26555ecb9e6949f8e57600c32b7b7a0ee6a3db0c1d0d5c6c48b9104febd83be5 got confirmed too, there is no unconfirmed CoinJoin left in your backend.

[nopara73]

Same as this: https://github.com/zkSNACKs/WalletWasabi/issues/3699#issuecomment-661047910

It may take a few days until the server's mempool gets in sync with others' mempools.

Fixed by the deployment of https://github.com/zkSNACKs/WalletWasabi/pull/4026 yesterday.

[molnard]

We figured out the problem. Previous Bitcoin Knots version let some tx-s into it's mempool those were evaluated as valid but were not aligned with the other part of the network. So it's mempool was different compared to the rest of the network nodes. Yesterday we upgraded to 0.20.0 and those tx-s will be cleaned out with time.

[molnard]

https://www.reddit.com/r/WasabiWallet/comments/hwzh04/resolved_on_unconfirmed_historybalance/

[ghost]

Another two transactions f566150bb858061cdc7acbea88efb5b0f7705cc8060763a4cf2f696a9dff81e7 and 3b2b5ffa4bf156395a87291900d8af8135e8c7688cd31a729821bd4a7a9c53d0, both rejected by the bitcoin network for txn-mempool-conflict and bad-txns-inputs-missingorspent.

Instead of clearing wrong mempool and solving the problem immediately your solution is: let's keep producing rejected CoinJoin transactions and just wait for a few days until the server's mempool "gets in sync" with others' mempools.

I will not bother with you ever again.

[nopara73]

Clearing the whole mempool would result in huge issues with orders of magnitude more inconsistencies. There's really nothing else to do, but wait for confirmations as new unconfirmed transaction chains will be accepted into the mempool correctly from now on. See the post @molnard noted, we traced back more than one transactions since then and always found that it's the result of the previous full node version's inconsistencies just like in the post.

[Quantris]

I commented on https://www.reddit.com/r/WasabiWallet/comments/hwzh04/resolved_on_unconfirmed_historybalance/ also.

It seems like this was fixed (numerous coinjoins succeeded over the weekend) but I noticed it seems to be happening for recent coinjoins again. I saw there was an edit to the reddit post and wonder if something was changed that reintroduced the problem? Or did we just get lucky over the weekend and still need to wait longer here?

One example is a4966cfffb9ed9d9c1fbd58dc890584b7d9f748e2afba645fa4b377422da45e0 which my full node does not know about. Even when I clear Wasabi's transactions.dat and restart, the backend eventually tells Wasabi that this txid has been broadcast.

EDIT: this might be better now; those transactions don't seem to reappear anymore after clearing transactions.dat & restart. Please do look into recent coinjoin broadcasts to evaluate (I had turned Wasabi off so I don't have data about recent ones).

[nopara73]

We're having issues with server load as we have a huge spike in userbase that can cause a lot of problems, so I cannot tell if we're (I can confirm the latest mempool inconsistencies) experiencing something new or not.