Bigchaindb: Backlog transactions can go through block pipeline many times

[ssadler]

Resolves #1140, Related to #1150

1. Tx goes into backlog
2. After 120 second, Tx is reassigned.
3. Changelog now includes same transaction twice, so it is considered many times for inclusion.
4. In the case of TRANSFER, double spend is detected. In the case of CREATE, the transaction is erroneously included multiple times.

Proposal:

• Don't listen to backlog changefeed
• When a transaction is picked up for inclusion, refresh the timestamp in the backlog.

[ttmc]

In your proposal, I think your idea was that rather than listening to the backlog changefeed, nodes could just query the backlog for assigned transactions on a regular basis. The node could order the transactions by 'assignment_timestamp'.

Refreshing the 'assignment_timestamp' when the assigned node picks it up is a nice idea, but will it prevent a transaction from ending up in multiple blocks? I'm not sure:

The assigned node has an internal queue of transactions lined up for consideration (for inclusion in blocks). How long does it take that node to go through all the transactions in that queue? If it's more than 120 seconds, then the transaction might get reassigned to another node and be considered by that other node as well.

I suppose it depends on how often the assigned node queries the backlog, and what it does with the 'assignment_timestamp' of transactions that it has already got in its internal queue. Does it reset them all to zero? That seems like a lot of extra database writes.

This is feeling a bit too complicated to reason about fully.

[ttmc]

Another proposal

First, keep the idea of querying the backlog regularly rather than listening to the backlog changelog.

Currently, the stale transaction pipeline reassigns stale transactions (in the backlog table). What if it _deleted_ them instead?

That would solve this issue, I think.

It would also help keep the backlog size bounded. That matters because if the backlog gets really full, transactions could spend a _long_ time there. Why? When the assigned node picks up all outstanding transactions assigned to it, it can only fill its internal queue: the older transactions will just get dropped until the next pickup time.

If a user posts a transaction to a BigchainDB cluster and they see it circulating around in the backlog for 10 minutes, then that's a bad user experience. Better to delete the transaction, and train users to know that if they wait more than M seconds, then their transaction is probably deleted.

120 seconds actually seems too long, but that's just me. It's a BigchainDB Server setting anyhow (currently called backlog_reassign_delay --- it would have to be renamed).

[ttmc]

It would be _nice_ to be able to tell the user when a transaction has just been deleted-because-stale, but that's trickier than it sounds: what if the transaction _was_ deleted-because-stale but it was also picked up at the last second, and is about to be written, inside a new block, to the bigchain table? Right now, there's no way to see if a transaction is in a node's block-building pipeline. Fixing that is beyond the scope of this issue.

[ttmc]

BigchainDB 2.0 is coming out soon and while there is still a mempool of transactions not yet in blocks, once a transaction leaves a mempool, it doesn't go back. It's either gone from that node or in the block (which is under-construction for a short while, then later committed).

Anyway, there is no possibility of transactions circulating between various undecided states forever. That's a thing of the past. I'm closing this issue.