Ethers.js: Add Block reorg policy

[gakonst]

It would be nice if ethers.js supported a way to handle blocks that got reorged and their events. A library that kind of does it is this https://github.com/hillstreetlabs/ethstream

Dagger instead waits for 12 blocks by default before it exposes events https://github.com/maticnetwork/eth-dagger.js/blob/master/src/wooden-dagger.js#L94

[ricmoo]

This is an interesting idea. We could add a provider event for providers.on(“orphan”, ...).

For now though I may just put together a quick cookbook example, to iron out the wrinkles and figure out what info people actually end up needing/experimenting.

[marcelomorgado]

I'm not sure but maybe this comment could help on this thread.
I've written some tests to simulate reorg/orphan scenarios using ganache:

1- block excluded

• create blockchain snapshot
• send transaction
• mine some blocks
• tx get confirmed
• revert to previous blockchain snapshot
• tx becomes unconfirmed

inside the listener, I've identified the condition: lastConfirmations > 0 && currentConfirmations == 0 that I've used to identify a reorg.

Note: confirmations == 0 <=> receipt == null

2- tx confirmations decreases

• send transaction
• mine some blocks (5)
• tx get confirmed
• create blockchain snapshot
• mine more blocks (20)
• revert to previous blockchain snapshot
• refresh provider *
• mine more blocks

inside the listener, I've identified the condition: lastConfirmations > currentConfirmations that I've used to identify a reorg.

Here is my question: I've only reach the condition above when I've instantate a new provider, otherwise the confirmations getted continues from the last confirmation before blockchain reversion.
That is a expected behavior from provider?

Thanks!

[jtakalai]

Problem with web3's on("changed") and "removed: true" type of API is that it doesn't really offer any kind of support and leaves the reorg handling completely to the dev. And mostly no "MVP" really handles reorgs very gracefully (because we all like to pretend Ethereum sweeps all that "blockchainy stuff" under the rug). ethstream's idea of manually specifying the "confirmation number" and supplying events for confirmed events is a step to the right direction.

An app developer who wants their app to run completely reliably on the side of a blockchain would have to build a state snapshotting system, and upon reorg revert to previous good checkpoint and replay from there. Not sure how much a library can really help there. I think it's mostly documentation: write as simple as possible boilerplate/example code that handles reorgs gracefully by reverting state then requesting event replay since the checkpoint.

In replay, the library might be able to help: provider could enable triggering the registered listeners using past events between given block numbers.

@marcelomorgado's idea of adding reorgs to testing is definitely a good one, and I hope that ends up supported in testing frameworks.

[ricmoo]

The v5 branch makes adding Ancillary packages much easier, and re-orgs are a fairly complex thing to handle in general.

The re-org policies I have added to v5 are:

• a block has been orphaned (a block is no longer in the longest chain)
• a transaction has been orphaned; i.e. no block in the longest chain contains the transaction any more; since most re-orgs still result in the same transactions being mined, usually this is what people will actually care about
• the order of two specified transactions has been changed; this can be an issue for front-running sensitive applications

Anything more complex can easily be added using ancillary libraries, but this should cover most cases, and so long as some number of confirmations if reached without triggering these, most applications should be able to "keep on truckin'".

I would love to hear other ideas, or proposed API though. This was just something I dreamt up and haven't actually used it in practice yet.

[pstuermlinger]

That sounds great, thank you!
Just out of curiosity: how do you test those situations? Are you mocking a rpc source or is it possible to simulate such situations with geth or parity?

[ricmoo]

Oh, they can all be detected using the normal API. :)

1. If getBlock(block.number) returns a different blockhash, the block is orphaned
2. If getTransaction(hash).blockhash fails 1, the transaction has been orphaned
3. If the A.blockNumber < B.blockNumber || (A.blockNumber === B.blockNumber && A.transactionIndex < B.transactionIndex), the transaction order has failed A after B.

I think. I still need to do some experimentation on various backends.

[pstuermlinger]

Sure! But, are you waiting until it happens on a public testnet or do you have a private one for this? For the latter case I have no idea how one could provoke a reorg for reliable testing during development.

[ricmoo]

Oh, for testing I will use the evm module I am working on to simulate these events. That will still be a bit though. Or, like you suggested, a simple RPC mock would be simple to put together. :)

[jtakalai]

I think Ganache quite well supports any kind of internal blockchain state manipulation. It would be great to integrate (or at least ensure and maintain compatibility) to Ganache, since that is currently the popular Ethereum simulator for testing; likely due to malleability compared to e.g. spinning a Parity instance for each test.

And reason is obvious: in tests, you want to be able to quickly build a harness and mock things; which for anything that ALSO should run real deployments (like Parity) merely adds to vulnerable surface, and hence has no reason to really be well supported. A framework had better support both, since for developers, thorough testing is (or should be) just as important as shipping.

[ricmoo]

The problem is Ganache has historically not been a very reliable simulation of what a blockchain does. I have not used it recently, but it has had a lot of bugs and non-standard behaviour. I believe it is getting better though.

I agree it is more work to set up a Parity node as a dev test, but it far more accurately emulates a real blockchain (since it is a real blockchain), and during development, I agree, agile turn-around time is important, but for integration, in a contract I plan to have people potentially tie up non-trivial assets, I want to make sure the tests are being executed correctly. Fast, but potentially incorrectly run test cases do not install me with confidence in my own code. :)

That said, Ganache is fully supported by ethers, so long as you are relying on standard behaviour, and I was told that there are flags you can pass to Ganache to operate in standards-compliant mode. But with the EVM package coming out, it will be easier to do many of the Ganache-like things. :)