Go-ethereum: Event filtering is slow

[jannikluhn]

System information

Geth
Version: 1.6.7-stable
Git Commit: ab5646c532292b51e319f290afccf6a44f874372
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.8.1
Operating System: linux

Issue

Getting logs from a large block range is slow, even if the filter doesn't match anything:

time curl -X POST --data '{"id":8,"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock":"0x0","toBlock":"0xf0be2", "address": "0x33990122638b9132ca29c723bdf037f1a891a70d"}]}' -H "Content-Type: application/json" http://127.0.0.1:8545 >> /dev/null

takes 27s instead of 0.02s promised in this stackexchange answer. Tested on Ropsten.

[karalabe]

There's a PR pending review/merge that replaces the entire event filtering to a completely different mechanism https://github.com/ethereum/go-ethereum/pull/14631. It should be orders of magnitude faster. Feel free to take it to a spin an report on actual numbers. I'd be curious for real world tests.

[karalabe]

Just did a fresh sync on Ropsten with the bloombits PR and tried your command:

$ time curl -s -X POST --data '{"id":8,"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock":"0x0","toBlock":"0xf0be2", "address": "0x33990122638b9132ca29c723bdf037f1a891a70d"}]}' -H "Content-Type: application/json" http://127.0.0.1:8545 >> /dev/null

real    0m0.168s
user    0m0.004s
sys 0m0.000s

$ time curl -s -X POST --data '{"id":8,"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock":"0x0","toBlock":"0xf0be2", "address": "0x33990122638b9132ca29c723bdf037f1a891a70d"}]}' -H "Content-Type: application/json" http://127.0.0.1:8545 >> /dev/null

real    0m0.012s
user    0m0.004s
sys 0m0.000s

Running the a full chain filtering for all ENS .eth events on Ropsten:

$ time curl -s -X POST --data '{"id":1,"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock":"0x0","toBlock":"latest", "address": "0xC68De5B43C3d980B0C110A77a5F78d3c4c4d63B4"}]}' http://127.0.0.1:8545 >> /dev/null

real    0m2.203s
user    0m0.008s
sys 0m0.024s

$ time curl -s -X POST --data '{"id":1,"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock":"0x0","toBlock":"latest", "address": "0xC68De5B43C3d980B0C110A77a5F78d3c4c4d63B4"}]}' http://127.0.0.1:8545 > /dev/null

real    0m0.658s
user    0m0.000s
sys 0m0.008s

[karalabe]

The PR was merged into master. Feel free to grab a fresh build and try it out yourself. Note, master does a few database upgrades, so you'll probably need to wait for that to finish (or resync from zero).

[jannikluhn]

Awesome, now it's literally a thousand times better (down to 0.03s). Thanks a lot!