Web3.py: Filters that don't rely on server state

[pipermerriam]

What was wrong?

Filters suck.

• Sometimes the server loses the filter_id
• You don't really have any clarity to what is actually happening under the hood on the server.
• You can't use filters with infura
• They don't work if you're load balanced across multiple nodes.

How can it be fixed?

I propose we implement a new opt-in middleware to handle log filters locally.

Consider the following generator.

from cytoolz import (
    merge,
)


def log_filter(web3, filter_params):
    # TODO: handle pending/latest/earliest and cases where these are not included in the filter params.
    from_block = filter_params['fromBlock']
    to_block = filter_params['toBlock']

    # TODO: handle dynamic block ranges to enforce reasonable request sizes.
    # TODO: handle things like `latest` and query what the latest block number is from the last block number that we checked.
    for block_number in range(from_block, to_block + 1):
        params = merge(filter_params, {'fromBlock': block_number, 'toBlock': block_number})
        logs = web3.eth.getLogs(params)
        yield from logs

This log_filter generator can be used under the hood in place of using the node based filters to handle all filter logic locally, only relying on the node for eth_getLogs which is stateless.

The middleware would then resemble the following.

def filter_middleware(make_request, web3):
    filters = {}
    filter_id_counter = itertools.count()

    def middleware(method, params):
        if method == 'eth_newFilter':
            filter_id = next(filter_id_counter)
            filter = log_filter(web3, params[0])
            filters[filter_id] = filter
            return {'result': filter_id}
        elif method == 'eth_getFilterChanges' or method == 'eth_getFilterLogs':
            # do appropriate logic to return logs.
        else:
            return make_request(method, params)

The middleware creates and tracks these generator functions locally, retrieving them when requests are made to retrieve the logs for a given filter and using them to return the appropriate log entries.
Each time a new filter is created, the filter backend will create one of these generators for it and keep track of it internally. Then, any calls to eth_getFilterChanges can just return the next(...) for the appropriate filter generator. We should be able to do something similar with eth_getFilterLogs by re-generating the original filter generator and then returning all of the values up-to-the-current state of the generator.

[pipermerriam]

I just realized this fits perfectly into the middleware framework. No need for a new API, just a new middleware.

[carver]

It's worth taking a look at the issues that the following project has with filters in v3: https://github.com/makerdao/pymaker/blob/7f2047b87d6982ce3cfadae07b8fad0026896202/pymaker/lifecycle.py

I think this proposal does a great job of addressing them all, but a thorough look before implementing this approach would be wise.

[lpassos]

Any idea when this could get in?

[pipermerriam]

I suspect it will be part of the next major release cycle but there's no firm timeline for when that will be.

[dylanjw]

Im starting on this.

[jasonrhaas]

Looks like the pull request is about to be merged in. Excited to try this out in the next release! I implemented something using filters only to find out that Infura doesn't support them :(

[dylanjw]

732 Was merged. @jasonrhaas Give it a try, and let me know what you think!

[jasonrhaas]

@dylanjw @carver @pipermerriam Any chance of getting this feature into the new web3py release? I am trying to use filters and have found them horribly unreliable and inconsistent. Hoping that this middleware is a good alternative.

[carver]

Sure, I should be able to get it out tomorrow morning. Note that it would be a v5 alpha release, so no guarantees about a stable API for a little while.