Web3.py: Log filter unable to match entries by unindexed argument values when argument is value is not 32bytes

[avatar-lavventura]

• Web3.py version 4.2.1
• geth version: 1.8.0-unstable

---

https://web3py.readthedocs.io/en/latest/filters.html?highlight=eventFilter#module-web3.utils.filters

Says:
event_filter = mycontract.events.myEvent.createFilter(fromBlock='latest', {'filter': {'arg1':10}})

But:
https://web3py.readthedocs.io/en/latest/contracts.html?highlight=createfilter

Says:
Contract.events.<event name>.createFilter(fromBlock=block, [toBlock=block, argument_filters={"arg1": "value"}, topics=[]])

---

More detail please see: https://ethereum.stackexchange.com/q/52521/4575

I have followed web3.py guide for filtering.

Through the contract instance api:

event_filter = mycontract.events.myEvent.createFilter(fromBlock='latest', {'filter':{'arg1':10}})

---

But I get an error for this line:

myContract.events.LogJob.createFilter(fromBlock='latest', {'filter': {'arg1':10}})

My main goal is the filter events based on their arguments.

Error I am having:

myContract.events.LogJob.createFilter(fromBlock='latest', {'filter': {'arg1':10}})
                                                                 ^
SyntaxError: positional argument follows keyword argument

[Q] How could I fix this error?

---

Please also note that argument_filters seems like does not work. After I use argument_filters I do not get any error, but filter returns [] even I provide correct event's keyword (dictionary of argument name and its value).

Please see an example:

blockReadFrom = 1899201;
myFilter = eBlocBroker.events.LogJob.createFilter(fromBlock=blockReadFrom)
print(myFilter.get_all_entries())

Returns, here you can see that there is emitted event with 'storageID': 0 under AttributeDict:

[AttributeDict({'address': '0x128c9F368F12C24Cc2a4f88dCDCf3daA13C9667e', 'transactionHash': HexBytes('0x8f7c50fe276057ea90985be6d14dc7abb79ee0430c66bd161861b582b7db0c97'), 'args': AttributeDict({'desc': 'Science', 'jobKey': 'QmRsaBEGcqxQcJbBxCi1LN9iz5bDAGDWR6Hx7ZvWqgqmdR', 'index': 21, 'storageID': 0, 'clusterAddress': '0x75A4c787c5c18C587B284a904165Ff06a269B48C'}), 'transactionIndex': 0, 'event': 'LogJob', 'blockHash': HexBytes('0x9782cb281aa72defe54e0f84055a03a0b1bf6c6f21bd3d276a1252098f83c15f'), 'logIndex': 0, 'blockNumber': 1899201})]

---

blockReadFrom = 1899201;
myFilter = eBlocBroker.events.LogJob.createFilter(fromBlock=blockReadFrom, argument_filters={'storageID':0})
print(myFilter.get_all_entries())

Returns:

[]

---

And interesting point that following line returns a valid output even though 'arg1': 999 or 'arg111': 999 etc. does not exists.

myFilter = eBlocBroker.events.LogJob.createFilter(fromBlock=blockReadFrom, argument_filters={'arg1': 999})

[dylanjw]

Thanks for catching the inconsistent doc.

It looks like there may be a bug with the argument filtering. Can you post the values for the following: myFilter.filter_params and eBlocBroker.events.LogJob.abi?

[dylanjw]

And interesting point that following line returns a valid output even though 'arg1': 999 or 'arg111': 999 etc. does not exists.

Im working on a smarter event filter builder that is abi aware, that will check that the arguments exist and are filtering on valid values.

[avatar-lavventura]

>>>print(myFilter.filter_params)
{'address': ['0x128c9F368F12C24Cc2a4f88dCDCf3daA13C9667e', '0x128c9F368F12C24Cc2a4f88dCDCf3daA13C9667e'], 'toBlock': 'latest', 'fromBlock': 1899690, 'topics': ['0x711007a4be830d68a6d2a7b6546227b676bf9e7d7a0e3c248552ed72cfec2441']}
>>>print(eBlocBroker.events.LogJob.abi)
None

=> I guess if the argument name does not exists, filter returns all catched events. Shouldn't be a check around here: https://github.com/ethereum/web3.py/blob/master/web3/contract.py#L1298

[dylanjw]

And that is the myFilter object created with the following?

myFilter = eBlocBroker.events.LogJob.createFilter(fromBlock=blockReadFrom, argument_filters={'storageID':0})

Can I see the whole contract abi? eBlockBroker.abi. How is eBlocBroker instantiated?

[dylanjw]

Also, can you double check the web3 version you are using? The double address looks like an older bug, pre 4.2.1.

eg

>>> w3.version.api
'4.4.1'
>>> 

[avatar-lavventura]

I have double checked it returns 4.2.1. Should I try with a higher version?

>>>print(web3.version.api)
4.2.1

[dylanjw]

Yes, go ahead and try with a newer web3 version. I just re-checked the pull request for the bug I mentioned, and it made it in the 4.3 release. What provider are you using? If it is ganache, that would explain the issue. Ganache has a bug where it doesnt find any events when the filter is passed multiple addresses.

[avatar-lavventura]

eBlockBroker.abi is from this link.

I am not using Ganache or newer heard of it :)

I am just running through a python code.
https://gist.github.com/avatar-lavventura/869c61837178066ca398ce6952ad9800

[avatar-lavventura]

How could I install latest version of web3.py ?

$ pip install web3
$ pip freeze | grep 'web3'
web3==4.2.1

[dylanjw]

$ pip install web3 --upgrade

[avatar-lavventura]

I have done following:

$ pip install --upgrade web3
$ pip freeze | grep 'web3'
web3==4.4.1

I am still having the same bug.

---

Example:

   blockReadFrom = 1899690;
   myFilter = eBlocBroker.events.LogJob.createFilter(
       fromBlock=blockReadFrom
   )
   print(myFilter.get_all_entries())

Returns:

[AttributeDict({'args': AttributeDict({'storageID': 0, 'clusterAddress': '0x75A4c787c5c18C587B284a904165Ff06a269B48C', 'desc': 'Science', 'jobKey': 'QmRsaBEGcqxQcJbBxCi1LN9iz5bDAGDWR6Hx7ZvWqgqmdR', 'index': 22}), 'logIndex': 0, 'blockHash': HexBytes('0xb812c42309dd82727c92850407fdac7920df95d7cb6d9016afe719da47c4e0af'), 'blockNumber': 1899692, 'transactionIndex': 0, 'event': 'LogJob', 'transactionHash': HexBytes('0x183e1d0a796ecacedb9c5eeb6f84cbf9a198db1554572d6d06d87aedf4c0120b'), 'address': '0x128c9F368F12C24Cc2a4f88dCDCf3daA13C9667e'})]

---

   blockReadFrom = 1899690;
   myFilter = eBlocBroker.events.LogJob.createFilter(
       fromBlock=blockReadFrom,
       argument_filters={'desc': 'Science'}
   )
   print(myFilter.get_all_entries())

Returns: []

[dylanjw]

can you also post the raw transaction receipt that the log appears in?

web3.eth.getTransactionReceipt("0x183e1d0a796ecacedb9c5eeb6f84cbf9a198db1554572d6d06d87aedf4c0120b")

Im trying to trace why the data filter regex isnt finding a match.

[avatar-lavventura]

print(web3.eth.getTransactionReceipt("0x183e1d0a796ecacedb9c5eeb6f84cbf9a198db1554572d6d06d87aedf4c0120b"));

Gives following output:

AttributeDict({'blockNumber': 1899692, 'logsBloom': HexBytes('0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000040000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000800000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000'), 'cumulativeGasUsed': 136406, 'from': '0x75a4c787c5c18c587b284a904165ff06a269b48c', 'transactionHash': HexBytes('0x183e1d0a796ecacedb9c5eeb6f84cbf9a198db1554572d6d06d87aedf4c0120b'), 'contractAddress': None, 'root': '0x3af94d22572558b2dcd802bcf910aa36c72bd6cd5abd30dbe63a6b2fbf83c15a', 'transactionIndex': 0, 'to': '0x128c9f368f12c24cc2a4f88dcdcf3daa13c9667e', 'blockHash': HexBytes('0xb812c42309dd82727c92850407fdac7920df95d7cb6d9016afe719da47c4e0af'), 'gasUsed': 136406, 'logs': [AttributeDict({'transactionIndex': 0, 'blockNumber': 1899692, 'topics': [HexBytes('0x711007a4be830d68a6d2a7b6546227b676bf9e7d7a0e3c248552ed72cfec2441')], 'transactionHash': HexBytes('0x183e1d0a796ecacedb9c5eeb6f84cbf9a198db1554572d6d06d87aedf4c0120b'), 'address': '0x128c9F368F12C24Cc2a4f88dCDCf3daA13C9667e', 'data': '0x00000000000000000000000075a4c787c5c18c587b284a904165ff06a269b48c00000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000001600000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000002e516d52736142454763717851634a6242784369314c4e39697a3562444147445752364878375a76577167716d64520000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007536369656e636500000000000000000000000000000000000000000000000000', 'logIndex': 0, 'blockHash': HexBytes('0xb812c42309dd82727c92850407fdac7920df95d7cb6d9016afe719da47c4e0af'), 'removed': False})]})

[dylanjw]

There is a bug in the data argument filter regex, where the regex assumes that all abi encoded values will be 64 characters long, which is incorrect.

Here is the generated regex:

reg =re.compile('^0x[a-f0-9]{64}[a-f0-9]{64}[a-f0-9]{64}[a-f0-9]{64}0000000000000000000000000000000000000000000000000000000000000007536369656e636500000000000000000000000000000000000000000000000000$')

Which does not match the log data:

reg.match('0x00000000000000000000000075a4c787c5c18c587b284a904165ff06a269b48c00000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000001600000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000002e516d52736142454763717851634a6242784369314c4e39697a3562444147445752364878375a76577167716d64520000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007536369656e636500000000000000000000000000000000000000000000000000')

I am currently working on a replacement for the regex data filtering that will compare the abi decoded values directly. For the meantime, I'll work on patching the regex filter to expect the correct length for each data value.

[avatar-lavventura]

Thanks @dylanjw .

=> Why does my contract has a very large encoded abi values? Is it related to given event string's length?
=> On which web3.py version will I be able to see your fix ? What if it will be more than 512 characters based on the fix?

[dylanjw]

The PR for the regex hack was closed, since I am working on replacement for the data filters. That will probably make it into a pre-release in a couple weeks.

In the meantime, if you are able to make changes to your contract, changing your contract event arguments to be indexed will avoid the issue.

[avatar-lavventura]

=> Could you give an example for "_contract event arguments to be indexed_" or how could I do that? Sorry I did not get that part. Should I use shorter arguments names ?

Example for my event is:

    event LogJob(address clusterAddress,
         string jobKey,
         uint index,
         uint8 storageID,        
         string desc
         );

[dylanjw]

Have a look at the solidity doc regarding indexed event arguments.

You can designate up to 3 indexed arguments per event like so:

    event LogJob(address clusterAddress,
         string indexed jobKey,
         uint indexed index,
         uint8 indexed storageID,        
         string desc
         );

For dynamically sized arguments, like arrays, bytes and strings, the keccak hash of the value will be stored in the log topic list, rather than the value. Using indexed arguments allows you to quickly search the blockchain for the transaction where an event argument value was emitted to the log, but for dynamic sized arguments that are indexed you cant retrieve the value, you can only search for known values.

It means you need to plan for how your queries are going to look. Basically, only index the argument used to filter the results, not the argument results you want to retrieve.
For example, if you will have the jobKey, and want to find the storageID or desc based on the jobKey, only have the jobKey indexed.

[avatar-lavventura]

Thanks for the link and your explanation, I didn't know indexing exist on events. On my case as you said clusterAddress is the indexed argument. I will update my contract based on your guidance and see will it be fix the error I am having.

---

I was thinking to do a basic solution where is no argument_filtering but I capture all the event in between the give block number and later check with ifcondition each of them, but this seems inefficient to check all the events one-by-one.

Simple approach for that would be:

def log_loop(event_filter, poll_interval): #{
    for event in event_filter.get_all_entries():
        # Here I check the indexed argument value for each event based on required value:
        if(event.args.clusterAddress == "0x75a4c787c5c18c587b284a904165ff06a269b48c"):
             print(event)
#}

myFilter = eBlocBroker.events.LogJob.createFilter(
       fromBlock=blockReadFrom,
)
log_loop(myFilter, 2)

[dylanjw]

Make the clusterAddress indexed, and configure the filter with your target address. That would be more efficient.

This issue is getting a little bloated. Feel free to post anymore filter related questions in the gitter channel.

[pipermerriam]

Closing as I believe this is covered in https://github.com/ethereum/web3.py/issues/836

[avatar-lavventura]

@dylanjw , @pipermerriam I am sorry to ask one more question related to this.

Overall, I added the indexed keyword for the clusterAddressargument.

    event LogJob(address indexed clusterAddress,
         string indexed jobKey,
         uint indexed index,
         uint8 indexed storageID,        
         string desc
         );

---

Later should the following approach work under Web3.py or should I do something else to capture the indexed argument variable?

   blockReadFrom = 1899690;
   myFilter = eBlocBroker.events.LogJob.createFilter(
       fromBlock=blockReadFrom,
       argument_filters={'clusterAddress': '0x75a4c787c5c18c587b284a904165ff06a269b48c'}
   )
   print(myFilter.get_all_entries())

[dylanjw]

Events can only have up to 3 indexed parameter, so you will need to remove the indexed keyword from one of your event parameters.

The web3 snippet looks correct. Web3 determines if an argument filter is an indexed or non-indexed argument based on the event abi, and will either sets up the correct topic arguments for the indexed arguments, or creates a regex to filter in the case on non-indexed arguments.

[avatar-lavventura]

Thanks @dylanjw ; I confirm that now it works if I provide an existed indexed argument, but if I provide a not existing argument as dummy it returns this error: eth_abi.exceptions.EncodingTypeError: Value of type <class 'str'> cannot be encoded by AddressEncoder. Please see the opened issue: https://github.com/ethereum/web3.py/issues/970