Web3.py: (feature request?) parse transaction receipts for event log data output?

[drandreaskrueger]

How to extract event logged output?

Please have a look first how it is working in javascript/truffle:
logArgs.contractId
txLoggedArgs

I really want to work with Python and not JS - and apart from some minor hickups it is looking good.

But now, I need to get a specific value logArgs.contractId from an event log after a contract call, and I am stuck.
Because when I try the same as above with the py transaction receipts that I get via

rec = web3py.eth.getTransactionReceipt(tx)

then this of course fails:

logargs = rec.logs[0].args

with

 AttributeError: 'AttributeDict' object has no attribute 'args'

because it simply does not exist - and that seems different from javascript/truffle?

Is it because the event output args are still all mixed up in the rec.logs [0].topics field (or in the rec.logs [0].data field) ?

Example (complete receipt below).

'data': '0x00000000000000000000000000000000000000000000000000038d7ea4c68000ec099eeb7c6385269c410da8eb2e028bf832c8418d731bf99f987e9d9357abe9000000000000000000000000000000000000000000000000000000005a0c6ae0'
'topics': ['0xa0085ce7b3f5d49ff7ac7b4f8caaf2f75147d0228c3811a6d3d41eb2029fe122', '0xa4162351c79a5ed77bfc705394785c93842b6fc2e55d2b302c86ee55540c5b24', '0x000000000000000000000000c62d9f5092737fa7689579e279a3a1dd5543fb9e', '0x00000000000000000000000025f579697ce3fe1cf122fc5ffae9f1d83f16c676'],

Wouldn't the ABI now help, to take those outputs apart?

Is there a helper function for that in web3py? which one?

If not, wouldn't it be useful to have? Or ... which external tool can I use now?

Or am I thinking this in a completely twisted way?
Is there a more direct way to access the answered contractId after a newContract call (solidity, abi) ?

because the web3.eth.contract(...).newContract(parameters, transact={...}) works well - but it returns only a a transaction hash. How does truffle actually manage it differently?

Thanks a lot.

---

{'blockHash': '0x9a120c06c6950f1a3fa29ebd2bfdd349652cda382087d3e402ccb0b56e87ac04',
 'blockNumber': 2076033,
 'contractAddress': None,
 'cumulativeGasUsed': 146500,
 'from': '0xc62d9f5092737fa7689579e279a3a1dd5543fb9e',
 'gasUsed': 146500,
 'logs': [AttributeDict({'removed': False, 'blockNumber': 2076033, 'logIndex': 0, 'data': '0x00000000000000000000000000000000000000000000000000038d7ea4c68000ec099eeb7c6385269c410da8eb2e028bf832c8418d731bf99f987e9d9357abe9000000000000000000000000000000000000000000000000000000005a0c6ae0', 'transactionIndex': 0, 'address': '0x131c7ba3ec8ebe69a6c58d084fd70adfacec76c5', 'topics': ['0xa0085ce7b3f5d49ff7ac7b4f8caaf2f75147d0228c3811a6d3d41eb2029fe122', '0xa4162351c79a5ed77bfc705394785c93842b6fc2e55d2b302c86ee55540c5b24', '0x000000000000000000000000c62d9f5092737fa7689579e279a3a1dd5543fb9e', '0x00000000000000000000000025f579697ce3fe1cf122fc5ffae9f1d83f16c676'], 'transactionHash': '0x3cffaa0926589452f9ead7eb016a8a63087051021beaab511fce366b5dcec3d8', 'blockHash': '0x9a120c06c6950f1a3fa29ebd2bfdd349652cda382087d3e402ccb0b56e87ac04'})],
 'logsBloom': '0x00000000000000000000000000000000000000000000002000000000000800400000000000000000000000000000000200000000080000400000000000000000000000000000000000000000000000000000002000200000000000000000000000000000000000000000000000000000020000000000000000000800000000000000000000000000000000000000100000000000000000000000008000000000000000000000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000010000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000',
 'status': '0x1',
 'to': '0x131c7ba3ec8ebe69a6c58d084fd70adfacec76c5',
 'transactionHash': '0x3cffaa0926589452f9ead7eb016a8a63087051021beaab511fce366b5dcec3d8',
 'transactionIndex': 0}

[pipermerriam]

This would fit well under the new my_contract.events.MyEvent api that we'll be implementing (in part borrowed from the web3.js API).

Something like this

• my_contract.events.MyEvent.processReceipt(receipt) would return a rich version of all of the matching logs from a given transaction receipt.
• my_contract.events.MyEvent.parseLog(receipt.logs[0]) would return the given log parsed to it's rich format.

A rich format for a log would allow things like:

• log.args: returns the list of arguments passed into the log.
• log.args[0]: returns the argument at position 0 in the log entry.
• log.args.someArg: returns the argument named someArg from the log entry.

[drandreaskrueger]

Fantastic.

That rich version of the log is exactly what I am hoping for (and not only me, I suppose).
Thanks a lot. And: Go go go :-)

ETA? (I'll continue with this on Monday - a part time job is great ;-) )
What can I use then? Do it manually? Any pointers? Thx.

Thanks a lot, Piper, your Python work is much appreciated! I have seen your main hall talk and panel discussion contributions. Looking forward to be able to watch the breakout session about Python soon.

[drandreaskrueger]

P.S.: Anyone who wants to try out live how that rich log txReceipt.logs[0].args.contractId looks like in truffle, the above mentioned code lines can be visited like this:

sudo apt install npm
sudo npm install -g npm
sudo npm cache clean -f
sudo npm install -g n
sudo n latest

sudo npm i -g ethereumjs-testrpc truffle
testrpc

git clone https://github.com/chatch/hashed-timelock-contract-ethereum
cd hashed-timelock-contract-ethereum
truffle test

[pipermerriam]

@drandreaskrueger

ETA unknown but likely soon (1-2 months?) since overhauling these APIs is somewhat high on the current TODO list. In the meantime... the following psuedo code should work.

from web3.utils.events import (
    get_event_data,
)
log_entry = receipt['logs'][0]  # pick the log entry you know corresponds to the event.
event_abi = my_contract._find_matching_event_abi('MyEvent')
rich_log = get_event_data(event_abi, log_entry)

Note that the rich_log object won't have the API outlined above, but it will have the extra keys event and args which contain the parsed event arguments and event name.

[drandreaskrueger]

Fantastic. Hero!

I got that working, with find_matching...:

event_abi = [m for m in abi if m['name'] == 'LogNewContract'][0]

Probably I could now extract the correct contractId, we'll see.

Problem solved.

---

Just curious:
In this case, fortunately, that contract method newContract fires an event in the end, which also contains the relevant return value contractId.

But I am wondering - what if not?

The newContract ABI

[{'constant': False,
  'inputs': [{'name': '_receiver', 'type': 'address'},
             {'name': '_hashlock', 'type': 'bytes32'},
             {'name': '_timelock', 'type': 'uint256'}],
  'name': 'newContract',
  'outputs': [{'name': 'contractId', 'type': 'bytes32'}],
  'payable': True,
  'stateMutability': 'payable',
  'type': 'function'},
...

mentions outputs:

   'outputs': [{'name': 'contractId', 'type': 'bytes32'}],

where do those outputs end up?

How could they be parsed?

[drandreaskrueger]

another question:

receipt={...
 'args': {... 'contractId': '\x9dÔMG¥6\x0fQ\x8dì\x8bGzõ´F\x96ü\t'
                                     '\x83¾Ð<\x16\x81\x9câı־i',  
              ...},
 'event': 'LogNewContract',
 'transactionHash': '0xb113b00306ba989c06f72dd73a3db6b3a88245ab1fd49b8c2a1675c5baf9372f',
 ...}
contractId = receipt['args']['contractId]

how do I convert that

len(contractId)= 32
type(contractId)= <class 'str'>

from 'str' to 32 bytes ?

I have tried these functions http://web3py.readthedocs.io/en/latest/overview.html#type-conversions but regardless what I tried I always ended up with funny (and per transaction differently) lengthed results.

---

EDIT: Looks like I can simply pass that <str> contractId as a parameter into contract_instance.getContract(contractId) (source) even though that would expect a <bytes32> contractId as input parameter. Odd.

[pipermerriam]

So it looks like you found your contractId in the log data. Currently, you can only get the return value from functions when interacting with the function via eth_call. In the case where your interaction involves sending a transaction, there is currently no way to retrieve the call data unless you do something like log it using events.

The contractId that you're looking at is encoded as bytes. You can get a friendlier more portable version by encoding it has hex using web3.toHex(bytes_value).

[drandreaskrueger]

Currently, you can only get the return value from functions
when interacting with the function via eth_call.

So constant functions, right?

In the case where your interaction involves sending a transaction,
there is currently no way to retrieve the call data

Alright, now that explains that. No amount of searching for solutions would have been enough.

unless you do something like log it using events.

Good to know. That means contract design has to take that into account.

And the EVM (or API?) guys should really consider to solve that problem.

Why would it actually be difficult, to just include the return value into the transaction receipt?

So it looks like you found your contractId in the log data.

Yes, perfect, thank you very much.

by encoding it has hex using web3.toHex(bytes_value).

I'll try that, thanks.

[drandreaskrueger]

by encoding it has hex using web3.toHex(bytes_value).

I'll try that, thanks.

And I see a problem (or I haven't understood something basic yet):

ropsten transaction  --> 0xa7397d1acae590b7b865eaf0fea806b3565e0cd229ab3453afd70fca96b1a184

contractId= GN¾•Á—èïµÓתÁ„ËET,·Þ
5xŽ¥é
type(contractId)= <class 'str'>
len(contractId)= 32
string? how to convert?

contractId_hex = web3.toHex(contractId)

contractId_hex= 0x474ec2bec2957fc381c297c3a8c3afc2b5c393c397c2aa02c38102c284c38b45542cc2b707171ac39e0d3578c28ec2a5c3a9
len(contractId_hex)= 102

but

ropsten transaction 0x3eca42175801a9648b98f57e847918b2d54030d66a84dac9a3ba35df64e4b719 --> 

contractId= ñd§'¬Þ¼Zp¿kFˆ užyV‚M@êp=?õÿ
type(contractId)= <class 'str'>
len(contractId)= 32
string? how to convert?

contractId_hex = web3.toHex(contractId)

contractId_hex= 0xc3b164c2a727c2acc39ec28fc2bc5a70c2bf7f6b46c2882075c29e7956c2824d40c290c290c3aa703d163fc3b5c3bf
len(contractId_hex)= 96

how can hex values be of different length (102 and 96) if both are 32 bytes?

[drandreaskrueger]

I have not found any function which does the below (but probably exists, and I just haven't found it. If not,) ...may this be useful:

def dictifyFnOutput(output, fnName, abi):
    abi_fn = web3.utils.abi.filter_by_name(fnName, abi)
    assert (len(abi_fn)==1) # only 1 function with that name?
    abi_fn = abi_fn[0]

    types = [arg['type'] for arg in abi_fn['outputs']]
    names = [arg['name'] for arg in abi_fn['outputs']]

    return dict(zip(names, output)), dict(zip(names, types))

returns two dicts useful for easier postprocessing:

{'amount': 1000000000000000,
 'hashlock': 'Ìò\x95\x99\x12yÃL\x8c\x8eQ7\x10\x886÷U\x8e\x9e\x97?;\x0es\x96v\n'
             '®¨ßo´',
 'preimage': '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
 'receiver': '0xc62d9f5092737Fa7689579e279a3a1Dd5543Fb9E',
 'refunded': False,
 'sender': '0x25F579697CE3FE1cf122fc5Ffae9f1d83f16C676',
 'timelock': 1511274928,
 'withdrawn': False}

{'amount': 'uint256',
 'hashlock': 'bytes32',
 'preimage': 'bytes32',
 'receiver': 'address',
 'refunded': 'bool',
 'sender': 'address',
 'timelock': 'uint256',
 'withdrawn': 'bool'}

[pipermerriam]

how can hex values be of different length (102 and 96) if both are 32 bytes?

I suspect you are not actually working with byte strings but rather encoded text strings (maybe utf8?). Try using web3.toHex(textstr=contract_id) and see if you get consistent return values.

• what version of web3.py are you using.
• what version of python are you using

Re: the function output by argument names

That does look useful but I'm not entirely sure what the right way for us to incorporate such a thing into the web3 contract API. cc @carver since this might fit well into the concept we've talked about for contract middlewares that do stuff like this to call and return data.

[drandreaskrueger]

I suspect you are not actually working with byte strings but rather encoded text strings (maybe utf8?).

Yes, I suspect that too.
But I am just using what the get_event_data rich log returns to me.

(How) would I have to further postprocess that answer?

Try using web3.toHex(textstr=contract_id)

c2 = web3.toHex(textstr=contractId)

TypeError: to_hex() got an unexpected keyword argument 'textstr'

Have you perhaps implemented that textstr= only recently?

version?

 web3 version = 3.16.2 
 python version =  3.5.3 (default, Jan 19 2017, 14:11:04)   [GCC 6.3.0 20170118]

as the 4.0.0 is beta, that 3.16.2 is the newest, right?

And text= does not help, as

c3 = web3.toHex(text=contractId)
c2 = web3.toHex(contractId)

give identical results:

EXAMPLE TRANSACTION 0xa7397d1acae590b7b865eaf0fea806b3565e0cd229ab3453afd70fca96b1a184 on ropsten

contractId= GN¾•Á—èïµÓתÁ„ËET,·Þ
5xŽ¥é
type(contractId)= <class 'str'>
len(contractId)= 32
string? how to convert?

c2 = web3.toHex(contractId)
c2= 0x474ec2bec2957fc381c297c3a8c3afc2b5c393c397c2aa02c38102c284c38b45542cc2b707171ac39e0d3578c28ec2a5c3a9
len(c2)= 102

c3 = web3.toHex(text=contractId)
c3= 0x474ec2bec2957fc381c297c3a8c3afc2b5c393c397c2aa02c38102c284c38b45542cc2b707171ac39e0d3578c28ec2a5c3a9
len(c3)= 102

EXAMPLE TRANSACTION 0x3eca42175801a9648b98f57e847918b2d54030d66a84dac9a3ba35df64e4b719 on ropsten
###

contractId= ñd§'¬Þ¼Zp¿kFˆ užyV‚M@êp=?õÿ
type(contractId)= <class 'str'>
len(contractId)= 32
string? how to convert?

c2 = web3.toHex(contractId)
c2= 0xc3b164c2a727c2acc39ec28fc2bc5a70c2bf7f6b46c2882075c29e7956c2824d40c290c290c3aa703d163fc3b5c3bf
len(c2)= 96

c3 = web3.toHex(text=contractId)
c3= 0xc3b164c2a727c2acc39ec28fc2bc5a70c2bf7f6b46c2882075c29e7956c2824d40c290c290c3aa703d163fc3b5c3bf
len(c3)= 96

[drandreaskrueger]

some more (rather random) testing:

EXAMPLE TRANSACTION 0xa7397d1acae590b7b865eaf0fea806b3565e0cd229ab3453afd70fca96b1a184 on ropsten

c4 = contractId.encode('utf-8')
c4= b'GN\xc2\xbe\xc2\x95\x7f\xc3\x81\xc2\x97\xc3\xa8\xc3\xaf\xc2\xb5\xc3\x93\xc3\x97\xc2\xaa\x02\xc3\x81\x02\xc2\x84\xc3\x8bET,\xc2\xb7\x07\x17\x1a\xc3\x9e\r5x\xc2\x8e\xc2\xa5\xc3\xa9'
type(c4)= <class 'bytes'>
len(c4)= 50

c4 = contractId.encode('utf-16')
c4= b'\xff\xfeG\x00N\x00\xbe\x00\x95\x00\x7f\x00\xc1\x00\x97\x00\xe8\x00\xef\x00\xb5\x00\xd3\x00\xd7\x00\xaa\x00\x02\x00\xc1\x00\x02\x00\x84\x00\xcb\x00E\x00T\x00,\x00\xb7\x00\x07\x00\x17\x00\x1a\x00\xde\x00\r\x005\x00x\x00\x8e\x00\xa5\x00\xe9\x00'
type(c4)= <class 'bytes'>
len(c4)= 66

EXAMPLE TRANSACTION 0x3eca42175801a9648b98f57e847918b2d54030d66a84dac9a3ba35df64e4b719 on ropsten

c4 = contractId.encode('utf-8')
c4= b"\xc3\xb1d\xc2\xa7'\xc2\xac\xc3\x9e\xc2\x8f\xc2\xbcZp\xc2\xbf\x7fkF\xc2\x88 u\xc2\x9eyV\xc2\x82M@\xc2\x90\xc2\x90\xc3\xaap=\x16?\xc3\xb5\xc3\xbf"
type(c4)= <class 'bytes'>
len(c4)= 47

c4 = contractId.encode('utf-16')
c4= b"\xff\xfe\xf1\x00d\x00\xa7\x00'\x00\xac\x00\xde\x00\x8f\x00\xbc\x00Z\x00p\x00\xbf\x00\x7f\x00k\x00F\x00\x88\x00 \x00u\x00\x9e\x00y\x00V\x00\x82\x00M\x00@\x00\x90\x00\x90\x00\xea\x00p\x00=\x00\x16\x00?\x00\xf5\x00\xff\x00"
type(c4)= <class 'bytes'>
len(c4)= 66

EXAMPLE TRANSACTION 0xc4c89abe771c6c9e0f23b37cec3ef49b4d5d5168b6eea257ec26c402dd22e449 on ropsten

c4 = contractId.encode('utf-8')
c4= b'\xc3\x99\xc3\x81\xc2\xb2n\xc3\xa7\xc3\x97\xc2\x8d\\\xc3\xb9X\xc2\x95\xc2\xbf\xc2\x8f\xc2\x93\xc3\x9dQ\xc3\x89\x1dW:G\xc3\x853g\xc2\xa8;\xc2\xa3\n#\xc3\x90\xc3\xa9\xc3\x86'
type(c4)= <class 'bytes'>
len(c4)= 51

c4 = contractId.encode('utf-16')
c4= b'\xff\xfe\xd9\x00\xc1\x00\xb2\x00n\x00\xe7\x00\xd7\x00\x8d\x00\\\x00\xf9\x00X\x00\x95\x00\xbf\x00\x8f\x00\x93\x00\xdd\x00Q\x00\xc9\x00\x1d\x00W\x00:\x00G\x00\xc5\x003\x00g\x00\xa8\x00;\x00\xa3\x00\n\x00#\x00\xd0\x00\xe9\x00\xc6\x00'
type(c4)= <class 'bytes'>
len(c4)= 66

with utf-8 ==> different lengths
with utf-16 ==> identical lengths! But why 66?

Again, I am only postprocessing the <class 'str'> that web3.utils.events.get_event_data(abiEvent, log) (probably this sourcecode) returns for an object of type bytes32.

[carver]

EDIT: Looks like I can simply pass that contractId as a parameter into contract_instance.getContract(contractId) even though that would expect a contractId as input parameter. Odd.

This is actually the best available answer for v3. v3 expects a latin-1 decoded string in bytes arguments. Is this sufficient for your solution?

If you really want to see the hex value in v3, you would need to do:

contract_id = '\x9dÔMG¥6\x0fQ\x8dì\x8bGzõ´F\x96ü\t\x83¾Ð<\x16\x81\x9câı־i'
id_bytes = contract_id.encode('latin-1')
id_hex = Web3.toHex(id_bytes)
assert id_hex == '0x9dd44d47a5360f518dec8b477af5b44696fc0983bed03c16819ce2c4b1d6be69'

[pipermerriam]

with utf-16 ==> identical lengths! But why 66?

32 bytes * 2 for hex encoding + 2 for 0x prefix.

Very odd that utf-16 worked... paging @carver any ideas?

[pipermerriam]

@drandreaskrueger is any of this on the mainnet/ropsent/rinkeby? can you provide abi, address, and transaction hash for a transaction with one of these values in it?

[drandreaskrueger]

expects a latin-1 decoded string

that's it. Hooray.

[drandreaskrueger]

EXAMPLE TRANSACTION 0xa7397d1acae590b7b865eaf0fea806b3565e0cd229ab3453afd70fca96b1a184 on ropsten

c6 = contractId.encode('latin-1')
c6= b'GN\xbe\x95\x7f\xc1\x97\xe8\xef\xb5\xd3\xd7\xaa\x02\xc1\x02\x84\xcbET,\xb7\x07\x17\x1a\xde\r5x\x8e\xa5\xe9'
type(c6)= <class 'bytes'>
len(c6)= 32

contractId_hex = c7 = Web3.toHex(c6)
c7= 0x474ebe957fc197e8efb5d3d7aa02c10284cb45542cb707171ade0d35788ea5e9
type(c7)= <class 'str'>
len(c7)= 66

EXAMPLE TRANSACTION 0x3eca42175801a9648b98f57e847918b2d54030d66a84dac9a3ba35df64e4b719 on ropsten

c6 = contractId.encode('latin-1')
c6= b"\xf1d\xa7'\xac\xde\x8f\xbcZp\xbf\x7fkF\x88 u\x9eyV\x82M@\x90\x90\xeap=\x16?\xf5\xff"
type(c6)= <class 'bytes'>
len(c6)= 32

contractId_hex = c7 = Web3.toHex(c6)
c7= 0xf164a727acde8fbc5a70bf7f6b468820759e7956824d409090ea703d163ff5ff
type(c7)= <class 'str'>
len(c7)= 66



md5-1869e4ef61e43ec762d865e2cbd58ae3



EXAMPLE TRANSACTION 0xc4c89abe771c6c9e0f23b37cec3ef49b4d5d5168b6eea257ec26c402dd22e449 on ropsten

c6 = contractId.encode('latin-1')
c6= b'\xd9\xc1\xb2n\xe7\xd7\x8d\\\xf9X\x95\xbf\x8f\x93\xddQ\xc9\x1dW:G\xc53g\xa8;\xa3\n#\xd0\xe9\xc6'
type(c6)= <class 'bytes'>
len(c6)= 32

contractId_hex = c7 = Web3.toHex(c6)
c7= 0xd9c1b26ee7d78d5cf95895bf8f93dd51c91d573a47c53367a83ba30a23d0e9c6
type(c7)= <class 'str'>
len(c7)= 66

that is looking very good. Thanks a million.

[drandreaskrueger]

latin-1 was the answer to the riddle.

So the web3.utils.events.get_event_data(abiEvent, log) returns a latin-1 encoded str for a bytes32 solidity object.

And that str can be turned into a more portable version by:

bytes32answer_hex = Web3.toHex(  bytes32answer_string.encode('latin-1')  )

and the reverse direction should be

bytes32input_string = Web3.toBytes(  hexstr=bytes32answer_hex  ).decode('latin-1')

thanks a lot! No need to dig deeper into code, no?

My problem's solved, I guess. Weeeeekeeeeend ! ;-)

[pipermerriam]

@drandreaskrueger glad to get this resolved.

[carver]

@dylanjw I'd recommend reading through this thread deeply: all the comments, drandreaskrueger's implementation, plus the js implementations.

The rich API in Piper's first comment seems like a good place to start. It's also worth thinking about other options, like:

f = contract.events.MyEvent.filter()  # same API as w3.eth.newFilter()
event_history = f.get_all_entries()
# wait a bit
recent_events = f.get_new_entries()

to create a filter that watches for that event. You could probably handle all the conversions in the returned logs by creating the LogFilter with a formatter function, like LogFilter(log_entry_formatter=contract.events.MyEvent.parseLog), where parseLog is one of the new methods you write.

Note that you don't have to worry about backporting this to v3 (or dealing with those latin-1 encoding issues) since v3 is on critical security patches only.

[carver]

Released in beta 9: http://web3py.readthedocs.io/en/latest/releases.html#v4-0-0-beta-9