Web3.py: Optional compatibility with geth PoA dev chain, to fix: Error could not format value <long hex string> as field 'extraData'

[bonedaddy]

What's Wrong

When using geth --dev, it uses a proof-of-authority chain for instant mining. Unfortunately, the mechanism for PoA is to add a bunch of bytes to extraData, which is not yellow-paper-compliant. We would like to have the default web3.py strictly reject non-compliant clients, but still enable people to use this popular development alternative.

Proposed Solution

Add a new middleware, called something like truncate_extra_data_middleware. It would trim the extraData field down to 32 bytes or fewer. Note that this would be best to go earlier in the middleware stack than the pythonic middleware, which raises the exception.

To be complete:

• [ ] A test showing that a long extradata does not raise an error, after adding the new middleware
• [ ] A test showing that a long extradata does raise an error in the default configuration
• [ ] A new middleware truncate_extra_data_middleware (probably constructed with a more general construct_truncate_result_middleware(whitelist_methods, field, truncate_to))
• [ ] Add to the API for the middleware stack (probably). Either something like .bury(obj) which puts a new middleware to the bottom of the stack, or a more general .insert(pos, obj) which inserts middleware to an arbitrary location. Up for discussion.
• [ ] Some basic documentation describing why and how to use the new middleware

Original Issue

Getting the following error message when trying to send a transaction via web3py:

Error Could not format value '0xd783010800846765746887676f312e392e32856c696e7578000000000000000086ddf484c77c385bf8ec5c04427ccb3b2624efc39c966ae16858213df5f87ca453022c474a9346faec0be34f6ec2c16da2a987fd08670465c3b70bb361848a8a00' as field 'extraData'

Here is the script im using

https://gist.github.com/postables/8b634de55033c27a7d870aaeb5f02103

[bonedaddy]

Here's the full stack track

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "transfer.py", line 40, in <module>
    hash = contract.transact({'from': sourceAddress}).transfer(address, int(vztAmount))
  File "/usr/lib/python3.6/site-packages/web3-4.0.0b5-py3.6.egg/web3/contract.py", line 1028, in transact_with_contract_function
    txn_hash = web3.eth.sendTransaction(transact_transaction)
  File "/usr/lib/python3.6/site-packages/web3-4.0.0b5-py3.6.egg/web3/eth.py", line 207, in sendTransaction
    get_buffered_gas_estimate(self.web3, transaction),
  File "/usr/lib/python3.6/site-packages/web3-4.0.0b5-py3.6.egg/web3/utils/transactions.py", line 161, in get_buffered_gas_estimate
    gas_limit = get_block_gas_limit(web3)
  File "/usr/lib/python3.6/site-packages/web3-4.0.0b5-py3.6.egg/web3/utils/transactions.py", line 152, in get_block_gas_limit
    block = web3.eth.getBlock(block_identifier)
  File "/usr/lib/python3.6/site-packages/web3-4.0.0b5-py3.6.egg/web3/eth.py", line 123, in getBlock
    [block_identifier, full_transactions],
  File "/usr/lib/python3.6/site-packages/web3-4.0.0b5-py3.6.egg/web3/manager.py", line 98, in request_blocking
    response = self._make_request(method, params)
  File "/usr/lib/python3.6/site-packages/web3-4.0.0b5-py3.6.egg/web3/manager.py", line 81, in _make_request
    return request_func(method, params)
  File "/usr/lib/python3.6/site-packages/web3-4.0.0b5-py3.6.egg/web3/middleware/gas_price_strategy.py", line 18, in middleware
    return make_request(method, params)
  File "/usr/lib/python3.6/site-packages/web3-4.0.0b5-py3.6.egg/web3/middleware/formatting.py", line 23, in middleware
    response = make_request(method, params)
  File "/usr/lib/python3.6/site-packages/web3-4.0.0b5-py3.6.egg/web3/middleware/attrdict.py", line 20, in middleware
    response = make_request(method, params)
  File "/usr/lib/python3.6/site-packages/web3-4.0.0b5-py3.6.egg/web3/middleware/formatting.py", line 30, in middleware
    formatter(response['result']),
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/usr/lib/python3.6/site-packages/web3-4.0.0b5-py3.6.egg/web3/utils/formatters.py", line 57, in apply_formatter_if
    return formatter(value)
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/usr/lib/python3.6/site-packages/eth_utils-0.7.4-py3.6.egg/eth_utils/functional.py", line 22, in inner
    return callback(fn(*args, **kwargs))
  File "/usr/lib/python3.6/site-packages/web3-4.0.0b5-py3.6.egg/web3/utils/formatters.py", line 70, in apply_formatters_to_dict
    raise type(exc)("Could not format value %r as field %r" % (item, key)) from exc
ValueError: Could not format value '0xd783010800846765746887676f312e392e32856c696e75780000000000000000ce01c0febfffab0bb40eb0c7c91c163fc2c5e709b50e53939ced0cabe36d4d575b98e0468080f949f3332b3ab0957ff27a52ec00bc98477c0f6dcc374cbab5a700' as field 'extraData'

[bonedaddy]

I was able to have my code work by running it on a private ethereum network, perhaps there is an issue with Rinkeby for some reason?

I saw a previous issue where someone also had specific issues with rinkeby,

[carver]

Yes, unfortunately the way geth does PoA seems to violate the yellow paper definition of extraData, as mentioned in #507 . We are undecided how to handle it.

[akolotov]

Hi!

I have found that in order to specify that a filed could have variable length variable_length is used in to_hexbytes: https://github.com/ethereum/web3.py/blob/ab3f9860886d2d9071f951eab19ba94fd2bbda0f/web3/middleware/pythonic.py#L68-L84

But the logic is works just for cases when the actual number of nibbles is less than required. So, if the filed contains more nibbles ValueError raised. Could you share thoughts why the logic works in such manner?

Could the changes in this logic solve the issue? I went through the code of pythonic.py and see 4 places where variable_length is set to True. Judging on these fields, probably, it will not break anything if the idea to handle fields with variable length as proper even if it is larger than expected is implemented.

[carver]

it will not break anything if the idea to handle fields with variable length as proper even if it is larger than expected is implemented.

The thing that it breaks is the strict enforcement of the protocol. :) The question at hand is: Should we enable behaviors that are non-protocol-compliant? In general, web3.py aims to reject invalid inputs and outputs, which helps us identify bugs and encourage consensus behavior.

One possible approach would be to have an optional middleware that truncates the extraData field before it is formatted into its "pythonic" type. Users could add that middleware when connecting to the dev chain.

_FWIW, if we don't care how long the field is, then we can just use HexBytes() directly, instead of using to_hexbytes(), which was written for a stricter limitation on the number of bytes allowed. Although we might want a cleaner type-checking error in HexBytes(), which seems better to have anyway._

[gitcoinbot]

__This issue now has a funding of 0.5 ETH (509.52 USD) attached to it.__

• If you would like to work on this issue you can claim it here.
• If you've completed this issue and want to claim the bounty you can do so here
• Questions? Get help on the Gitcoin Slack
• $14297.18 more Funded OSS Work Available at: https://gitcoin.co/explorer

[bonedaddy]

I think compatability should be implemented since web3py can also be used for private Ethereum chains (POA) not just the mainnet.

I have a client that I work with whose thinking of exploring POA networks like Rinkeby is for production use and I won't be able to use web3py for that.

I think we should do something similar to Oraclize where their smart contract can identify the different networks it's on to fulfill different needs but it's all done in one contract. Perhapd we could leverage network ID to indicate whether or not we would need to have to work around?

[carver]

Perhapd we could leverage network ID to indicate whether or not we would need to have to work around?

This is an interesting idea. It's possible that we could add the middleware to the default list if it only applies when the network ID matches some whitelist. This is a relatively small change, and doesn't affect the bulk of the proposed solution above.

[akolotov]

Sounds good. Will it be a whitelist of the networks which follow standards?

[carver]

Will it be a whitelist of the networks which follow standards?

No, the default should be to follow standards. The whitelist would be for networks that deviate in some specific way, like PoA chains, and would cause the middleware to trigger.

[bonedaddy]

It's possible that we could add the middleware to the default list if it only applies when the network ID matches some whitelist.

Oraclize does something similar to this as well so I think this would be the best way. Perhaps we don't have to hard code the white list, but make it easy for the user to configure the white list themselves? This way we can ensure that by default web3py follows protocol standards and IF someone needs to they can add the whitelist?

[akolotov]

I like an idea to easily add my private PoA dev network to whitelist instead of having two different branches of code for a dev network and for a public network.

[carver]

I would design it as:

• a default middleware that only truncates extraData if the network ID in a whitelist matches
• an optional middleware that always truncates extraData

That way:

• If you are connecting to a compliant network, or a known PoA network, it will "just work" out of the box.
• If you have a custom chain on PoA that is not in the whitelist, you just add that optional middleware to your web3 instance.

[dongsam]

This is emergency monkey patch for solve this error in POA network

from web3 import Web3, HTTPProvider
from web3.middleware.pythonic import (
    pythonic_middleware,
    to_hexbytes,
)

size_extraData_for_poa = 200   # can change

web3 = Web3(HTTPProvider(YOUR_NODE_URI))
pythonic_middleware.__closure__[2].cell_contents['eth_getBlockByNumber'].args[1].args[0]['extraData'] = to_hexbytes(size_extraData_for_poa, variable_length=True)
pythonic_middleware.__closure__[2].cell_contents['eth_getBlockByHash'].args[1].args[0]['extraData'] = to_hexbytes(size_extraData_for_poa, variable_length=True)

[vs77bb]

@akolotov Would you like to claim this on Gitcoin? @carver @postables Otherwise we can pub this one out today!

[akolotov]

Not sure that will have enough time to prepare a proper fix for this if you meant this by claiming.

[vs77bb]

@akolotov That's what I meant - thanks for letting me know :)

[bonedaddy]

@dongsam Thanks! The monkey patch works.

When I have some free time in the next week I'll poke around to see if I could help add a proper fix.

[vs77bb]

Hi @postables, if you are interested in claiming the bounty, feel free to claim the issue on Gitcoin to show your interest 🙂 cc @owocki

[dongsam]

I'm trying to add middleware for this issue

A new middleware truncate_extra_data_middleware (probably constructed with a more general construct_truncate_result_middleware(whitelist_methods, field, truncate_to))

but I wonder PoA Clique algorithm work fine even if we truncate extra data like above,

expanding extra data length is not good solution? because yellow-paper-compliant?

[carver]

expanding extra data length is not good solution? because yellow-paper-compliant?

You can't really add a middleware that changes how the middleware above it in the stack works (the pythonic middleware will always enforce that extraData length.

Another option is that you could add another key authorityData, and move the value from extraData to it.

[pipermerriam]

Another option is that you could add another key authorityData, and move the value from extraData to it.

I like this solution.

[pipermerriam]

This issue is, or whatever pull request you open is the best place to contact either me or @carver to ask questions. Check the documentation on read-the-docs for how to get a dev environment setup.

The gitter chat room for this repo is also a good place to ask questions.

[carver]

Check the documentation on read-the-docs for how to get a dev environment setup.

I'm not certain that the developer setup is on read-the-docs, it might only be in the readme:
https://github.com/ethereum/web3.py#developer-setup

I think that's a good thing: to not intermingle user-facing docs with developer-facing docs...

going down to code level only if needed.

I can't imagine how you will solve this without some code, but 💯 if you can.

[dylanjw]

You can avoid the long sync time by running a local geth node with the —dev option. I think that would be the easiest way to set up a test.

[dylanjw]

From what I understand of the original issue, starting a geth client in dev mode and running through the QuickStart section of the web3 readme, except connected to the geth client instead of the tester provider, should be enough to trigger the issue.

[carver]

Oh I ended up writing this middleware because I needed it for the external vs public logging issue. Will post soon

[carver]

First step of two steps is posted: https://github.com/ethereum/web3.py/pull/648

[carver]

Second step is to add an optional middleware for geth-dev compatibility. I'll post it and link here after #648 closes.

[gitcoinbot]

__Work has been started on the 0.5 ETH (427.76 USD) funding by__:

1. @lazaridiscom
2. @lazaridiscom

__Please work together__ and coordinate delivery of the issue scope. Gitcoin doesn't know enough about everyones skillsets / free time to say who should work on what, but we trust that the community is smart and well-intentioned enough to work together. As a general rule; if you start work first, youll be at the top of the above list ^^, and should have 'dibs' as long as you follow through.

On the above list? Please leave a comment to let the funder and the other parties involved what you're working, with respect to this issue and your plans to resolve it. If you don't leave a comment, the funder may expire your submission at their discretion.

• Learn more on the gitcoin issue page
• Questions? Get help on the Gitcoin Slack
• $9158.8 more Funded OSS Work Available at: https://gitcoin.co/explorer

[pipermerriam]

@lazaridiscom I'm going to leave this to @carver , he'll be your contact going forward on this issue as I'm too bogged down with other responsibilities.

[owocki]

👋 hi folks... i believe we fixed the issue on our side. thanks for your patience

[carver]

@lazaridiscom I'm having trouble tracking down: which PR did you submit for this bounty?

[pipermerriam]

The silence you're seeing isn't intentional. Just being oversubscribed and missing this come through.

I find it best to assume incompetence before assuming malice 😄

This wasn't on my radar until about an hour ago. I'm doing some fact finding and catching up. I'll get you resolution by end of week.

[pipermerriam]

@lazaridiscom thanks for the ping, I did indeed lose track of this again. :(

I've decided that we won't be issuing the bounty for this issue. While we/I appreciate the work you did on this issue, it was not in a state that was mergable due to the lack of both tests and documentation.

The lead dev on this project (@carver ) chose to move forward independently to get this wrapped up. I would posture that this decision was made purely based on what he thought would get things done quickest.

In the future we'll try for better communication on our part, as well as being more aware in cases like this that someone has put in work with the expectation of receiving the bounty.

[pipermerriam]

I trusted the ConsenSys brand

This project isn't affiliated with ConsenSys

I appreciate your feedback. This whole bounty thing is new for us and we're still figuring out how it fits into our workflow. I'll be sticking with my decision not to pay out the bounty. I apologize for the failures on our part.

One extra piece of information, not intended to try and change your mind, but rather give you more context. The ether for the bounty is my own personal funds. We have a few bounties that have been funded by gitcoin itself, but the majority of them at this point have been personally funded by me.

[owocki]

Laz, I'm sorry you're not satisfied with the outcome here... But the contributor guidelines and the issue specification of this issue itself is pretty clear that there needs to be tests/documentation. You didn't need to wait for any code review to get this information.

Really? Have you actually invested time to review this? You are twisting the timeline.

I'll remind you, again, of the Gitcoin code of conduct 1 when interacting with others through the platform.

We are doing a retrospective with Piper & the web3py team to figure out how both Gitcoin can learn from this and how we can better communicate best practices to users of the bounty platform.

I'd happily send out a courtesy tx to compensate you for the time you spent on this but last time I did that you sent me a refund without any explanation. Please ping me on slack if you'd like to revisit.

[bonedaddy]

has this been fixed? I'm no longer experiencing the error I was using a python script on a PoA network of my own and didn't get any tracebacks or errors indicating extra data length fields were detected, however, I did not include the patch mentioned by dongsam earlier in the thread.

[jfdelgad]

Was this solved in some way? I am getting the same error using rinkeby

w3.eth.sendTransaction({'to':w3.eth.accounts[0],'from':w3.eth.accounts[1],'value':w3.toWei(1,'ether')})
ValueError: Could not format value '0xd783010802846765746887676f312e392e34856c696e757800000000000000004828e79588880b1fc9557dc3ee74d3567f269e4fc86b8319cc24176b8d35f8fc47aa7196f01c54907983a67e74341a8b3571d7edb36f6fbd0f9fd0714024e43101' as field 'extraData'

Any help would be appreciated

[dylanjw]

@jfdelgad Have a look at the documentation for the optional PoA middleware: http://web3py.readthedocs.io/en/latest/middleware.html#geth-style-proof-of-authority

[carver]

This is now on stackexchange for discoverability, as well: https://ethereum.stackexchange.com/questions/44130/rinkeby-failure-with-web3-py-could-not-format-value-0x-as-field-extrada