Go-ethereum: Estimated gas doesn't match the actual gas usage.

[frozeman]

I'm on a private chain so no one else modifies it, in this case the estimated gas usage should be equal the actual gas usage.

I'm using the wallet contract and making a multi sig transaction.

Estimate call:

{"jsonrpc":"2.0","method":"eth_estimateGas","params":[{"from":"0x990ccf8a0de58091c028d6ff76bb235ee67c1c39","to":"0x17b96218bdc36434421b79c266fae5e82643c28b","data":"0xb61d27f6000000000000000000000000922a519ac926f69856fcfc1b2b8b846cfb3f6b4e00000000000000000000000000000000000000000000006c7974123f64a40000000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}],"id":5878}

Response: 80999 gas

{"id":5878,"jsonrpc":"2.0","result":"0x13c67"}

Send real transaction

{"jsonrpc":"2.0","method":"eth_sendTransaction","params":[{"from":"0x990ccf8a0de58091c028d6ff76bb235ee67c1c39","gasPrice":"0xd4d645fea","gas":"0x2c307","to":"0x17b96218bdc36434421b79c266fae5e82643c28b","data":"0xb61d27f6000000000000000000000000922a519ac926f69856fcfc1b2b8b846cfb3f6b4e00000000000000000000000000000000000000000000006c7974123f64a40000000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}],"id":5900}

Get receipt

{"jsonrpc":"2.0","method":"eth_getTransactionReceipt","params":["0xcf10cb757d71283cb5998dc97f0d8cdbadf497f01c164ba8199dac5493bd4c82"],"id":5940}

Response: 171998 gas

{"id":5940,"jsonrpc":"2.0","result":{"transactionHash":"0xcf10cb757d71283cb5998dc97f0d8cdbadf497f01c164ba8199dac5493bd4c82","transactionIndex":"0x0","blockNumber":"0x35e","blockHash":"0xf3546823ec552537fa8dd6bc891b0f1325cb37d6b842c577c7bc1c663c1b3cb8","cumulativeGasUsed":"0x29fde","gasUsed":"0x29fde","contractAddress":null,"logs":[]}}

[frozeman]

Thats a big difference and makes it impossible to give the user a good estimation.

[frozeman]

Providing a high gas amount suddenly lets the estimation go to 166078 gas, which is close to the real number.
I would suggest to set the provided gas by default to 50mio, so that the callee doesn't have to do that. (but let the user overwrite the default)

Additionally i think some storage modification gas is not calculated in therefore the difference of 5920 gas??

[frozeman]

Interestingly if i'm under the dailylimit in the contract the estimation is now correct (49326 gas), when i provide ha high gas amount.
So the problem seems to happen when confirmations is needed, which modifies the state to store the pending tx. So it could be the storage, which is not calculated in.

[obscuren]

Are you providing multiple transactions perhaps? This does not work. Each eth_call has it's own context, so to speak. Transactions do not share the same context.

[frozeman]

as the example above shows, its only one transaction, which i "estimate" before.

[frozeman]

So right now the difference is low, but if its really the storage change, which is not accounted for, it can be become a rather big difference when using a contract which models the storage a lot.

[frozeman]

I just tried to use the eth_estimateGas function to estimate a contract deployment and it seems to be wrong.

the contract would cost ~1.8mio but it estimates me 300k gas only.

This could again have to do with that the storage modification costs are not counted in?
(contract changing the store and the cost to store the contract itself) (?)

This has IMHO priority when we want to release the wallet, so user can get proper estimates, and i don't have to hardcode a gas of 3mio, as i add a data field and this allows to use the wallet even to deploy contracts and send data along (required for Kraken deposits)

[lastperson]

I believe this is due to a gas refunds non deterministic logic. Here is the description of the problem from inside the solidity, but it remains the same from outside: it is impossible to say (with constant number of tries) the minimum amount of gasLimit that should be sent with transaction. This might be resolved with implementing the tx.gasRefund counter inside the EVM.

[brendan87]

I'm getting a ~32000 gas estimate for a transaction that actually costs about 350k. The estimation seems to be okay on most other transactions. The transaction in question contains 7 arguments that are all arrays. The transaction functionally modifies storage variables (appends to them) as well, in a loop. Not sure if any of that is helpful

[lastperson]

@brendan87, when doing estimate call, try specifying gas: 3000000 and see if it will help.

[brendan87]

@lastperson I am setting the gas to 5 million when sending off the estimate call

[almindor]

Probably not directly related to this issue on the technical level but even a basic send transaction is currently estimated wrong at 20001 gas. No idea why the extra 1 gas is added there.

[karalabe]

@almindor That's a side effect of binary search + no way to figure out if a tx was successful or not (beside gas use). It's being fixed in the next release though! :)

[MicahZoltu]

What is the status of this. Should we just assume that estimateGas is completely broken and figure out other mechanisms for estimating gas? Reading through the comment history, it appears that there isn't a way to get Geth to estimate gas costs correctly if the call does any storage.

[karalabe]

Should work correctly for all transactions. Are you having issues with the current code?

[MicahZoltu]

Yes. We have integration tests that work fine in Parity on a local single-node chain that are using estimateGas to figure out how much gas the next transaction will cost, but the same script with Geth results in transactions failing due to out of gas. Looking at the gas estimates, they are _way_ off like the original report suggested.

[karalabe]

@MicahZoltu Could you send us a repro transaction?

Geth's gas estimator uses binary search, based on the outer status of the transaction. If the transaction fails for whatever reason, the gas consumed is bumped. If the transaction succeeds, the gas goes down.

The only way this could fail estimating if there would be an internal transaction which fails, but the outer one doesn't mind and continues executing successfully. Our estimator assumes that outer transactions report errors in case internals go belly up. Perhaps this is why it doesn't estimate correctly for you?

[MicahZoltu]

I have been super busy lately so I'm not sure I'll have time to put together a formal repro case, but factory contracts submitted with no gas seem to be the source of my troubles.

contract MyFactory {
  function create() returns (address) {
    return new MyContract();
  }
}

If I call eth_estimateGas on that, I'll get back an accurate gas estimation and can then follow-up with an eth_sendTransaction call that will succeed. However, if I just do eth_sendTransaction with no gas included (so Geth estimates the gas required internally) then the call will fail. This _appears_ to happen anytime a transaction includes a new Foo() somewhere in the call chain.

Note: the above alone may not reproduce the problem, though I suspect it will and if I have time to create a repro case that is where I'll start. Our actual repro case is much more complicated, though if you really want to dig then calling .initialize() is what fails unless you comment out the new Delegator(...) in ReputationTokenFactory.

[karalabe]

I'm not sure Geth estimates gas usage if non is provided. I've seen a lot of txs where is just defaults to 90K.

[karalabe]

I think it was a 'bug' I wanted to look closer into. Perhaps it was a backup plan when our estimator wasn't binary search based

[MicahZoltu]

I _believe_ the transaction that was failing was larger than 90k without the new Delegator(...) in it. I want to say it was something like 200k without, 400k with.

[nuliknol]

karalabe , what binary search you are talking about? All you have to do is to provide a high gas limit, execute a Call() and then get the amount of gas consumed by the Call. That's all you have to do to get gas usage. What binary search ????

[karalabe]

That doesn't take into account refunds, which result is a lower gas consumption than actually needed. It also doesn't take into account stipends that are needed, but may not be actually used.

[karalabe]

Closing this thread as gas eetimation work correctly.