Go-ethereum: Transactions in Status 0 in a private POA Network

[otto-mora]

Hi there,

We seem to be receiving status 0 transactions whenever we try scenarios in our network with a higher number of transactions involved. Even after checking the transaction a few minutes after they remain in Status 0 permanently.

System information

Geth version: 1.9.15
OS & Version: Ubuntu Linux

Expected behaviour

Submit a transaction for an NFT transfer or mint and receive a transaction in status 1, even if it takes longer to get a result due to the higher number of transactions being requested.

Actual behaviour

After getting a few hundred transactions through the network, we start getting some transactions in Status 0:

{
    "to": "0x285Ee90f706CbF03b91aFeCA01b63825aB5824A5",
    "from": "0x35128869b8c24a76c1850728B5dE1D94d53344C9",
    "contractAddress": null,
    "transactionIndex": 0,
    "gasUsed": {
        "_hex": "0x66bf"
    },
    "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
    "blockHash": "0x4f678da466099356f0a5f1af642588ac34e381058ac96430e03d230244f09c7a",
    "transactionHash": "0xcbe9ff154cf90536f271f05aec0ae07a1621bfbba044d079ace52518c0e6f4aa",
    "logs": [],
    "blockNumber": 7142,
    "confirmations": 1,
    "cumulativeGasUsed": {
        "_hex": "0x66bf"
    },
    "status": 0,
    "byzantium": true
}

Initially we thought this might be something related to byzantium and our genesis block not enabling the latest hardforks (since the byzantium value is set to true in the transactions receipt, however even after changing our genesis block we are still getting this status 0 on certain occasions when we try to run scenarios with a higher number of transactions.

This how the hardforks are setup in the genesis block:

{
  "config": {
    "chainId": 15,
    "homesteadBlock": 0,
    "eip150Block": 0,
    "eip150Hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "eip155Block": 0,
    "eip158Block": 0,
    "byzantiumBlock": 0,
    "constantinopleBlock": 0,
    "petersburgBlock": 0
}

We even implemented some logic to wait 15 seconds before re-attempting to submit a new transaction again (thinking that we might be overwhelming the geth node with too many concurrent transactions) however this seems to be failing even after several re-attempts (up to 20 re-attempts)

Any ideas as to why this might be occurring? Are we missing anything in our genesis block or anything else?

[karalabe]

Status 0 means the tx failed execution. It's not related to the amount of txs or the amount of time you wait. Something failed in the EVM.

In your genesis block, you are still missing the latest fork, istanbulBlock. Perhaps your Solidity compiler relies on an opcode introduced in Istanbul, and that's the issue? Could you try enabling that last fork too and check again?

[garyng2000]

yes, that hard fork config is important

[otto-mora]

Thank you Peter and Gary! Good suggestions indeed. We have enabled the istanbulBlock and even though we are still getting the error, we at least are able to obtain more details on the error by performing an eth_call :

{
    "jsonrpc": "2.0",
    "id": 1,
    "error": {
        "code": -32000,
        "message": "execution reverted"
    }
}

I am not quite sure the meaning of error code 32000, though this link refers to it as an "ABI Mismatch" not sure what that means.

I appreciate any other suggestions you may have. We already tried increasing the gas limit to see if that would make it go away, but this did not resolve it. Other ideas we have are: 1) Switching from EthersJSv4 to Web3, 2) Trying to implementing a queue mechanism so that transactions are sent to the Ethereum node one at a time (thinking that it might be a concurrency problem of some kind).

Here is a more detailed look at our Genesis block settings:

{
  "alloc": {
      "0x0c1c28336F5F256bD6657215F00eE83121e51336": {"balance": "7500000000000000000000000000000"}
  },
  "config": {
    "chainId": 15,
     "homesteadBlock": 0,
    "eip150Block": 0,
    "eip150Hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "eip155Block": 0,
    "eip158Block": 0,
    "byzantiumBlock": 0,
    "petersburgBlock": 0,
    "constantinopleBlock": 0,
    "istanbulBlock": 0,
    "clique": {
        "period": 0,
        "epoch": 30000
     }         
  },
   "nonce": "0x0",
  "timestamp": "0x59e50516",
  "extraData": "0x41756775720000000000000000000000000000000000000000000000000000000c1c28336F5F256bD6657215F00eE83121e513360000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
  "gasLimit": "0x7270E0",
  "difficulty": "0x1",
  "mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "coinbase": "0x0000000000000000000000000000000000000000",
  "number": "0x0",
  "gasUsed": "0x0",
  "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000"
}

Geth version: 1.9.15

Thank you!

[garyng2000]

you are still missing another hard fork after Istanbul. I don't know if that is important but based on my experience in another reported 'issue', make sure they are all there. another thing or two which I have found, 'mint on demand'(block period 0) can be tricky, I end up set it to 15(sort of the same as POW) and on the safe side, I space out each hard fork(i.e. each has different block number). It may or may not make a difference but I just try to avoid corner cases.

[karalabe]

@otto-mora Could you provide a repro that we might check. It's probably easier to see what could be wrong if we had something to run ourselves.

@garyng2000 The only fork after Istanbul was Muir Glacier, but that just adjusted the ethash PoW difficulty, so it's not relevant for this chain/issue.

[lucachaco]

Hi @karalabe,
When we call a function (_mint_) on our smart contract we don’t get the error if we set the _gasLimit_ override option as an extra parameter. The following code shows this example.

const instance = new ethers.Contract(contractAddress, contractABI, provider);
const wallet = await getWallet(privateKey);
const  contractnstance = contract.connect(wallet);
const overrides = {
  gasLimit: 750000,
};
contractInstance.mint(… , overrides)

But the above option is not a good fit for us since we want to sign and send the transaction separately.
That's why we want to use the _wallet.sign_ function as shown in the following code.
In summary, we get the error if we don't set gasLimit overrides option. How can we get the same _'gasLimit overrides option'_ behavior using the approach below?

const wallet = await getWallet(privateKey);
const tx = {
  ...transaction,
  gasLimit: 750000,
  nonce: transaction.nonce || (await getTransactionCount(wallet.address)),
};
const  signedTransaction = wallet.sign(tx);
const provider = await getProviderResolver();
provider.sendTransaction(signedTransaction);

[otto-mora]

Hi @karalabe,

You can see the error in a public repo that was posted by @lucachaco : https://github.com/lucachaco/etherum-node-runner

As he indicated, the issue goes away when we add the gasLimit override, however we would prefer to be able to construct the transaction separately, by first signing it and then sending it (separate components in our application do these functions). Is there a way to do that?

Thank you!!

[garyng2000]

@otto-mora

a bit confused here. This is pure client side thing, nothing to do with go-Ethereum. I am pretty sure I have done this before using Nethereum, basically construct the tx as raw data(no wallet needed, I don't even think gasLimit is needed other than may be estimate) then sign it and sendRawTransaction or something like that.

[garyng2000]

talking about gasLimit and estimate(if you do that before submit), make sure give it some room as the estimate changes from version to version and the real gas used also depends on the state at the time of execution. I used to add 50% but in 1.9.15, I have to make it like 200%.

[otto-mora]

Hello Gary and Peter,

Thank you both for your comments. After much head-scratching inside our development team, we were able to figure out that the error was caused by us using a random function in javascript to assign the NFT token id. The code in question was:

const tokenId = parseInt((Math.random() * 100000).toString(), 10);

Amazing how something so simple can cause so much chaos :-p

Lessons learned:

• NEVER trust the math.random function from Javascript V8, you can easily get collisions when frequently used
• Handle the revert reasons from the EVM better, in particular open zeppelin has a require check for the token ID that we should have been detecting better from our side.

Thank you again. Closing this issue now.

[garyng2000]

@otto-mora
glad you find it. In a related note, the 1.9.15+ of returning failed reason helps a lot for these logic related issues. At least, if you have a require(condition, 'something is not supposed to be') in the contract, at least it is easier to pin point than the previous just 'failed' and in some clients it said out of gas(and that is completely unrelated) :-)