Web3.js: How to reliably send signed transactions programmatically?

[woniesong92]

x-posting from https://stackoverflow.com/questions/49144533/sending-offline-transactions-to-mainnet-is-flaky.

What's the best way to sign and send 100 transactions? So far, I've tried three different methods:

• Remix
• MEW
• Ethereum-tx

Remix works more reliably than MEW, but there's a critical bug where it doesn't accept array as an argument (e.g. uint[6]). MEW doesn't have this bug, but you cannot send multiple transactions at once. Subsequent transactions get signed with the identical nonce and you're forced to wait until the first transaction gets confirm for nonce to increment.

So I resort to programmatically signing and sending offline transactions. I assumed it would be the best solution, but there are some interesting caveats:

1. Pending transactions do not show up in etherscan so you have no way of knowing whether your transactions are being processed or not
2. Transactions seem to fail randomly! Even after you test that sending hundreds of transactions to the same contract on Ropsten works perfectly, reproducing the same behavior on mainnet is nearly impossible. For example, if I tried sending 100 transactions continuously, only the first 27 transactions would be confirmed, and the subsequent ones would fail for no reason.

This is my code snippet:

import Web3 from 'web3'
import EthereumTx from 'ethereumjs-tx'

const web3 = new Web3(new Web3.providers.HttpProvider(INFURA_URL))
const Contract = new web3.eth.Contract(ABI, CONTRACT_ADDRESS)

const createItem = (name, price, nonce, callback) => {
  console.log(`Nonce: ${nonce}. Create an Item with Name: ${name}, Price: ${price}`)

  const data = Contract.methods.createItem(name, price).encodeABI()
  const tx = new EthereumTx({
    nonce: nonce,
    gasPrice: web3.utils.toHex(web3.utils.toWei('4', 'gwei')),
    gasLimit: 400000,
    to: CONTRACT_ADDRESS,
    value: 0,
    data: data,
  })

  tx.sign(new Buffer(MAINNET_PRIVATE_KEY, 'hex'))

  const raw = '0x' + tx.serialize().toString('hex')

  web3.eth.sendSignedTransaction(raw, callback)
}

export default createItem

Can someone shed light into this? I would appreciate any guidance on this issue.

[gburtini]

I know this is a workaround - but, you could ostensibly do what you're trying to do with a smart contract method that supports creating multiple items at once.

How are you setting nonce?

[jackson-sandland]

I'm running into the same issue. I saw something about an increment method that you can call to change the value of the nonce. It would be great if you could send off multiple transactions, and not have to wait for the first to clear, before being able to send again.

[s-a-y]

👍
Always have the same issue, running my own geth nodes, but can't get it to work reliably. Some transactions are submitted successfully after 10 min. some other transactions are not submitted after an hour of trying.

[gburtini]

@woniesong92 @s-a-y.

You basically cannot reliably do what you guys are trying to do. Here's a set of problems that arises:

• Transactions only succeed in nonce order. If an early nonce is pending (or worse, went missing), the other transaction will pend until the early nonce has been consumed (or it gets removed from the mempool).

• There's no hard rule for when a transaction will drop from the mempool. This is scary when you've made a nonce error or an intermediary nonce has not reached the network for some reason because you don't know what is going to happen when you finally post that nonce.

• Many transactions with the same nonce can be sent. They are very likely to be prioritized by highest gas price (because miners are incentivized to do exactly that). One useful trick when something strange has happened is to clear out your nonces by sending a bunch of high gas price, zero value transactions. You might call this an increment method, @jackson-sandland. Remember this comes with a cost of 21000 * gas price. A higher gas price probably makes this "safer," so the cost is proportional to your safety requirement.

• A lot of tools do one of two things to handle nonces: a live read from getTransactionCount() or a read from getTransactionCount() followed by incrementing for each additional transaction you send. Neither of these work reliably: for the first, transactions are sometimes unconfirmed (or worse, pending but not visible in the pool) yet. This seems to especially happen if gas price is below safemin, but I'm not entirely sure what is happening here. For the second, if any other system sends a transaction with the same address, it will not work.

So, how do we work around this?

1. A smart contract is a straightforward way to reduce a transaction from many sender nonces to few sender nonces. Write a contract that sends all your different transactions, then send the budget to that contract. This is a relatively high cost (in terms of time/effort/expertise) way to solve the problem.

2. Just do it anyway, batch style. When I've had to send many transactions manually, I've batched them in to sets of 10 or so and gone for it. Incrementing the nonce manually each time (because the transaction is usually not on the network yet) and then waiting sufficiently long for all the transactions to _confirm_. Don't rely on pending transactions on Etherscan or similar to determine whether this is working, as things often vanish unpredictably from this level. Never reuse a nonce for a different transaction that isn't a 0ing high gas transaction - you will screw it up and you'll end up sending the same transaction twice by mistake.

3. Serialize. One-by-one you post a transaction, wait for it to confirm, increment your nonce. This is probably the best of the easy-to-implement automated solutions. It will not fail. It might buffer forever if you have a constant stream of transactions. It also assures you can never have more than one transaction per block, limiting your throughput to 4 a minute or so.

4. Fire and retry. This is a little sketchy because it involves reusing nonces for different transactions. Send all (or some large batch) of your transactions to the network. If any fail, restart from the failure nonce and send again. There's possibly a more intelligent solution where you just try to swap out the missing nonce. You'll need to be very careful you never send a transaction that is secretly in the pending-but-not-visible pool.

5. New address for every transaction. A buffering step of distributing funds to your own addresses ensures you never screw it up for other people. It does double your transaction time and cost though.

I think some variant of 4 is what most of the big services (like pools and exchanges) do. Some of these can be improved by splitting the budget over a few addresses as available (reducing the collision frequency). Hope that helps. Let me know if you come up with anything better.

[sreerajs]

Having the same problem. Since the nonces are sequential, the nonce overwriting, nonce gaps etc are very problamatic. getTransactionCount only return confirmed transactions and transaction pool methods to check for pending transactions are not working for me. Even if it worked, what if a pending transaction does'nt get confirmed. Can anyone help out

[marcosmartinez7]

@sreerajs what about this?

  web3.eth.getTransactionCount(address, "pending", function (error,result){
                console.log("Nonce "+result);
            });

the "pending" parameter includes also the pending transactions

[nivida]

This question should be on https://ethereum.stackexchange.com/