Truffle: Using Truffle to deploy to mainnet

[skmgoldin]

Issue

Mainnet is different than a test RPC. Mainnet is different from Rinkeby. It's even different from Ropsten! Its differences cause some of Truffle's deploy logic to get in the way of being useful. Here are two things that can break in truffle publish because of mainnet's differences:

Error: Transaction 0xdeadbeef wasn't processed in 240 seconds!
Ethereum has 15 second block times, sure, but in 2018 taking advantage of those means paying a lot in gas. More often, particularly when deploying contracts, you're more likely to be willing to wait, so you can pay a low gas price. It's _expected_ that in today's congested network, some transactions will take well more than 240 seconds to succeed.

My suggestion: drop this timeout completely. Alternatively, make it user-configurable.

replacement transaction underpriced
This is a less surface-level issue whose root cause I can only speculate at, but I bet I'm right.

The uncle rate is high these days. This means it's possible that whatever node Truffle is attached to can receive a block with a transaction Truffle was waiting to see mined in it, but that block can wind up an uncle in actuality. Truffle acts on the "first contact" and goes to fetch a nonce from somewhere that has different information on the world state than the node it originally talked to (if you're talking to INFURA, as I often do, it's probable that blocks don't propagate around their nodes instantly). So Truffle sends out a transaction with a nonce that's already in the mempool, and because the transaction gas hasn't changed, this transaction gets rejected by the node for being "underpriced" (you have to up the price to do a replace-by-fee for most nodes I believe).

Here's a full example of what this looks like on the console. It's supposed to skip the contents of migration 2 for mainnet deploys, it's just choking on the save transaction which is supposed to follow:

Running migration: 1_initial_migration.js
  Deploying Migrations...
  ... 0x7977d780417552dab8ab579316274455be65a0e86a609ebf49adca32a12b6895
  Migrations: 0xa62c7f5038060dc25dc233e77ff7e69fa9d84ac7
Saving successful migration to network...
  ... 0xb84beb10b1d19d92607f05d8878f78efe3c195b7935d677d151f0a29fef033f9
Saving artifacts...
Running migration: 2_optional_for_test.js
skipping optional dev-only deploy of optional contracts.
Saving successful migration to network...
Error encountered, bailing. Network state unknown. Review successful transactions manually.
replacement transaction underpriced

My suggestion: let the user specify a block depth to wait on a transaction being mined to before sending follow-on transactions, set the default for mainnet to 10.

These things make it frustrating to use Truffle to deploy to mainnet, and it _does_ have the potential to be expensive: you have a multi-tx script that runs, but then the migration save transaction gets an underpriced error, for example.

[skmgoldin]

So this is actually really messing up my productivity! I spend 80% of my time on a testRPC and 19% on Rinkeby... But this is kinda blocking me from going to mainnet.

I've re-written my deploy pipeline to fail less hard if I get transaction underpriced errors, but to get around the 240 second timeout I'm just going to have to specify a very high gas price. I could end up paying hundreds of dollars to deploy, when I'd be happy deploying overnight with a lower gas price.

I think these two things together are kind of an emergent "bug" in Truffle given the recent rise of persistent network congestion.

[elie222]

How do you get it to skip 1_initial_migration.js when deploying to rinkeby or mainnet?

Any ways around all these issues by not using truffle? If so, what would you recommend?

[cgewecke]

@skmgoldin Thanks for pressing for a fix for this.

For anyone else running into this problem, Truffle had a bug that accidentally limited the wait time to 4 minutes. It's fixed in the latest release.

Unfortunately, while looking into this issue, it's been discovered that that web3 has it's own hardcoded wait limit: 50 blocks. (Code here).

For the moment anyway, to be mined via web3 (which Truffle uses), you'll have to set your deployment gas price high enough to get picked up within 50 blocks.

@elie222 Apparently web3 1.0 does not contain this limit. Truffle currently uses web3 0.x while 1.0 remains in beta. You could try writing your own deployment script with that? Not sure what else is available but please feel free to list anything you find here. Could be helpful to others.

Closing, thanks for reporting.

[nickjm]

@cgewecke to my knowledge web3 1.0 is not in beta, while web3 2.0 is in beta. What's the current status of this issue? Does the latest truffle (v4.0.6) still use web3 0.0? Is there a way to circumvent these timeouts now?

[cgewecke]

@nickjm Not sure how meaningful this is but if you npm view web3, every 1.x release is tagged beta. That said, it seems stable and we've started upgrading truffle-contract to web3 1.0 at truffle-contract 95.

Unfortunately I was wrong about the 50 blocks limits being removed in 1.0 - it remains in the new code here and ultimately here. In order to circumvent this we'll need to catch the error when it comes back from web3 and continue to poll the chain indefinitely if necessary. I'll put that logic into 95 next week.

Will probably be a while before 95 makes it into a release because it comes with numerous breaking changes. However, will ping you here as soon as it goes out in a pre-release build so you can test it if you're interested - would be super helpful to get some feedback on whether or not this problem is fixed.

[moda20]

I had the issue with the 240 second timeout, on transactions made by truffle contract but not eth mainnet. i had it on my local quorum network with 5 nodes. i can't understand why would a transaction have to wait in an enviroment where there is no gas at all.