Go-ethereum: Transactions not being broadcasted

[vogelito]

System information

Geth version:

Version: 1.6.5-stable
Git Commit: cf87713dd42162861b7ed227f79f0638a33571df
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.7
Operating System: linux
GOPATH=
GOROOT=/usr/local/go

OS & Version: Ubuntu 14.04.5 LTS, Trusty Tahr

Expected behaviour

eth.sendTransaction returns a hash and the transaction should be broadcasted to the network.

Actual behaviour

eth.sendTransaction returns a hash but the transaction is never broadcasted to the network.

Steps to reproduce the behaviour

This tends to happen when the network is congested (usually around large ICOs). But calling sendTransaction during this time returns transaction hashes and these seem to just get lost in the ether.

More background

We operate an ETH exchange and stopped withdrawals about 18 hours ago to prevent getting ourselves into a bigger mess and stop adding things to the queue. We had to upgrade and restart geth yesterday as 1.6.0 was refusing to sync and was losing peers. 1.6.5 is now syncing without issues.

When sending transactions we use dynamic gas prices, using the recommended value returned by eth.gasPrice.

Even after upgrading to 1.6.5 we see this behavior. Our node hasn't restarted since we upgraded to 1.6.5, we have txhashes returned by geth when calling sendTransaction, but txpool.content.pending[sending_address] returns undefined.

More questions

1. When is it safe to credit funds back to users? I understand it's theoretically possible that after calling sendTransaction the txs were broadcasted and are still in some other node's txpool
2. How do we prevent this from happening in the future?
3. What else can we do to help you debug this issue?

[holiman]

Thanks for the report.

1. It's obviously not in txpool.content.pending[sending_address]. How about txpool.content.queued[sending_address] ? If the tx is deemed non-executable, e.g. due to a nonce-gap, it would be wind up in queued instead of pending

Some answers:

1. If you sign a transaction X to a user A , with nonce N. And then "lose" that transaction, then that transaction can be used _in eternity_ , or until you create a new transaction Y that also has the nonce N, and is included in a block. The transaction Y can be to yourself, to clear that tx and 'burn' that nonce.
2. You can keep a record of nonce : transaction_info , so you know which transaction has gone missing when you notice that there's a nonce-gap keeping the other transactions from executing.
3. Provide details on how you submit transactions.
   
   
   
   • Sync or async?
   
   
   • Which rpc-method - eth_sendTransaction or eth_sendRawTransaction or personal... ?
   
   
   • Do you suffer from nonce-gaps (which you should, if transactions get lost), or not (which implies that transactions are dropped and the next one get's the same nonce).
   
   

[vogelito]

Thanks @holiman.

I can confirm that the txs are not in txpool.content.queued[sending_address]. We actually monitor the length of txpool.content.queued[sending_address] and can confirm that we never see it increase, so this doesn't seem to be a nonce-gap issue.

We use eth_sendTransaction and I _believe_ that nonces are somehow getting rewritten by geth. We do not manage nonces ourselves, we let geth manage them for us. We do the eth_sendTransaction call synchronous.

I have something that is repeatedly storing the output of txpool.content.pending[sending_address]. I have the output of 20 some transactions that appear on my dump but never made it into the blockchain. However, there are other, completely unrelated transactions, which did make it into the blockchain with the nonces of the 20 some transactions I have a record of.

Geth was not restarted during this time (pid has been up for 77305 seconds at the time of writing).

[holiman]

I have the output of 20 some transactions that appear on my dump but never made it into the blockchain.

What do you mean by "output" - the returned hash or the complete object(s) ?
Were the transactions larger than the others around it ('around' in terms of nonce-space) - I'm wondering if there was maybe too low balance for one, so it was quickly dropped, and then the next came in. Has the balance been hovering close to zero ?

[holiman]

I'd be interested in comparing one that failed (the fulll thing) with the one that replaced it. If you have the full details, you can mail one to me at martin.swende on the domain ethereum.org if you don't want to post it publically.

[vogelito]

What do you mean by "output" - the returned hash or the complete object(s) ?
Were the transactions larger than the others around it ('around' in terms of nonce-space) - I'm wondering if there was maybe too low balance for one, so it was quickly dropped, and then the next came in. Has the balance been hovering close to zero ?

By output I meant the contents of calling txpool.contents.pending[sending_address]. So the complete objects. The balance was not even close to zero.

Sent you an email with the details.

[vogelito]

Our node kept some of the transactions that have been invalidated by other transactions with the same nonce. They were obviously not mined, but they don't appear in in the txpool. @holiman I've sent you another email with these additional details.

[ucwong]

@holiman What you mean about "lose" ? like not in the queue ? And where this txn will go , will it be included by blockchain in the future?

[holiman]

@holiman What you mean about "lose" ? like not in the queue ? And where this txn will go , will it be included by blockchain in the future?

I mean, if you sign a transaction, and later on cannot find that transaction - and it isn't included in a block, but has potentially already been broadcast anyway. That tx is valid forever, or until you replace it with another transaction with the same nonce.

[ucwong]

@holiman Thank you for your reply. So what I should do is waiting, It will be included by blockchain some days, I think. right?

Nothing return with this command, anything dangerous?

[ucwong]

@holiman how to send transaction with the same nonce ?

[holiman]

@ucwong if you have a similar issue, please either enter some details ( so we know if it's the same) or open a separate ticket.

You can send a transaction with the same nonce simply by specifying nonce:
var tx = { from : eth.accounts[0], to : xxx , nonce : some_nonce} ; eth.sendTransaction(tx). But you need to know the nonce of the transaction you're trying to replace.

[ucwong]

@holiman Yes, I think I have the similar issue.
I sent eth and receive a txnId and recipient, I think it should be broadcasted to eth network. But I can't find it in blockchain.
https://etherscan.io/tx/0x638d8e91a55226a00f582e571f76cf1b08ee02ad991bf0a58254ef849a5ce46a
I can't confirm what is status of this txn now and what should I do.
btw, I only know the txnId but nonce. where can I find the nonce?

[holiman]

@ucwong I see you have a ticket at https://github.com/ethereum/go-ethereum/issues/14672 . Let's keep your info there for now, because I still think they are separate, since this ticket deals with massive number of transactions.

[ucwong]

@holiman thank you

[vogelito]

As an update:

We had roughly 70 transactions that disappeared as described above (geth returned a transaction hash, the transactions appeared at some point in the txpool then they were replaced by other transactions with the same nonce). When we send transactions through the json RPC, we send the following way (pseudo code, but all relevant bits in here):

tx = {
    "from": sending_address,
    "to": destination_address,
    "gas": 0xA410, // 42000,
    "gasPrice": eth_gasPrice(), // use whatever geth recommends
    "value": amount_to_send
};
hash = eth_sendTransaction(tx);

To reiterate, we let geth manage our nonces.

The above code is called via a single threaded queue processing system that simply monitors the queue and pops elements when/if there are any transactions waiting to be processed.

Given that we had those 70ish transactions that needed to reprocess (and that we had already built the infrastructure to check for missing ETH transactions and to record nonces, etc...), I decided to write something that would try to send those 70ish transactions as quickly as I could figure out how to via the json RPC interface of geth using an asynchronous, non-blocking, process. 70 transactions is not a whole lot, but every single transaction ended up in the ethereum blockchain without an issue.

The only times we've seen this issue is when the ethereum network is under high stress.

We start geth with the following parameters:
--rpc --rpcapi "personal,eth,web3" --targetgaslimit 1000000 --cache 1024

Happy to try some more parameters before the next ICO and report further.

[etscrivner]

Can confirm that Coinbase is seeing this as well following the upgrade to v1.6.5 and continuing into v1.6.6.

The problem is solved by restarting or re-deploying the nodes and then rebroadcasting the transactions. When the logs are examined the nodes are happily syncing and downloading blocks -
they even broadcast the occasional transaction - but for some reason the majority of transactions are going into a blackhole from which they are never broadcasted.

Happy to collect more data / logs if they would be helpful in getting to the bottom of this. Currently we see this every 1-4 days and have taken to running a group of hot secondaries to quickly recover when this issue occurs. My wild guess would be that this is somehow related to transaction volume? Hence why it doesn't appear to be very common.

[krobertson]

@holiman do you think #14737 may resolve this issue? I was looking into it at Coinbase and noticed the new logic in master, and saw v1.6.7 was just released with it. Comparing 1.6.6 with master, it seemed like it could be possible that the transaction would be accepted, but then evicted before it is processed due to either gas price or capacity when network traffic is high.

[holiman]

Yes, I have high hopes that it will :)On Jul 12, 2017 1:16 PM, Ken Robertson [e-mail] wrote:@holiman do you think #14737 may resolve this issue? I was looking into it at Coinbase and noticed the new logic in master, and saw v1.6.7 was just released with it. Comparing 1.6.6 with master, it seemed like it could be possible that the transaction would be accepted, but then evicted before it is processed due to either gas price or capacity when network traffic is high.

—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

[vogelito]

We have migrated to v1.6.7 as of this morning and will report any future findings.

Unfortunately, this morning (while still on v1.6.6) we saw geth return hashes for txs that didn't get broadcasted. However, this happened when the account had a balance close to 0, so it might be related to #14361 instead.

[krobertson]

We upgraded to v1.6.7 the day it came out and haven't seen the issue since then. We're continuing to monitor.

[vogelito]

We unfortunately had to downgrade to v1.6.6 due to #14838

[vogelito]

This issue has gotten worse as our transaction volume has increased.

[krobertson]

We've still been experiencing the issue as well. We've got in the habit of just regularly cycling our nodes.

[vogelito]

We just updated to 1.7.0 to see if this solves the issue

[raj-wadhwa]

I'm also facing the same issue. Can you please confirm if the issue is solved with the latest version?
Thanks in advance.

[phutchins]

I believe I'm hitting this same issue as well. Sending out 5300 transactions leaves a period of time where my node must stay online for all of the transactions to go out. I'm on 1.7.0. I'm getting hashes from the send but if I shut my node down too soon after, some of the transactions don't go out.

[vogelito]

We haven't seen this issue since upgrading to 1.7.0

[CockyCat]

I'm also facing the same issue.

[plavsicm]

Hello,

I think I have the same issue. I wanted to transfer my ETHs from CEX.io to Poloniex and transaction was successfully generated and confirmed on CEX. Unfortunately, it was never seen on Poloniex (even though amount is higher than 0.5 ETH which is minimum for Poloniex). Is there a way to check what happened.

This is the transaction ID:
0xf63b25137e4e1321825275f97da8573f3ed6dcf86c29564aae4cb2ac561f13b5

And on etherscan.io I can't see it.
Can you please help or give guidance what should I do?
Thank you in advance.

Best regards,
Milan

[jaycube2008]

Am having same issue here, i withdrew 11.9 ether from my wex account to my cex.io ethereum wallet last night and till now i havnt received my ethereum yet and the transaction is not showing up on blockchain, and i cross checked the address i sent it to, the address is correct but not transaction showing up on blockchain.
Duplicate of #

[jaycube2008]

Plavsicm were you able to resolve the issue?

[plavsicm]

Hi @jaycube2008 ,

I was not able to solve it, as it just happened to me two days ago.
I suggest following steps for you:

• Create tickets on both sender and recipient side (wex nad CEX)
• Try to resend/broadcast transaction with https://etherscan.io/pushTx

I was not able to broadcast transaction and I get the following error:
Error! Unable to broadcast Tx : {"jsonrpc":"2.0","error":{"code":-32602,"message":"Invalid RLP.","data":"RlpIncorrectListLen"},"id":1}

Does anyone knows why this error happens, as I could not find the reason on google? Additionally, is there any other way to try to resend transaction. I tried many ways on the web, and not sure why this error happens.
Thanks in advance.

Best regards,
Milan

[holiman]

Error! Unable to broadcast Tx : {"jsonrpc":"2.0","error":{"code":-32602,"message":"Invalid RLP.","data":"RlpIncorrectListLen"},"id":1}

That sounds buggy. If you can provide us with an example of the RLP-data which caused that, we could investigate if it's some error in the RLP decoder, or simply if the RLP (the transaction) has been incorrectly generated.

Note, though; that if you provide the RLP-data publically, and it's a valid transaction, there is a possibility that someone executes the transaction. I'm guessing that's not a problem, but just so you're aware.

[plavsicm]

If someone executes tx, he cannot change the recepient, amount, addresses etc. I assume hash that is given is final and cannot be changed, or I'm mistaken?

Maybe to send to you privately?

Thanks a lot!
Milan

[holiman]

he cannot change the recepient, amount, addresses etc. I assume hash that is given is final and cannot be changed, or I'm mistaken?

You are not mistaken. The only difference between posting via geth or publishing in a GH ticket is that you link your GH-profile to your address. It's up to you, you can send to .swende at ethereum.org

[plavsicm]

Hello @holiman ,

Thank you for the answer.
Here is the transaction ID:
0xf63b25137e4e1321825275f97da8573f3ed6dcf86c29564aae4cb2ac561f13b5

Thanks again and all the best,
Milan

[jaycube2008]

plavsicm my ETH as been returned back to my wallet. What about yours

[holiman]

@plavsicm I see. So the problem is that what you have is the tx id, which is a hash, and not the actual RLP-encoded transaction. That's why it generates error in the rlp-decoder. You cannot broadcast it yourself, with only a hash.

[plavsicm]

@jaycube2008 good to hear that. Unfortunately, for me, I still don't have my ETH in the wallet.

@holiman What is your suggestion than in this situation? What can I do (to broadcast the transaction again) or in general in this situation? It cannot be that the funds are simply gone.

Thanks,
Milan

[jaycube2008]

plavsicm, i just kept on sending email to them, but i didnt get response until i went to on their twitter page and started bombarding with messages and also found out i wasnt the only one with these issue. It is better to find their twitter page and make it public, obviously by doing this, they know the whole world is seeing and reading it.

[plavsicm]

@jaycube2008 thanks! Good advice. You were complaining on the sender, not receiver, right. I will try :)

[jaycube2008]

yes on the sender dear.

[vogelito]

@holiman the problem has returned!

geth version
Geth
Version: 1.7.3-stable
Git Commit: 4bb3c89d44e372e6a9ab85a8be0c9345265c763a
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.7
Operating System: linux
GOPATH=
GOROOT=/usr/local/go

Exact same behavior:

1. Invoked eth_sendTransaction via RPC, which returned a txid (call it x)
2. If I attach to the geth console and call eth.getTransaction('x'), I get back a transaction
3. Said transaction has not been broadcasted
4. The tx is neither on the queued or on the pending txpool:

> txpool.content.queued[eth.getTransaction('x').from]
undefined
> txpool.content.pending[eth.getTransaction('x').from]
undefined

Any thoughts on how I can help you debug this issue?

[vogelito]

All the transactions were available in the eth.pendingTransactions array. Apparently the gasPrice supplied (which was as recommended by geth's eth.gasPrice at the moment of the tx) was not enough.

Rewriting some low gasPrice tx's eliminated this issue.

I used this function:

eth.resendgwei = function (tx, gasPriceInGwei, gas) {
  if (gasPriceInGwei) {
    tx.gasPrice = web3.toWei(gasPriceInGwei, 'gwei');
  }
  if (gas) {
    tx.gas = gas;
  }
  tx.data = tx.input;
  return eth.sendTransaction(tx);
};

[LeetJoe]

@vogelito Hello I meet the same problem and it is Geth/v1.7.2-stable-1db4ecdc/linux-amd64/go1.9.

I have tried to increase the gas amount and gas price in a very high level but the transaction still can not be found in the block explorer. I doubt that the transaction is not broadcast at all.

[LeetJoe]

And I have a pendingTransactions as long as 201. So many blocked.

[holiman]

@LeetJoe The node will keep your locally generated transactions around. Other nodes will simply ignore them if they're not likely to be included anytime soon (too low gasprice). The problem is probably that you're supplying too low gas -- I don't know why you assume that they're not being broadcast.

Due to the kitty-craze, according to https://ethgasstation.info/ , the recommended 'safe low' gas price is 46 Gwei. What are you using?

[LeetJoe]

@holiman Thanks for the reply. I have checked the gas price and amount and I don't think it is low.

We have many pending transactions, some of them maybe is using a low price, but there is one using the price of 3800,000,000,000 and gas 500,000 which I think high enough and it has been in the pending queue for almost 4 hours.

Is the pending queue a FIFO list?

[holiman]

Is the pending queue a FIFO list?

No, but for any account (presumably you are sending from one account), there's ONE and ONLY ONE possible next transaction, the one with the right nonce. If you have a couple of transactions (nonce 2,3,4,5,6,7,8,9), and the one with nonce 2 has a low gp, that one will prevent the other ones from being executed.

That's why @vogelito said "Rewriting some low gasPrice tx's eliminated this issue", because he replaced the ones that were blocking the rest.

[holiman]

@vogelito why are you doing tx.data = tx.input;? Oh wow I just realised the APIs are inconsistent..

[LeetJoe]

@holiman Thanks.
It is the first time I use the eth console so I am not quite familiar with it. I only use rpc before.
It is exchange here and in my case to clear all the pending queue maybe is the best choice. I will search the document for it.

[holiman]

@vogelito https://github.com/ethereum/go-ethereum/issues/15628

[LeetJoe]

@holiman

I tried vogelito's method but this does not work for me. I set the gas and gasPrice like this
gas: 600000, gasPrice: 60000000000,
but the pending transaction is still there for about half a hour.
I have confirmed that the transaction has the minimum nonce number in the list.
Is it still too low?

[LeetJoe]

And I get an error message saying
Failed to journal local transaction err="no active journal"
And the numbers of this error is the same as the blocked transaction numbers.
I googled the problem but few answers found.
Anybody meet the same problem?

[holiman]

I can't say, without more info.. However, a gas at 600K is pretty large, which means less chance of getting it in ( since it takes the space of 28 regular transactions).

[holiman]

@LeetJoe transaction journalling is only enabled if you specify txpool.journal=<file>. It means that local (your own transactions) are saved to disk, so they survive restarts.

[LeetJoe]

@vogelito 's method works. It seems I need to wait longer.
Thank you very much. @holiman @vogelito .

[vogelito]

Thanks for filing the additional bug report @holiman

[vogelito]

Hi - any ideas why this would happen?

> eth.gasPrice
21000000000
> eth.pendingTransactions[0].gasPrice
51000000000

That pending transaction with gasPrice of 51GWei has been sitting there for 5 hours doing nothing...

I had to get the raw transaction and push it via etherscan

[turk-fx]

I have this same issue. I got the TIX, but transaction does not showup in the block. Is there anyway to fix tix?

Tix:
0x1ed20c585d05e15fde11e7f4f6be94ca51dc00de0c1cffd58aca38b397e58a8d

and tix
0xa6bf48310c77872f79dea67a825cc0b9f89f915f555408c5301aafdd9914355a

[jac2]

Your TX has been broadcast to the network. This does not mean it has been mined & sent. During times of extreme volume, it may take 3+ hours to send. 1) Check your TX below. 2) If it is pending for hours or disappears, use the Check TX Status Page to replace. 3) Use ETH Gas Station to see what gas price is optimal. 4) Save your TX Hash in case you need it later: 0x2cfb46e3d21f8861962e493ed71f17b63a79629a25c1ca3c43ed6586aa93889b

https://etherscan.io/tx/0x2cfb46e3d21f8861962e493ed71f17b63a79629a25c1ca3c43ed6586aa93889b

Initially when I clicked this link, I can see the transaction is pending. There after, it is none to say "not found".

How to check if this transaction is returned back to my account? Or if it is returned, it won't show this transaction at all? I can't find this transaction in my eth wallet but I can't remember how much ETH I have before sending this.

[wired00]

Hi Possibly a stupid question, but I've just created a transaction and its sitting in eth.pendingTransactions.

Its Tx: 0xfcc8dfc04e96959e50495e69f148c08597f98c781be742c9819e26699a422fd4

gas: 21000,
gasPrice: 4830000001,

i used geth --maxpeers 0 console to start geth without syncing the blockchain. Is this ok? Or do I need to allow sync mode in order to send out my transaction?

Also, can I merely now start via geth console? Will my pending transaction then auto send out once the blockchain is full synced? (Its been a couple weeks since i last synced)

Thanks for any help

EDIT: Oh, actually its now appearing in blockchain explorer as pending confirmation. Even though i haven't synced blockchain to my machine yet. https://etherscan.io/tx/0xfcc8dfc04e96959e50495e69f148c08597f98c781be742c9819e26699a422fd4

[xenhim]

I really do not understand what the correction is here. But it is not found a block. 4 days have passed. I used a nicehash test up to 550gb / s for 5 hours. still can not find a block. What the hell is going on?

[holiman]

I really do not understand what the correction is here. But it is not found a block. 4 days have passed. I used a nicehash test up to 550gb / s for 5 hours. still can not find a block. What the hell is going on?

I don't understand a single bit about what problem you are having. Syncing? Mining? Getting transactions into blocks?

[achillesss]

@vogelito

eth.resendgwei = function (tx, gasPriceInGwei, gas) {
  if (gasPriceInGwei) {
    tx.gasPrice = web3.toWei(gasPriceInGwei, 'gwei');
  }
  if (gas) {
    tx.gas = gas;
  }
   // this should be: tx.data = tx.data, or be removed
  tx.data = tx.input;
  return eth.sendTransaction(tx);
};

[vogelito]

@achillesss take a look at https://github.com/ethereum/go-ethereum/issues/15628

[axcrest]

I also have the problem where I send a transaction, get back a hash and the transaction just gets lost somehow. I am running this on the Rinkeby test network. The tx hash returned is 0x5240d8c2b9f57d01a807eae0214134b175b454ce22190f167c26576175483455 and the nonce for it is 182064.

We are using sendRawTransaction. I did notice in my logs that another transaction was given the same nonce, that hash was good and successful.

How is it possible that I received a txhash for my signed transaction that just disappeared???

[Aesmel]

I also have the problem where I send a transaction, get back a hash and the transaction just gets lost somehow. I am running this on the Rinkeby test network. The tx hash returned is 0x5240d8c2b9f57d01a807eae0214134b175b454ce22190f167c26576175483455 and the nonce for it is 182064.

We are using sendRawTransaction. I did notice in my logs that another transaction was given the same nonce, that hash was good and successful.

How is it possible that I received a txhash for my signed transaction that just disappeared???

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[quantumproducer]

I'm still having this issue, both smart contract deployments & sending ether are failing.
https://github.com/trufflesuite/truffle/issues/677

[ryanschneider]

@holiman I believe I've found the source of this issue: the sendLock channel that is acting as a "semaphore" in feed.go is not FIFO, so if there's a steady stream of promoteTx calls, some of the go routines spawned via go pool.txFeed.Send(TxPreEvent{tx}) keep getting supplanted by newer calls to Feed.Send(). On busy nodes I've seen goroutines spawned from promoteTx stay blocked on f.sendLock for minutes to hours at a time..(!)

I'll be spending this upcoming week focusing on this issue starting tomorrow, so will hopefully have more to report in the next couple days.

Also potentially related (and where unfortunately I spent all my time on Friday): in ProtocolManager.txBroadcastLoop() the BroadcastTx function is a blocking function (there's a blocking select in protoRW.WriteMsg). This is what I originally thought was causing the issue, but I no longer think that's the case (but still might want to be fixed: my 100% hacky solution was to add a default to the select, but even after that I still see jammed-up promoteTx calls under load).

[qaaa9]

Yes
E

[ryanschneider]

Hmm, so during my testing I wasn't able to come up with a scenario where I could force the goroutines in promoteTx to stay waiting on the f.sendLock for long times like I see under production loads. I did confirm that the sendLock isn't released in FIFO order, but even then I couldn't cause enough contention to keep the txs from being promoted for minutes like I saw last week.

However, I've added 3 different commits to a test branch that I would love feedback on. They all help mitigate the issue in different ways, but aren't a 100% solution:

https://github.com/ryanschneider/go-ethereum/commit/b70b704e155414fc2c950a2e65f7c01cdfbba35c - this does the txFeed.Send calls from inside loop instead of via go txFeed.Send like before, using an intermediate channel. since making this change and testing it on a relatively high load server for a couple hours I haven't had any issues w/ transactions being promoted.

https://github.com/ryanschneider/go-ethereum/commit/5f03bf6b42daf052172de010dc78adc6328fd78a - this waits at most 300ms in WriteMsg in the devp2p layer for the write channel to become ready. I fully admit I picked the 300ms cutoff more or less at random, but during my testing last week I saw that a slow peer could cause issues with ProtocolManager.BroadcastTx().

https://github.com/ryanschneider/go-ethereum/commit/66328c77984beba34ce4050b95a1f4157c2c14a9 - This is probably the most controversial of the commits: it adds a --txpool.noremotes option to avoid adding remote txs from the dev p2p layer to the txpool (so only txs from RPCs are added to the pool). During my testing this had the highest effect on tx propagation, so it allowed a node dedicated to sending txs perform well without having to tune the pool. However, this does prevent the node from propagating p2p txs, so I'm not convinced it should actually be upstreamed.

Anyways, I'm testing different combinations of these 3 commits for performance. As I said, none of them directly address the "soft lock" on <-f.sendLock, but they do work around it various ways.

I also realized today that the metrics/monitor system is working great again (it was broken in 1.7.x), so I'll probably add some metrics around tx promotion to help determine what sort of load is needed to get the promoteTx calls to hang.

[kristieguo]

@ryanschneider @holiman Hello, my team has developed an exchange custody product which provides ETH deposit and withdrawal. We use geth eth.getTransaction(hash) and eth.getTransactionReceipt to get transaction details, use eth.sendSignedTransaction to send raw transaction, and use eth.getGasPrice to get suggested gas price. On main net we usually encounter "unable to locate transaction hash" on etherscan. If we query this hash using eth.getTransactionReceipt(hash), it would just return null. Approximately this problem occurs 1 time out of 100 withdrawals. The fact is even we waited a few days the transaction still cannot be located. BTW, we are using geth 1.8.6 version. Since this problem affects our product behavior and operational strategy, I have the following questions:

1. What is the possible reason of this problem?
2. How to avoid this problem?
3. Could this problem be prevented from geth?
4. Should we resend the transaction?
5. If so, how long we should wait before resend the transaction?
6. What if the old transaction hash becomes valid?

I appreciate any help. Thank you.

[quantumproducer]

I read somewhere about a 50 block limit, that geth stops trying to publish contracts after 50 blocks?

[ApsOps]

I've hit this multiple times as well.

Geth logs say:

INFO [05-09|13:53:25] Submitted transaction  fullhash=0xREDACTED recipient=0xREDACTED

But this txid doesn't show up on etherscan.

It's also _not_ present in txpool.content.pending[sending_address] or txpool.content.queued[sending_address].

/ # geth version
Geth
Version: 1.8.7-stable
Git Commit: 66432f3821badf24d526f2d9205f36c0543219de
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.10.1
Operating System: linux
GOPATH=
GOROOT=/usr/local/go

[vogelito]

They should be present in eth.pendingTransactions.

Our current fix for this is to get the list of pending transactions and manually relay them to some of our other nodes.

[ApsOps]

@vogelito I checked eth.pendingTransactions as well. They're not present.

Is there a possibility that these pending transactions are cleared when geth node is restarted?

[vogelito]

Perhaps. The safest thing to do, in my opinion, is to resubmit your transaction by specifying the nonce that your lost transaction should've had.

[adamschmideg]

@vogelito are you still having the original issue?

[vogelito]

@adamschmideg unclear to be honest. We have a secondary parity node and a script that relays transactions to it.

This has been working fine for months but i’d be happy to turn it off and monitor for any problems

[adamschmideg]

@vogelito I"m going to mark this issue as a nice to have since it's not critical for you and not many other users reported it as a problem. Please, let me know if you disagree.

[vogelito]

Sounds good to me. Thanks

[fjl]

Related: https://github.com/ethereum/go-ethereum/pull/19705

[vkuznecovas]

Can confirm that I'm also encountering this issue. We're currently in test phase on goerli but occasionally are seeing that transactions do not get broadcast to the network.

I'm using go-ethereum/ethclient sendTransaction method.

I'm getting back a transaction hash, but unable to find the transaction afterwards. They just seem to vanish into thin air.

It seems to be related to congestion on the network as it fails with higher probabilities during times of heavy congestion(when the transactions that do appear on the network tend to stay in a pending state for quite a while) although this is not a guarantee.

Also seems more likely to happen if I'm making quite a few transactions in a short period of time but can not reliably reproduce.

[developer2belfrics]

@holiman The transaction is sent to the geth server. I confirmed by eth.getTransaction(hash). from eth.getTransactionReceipt(), i got null. Even I checked in txpool.content.queued[sending_address] and txpool.content.pending[sending_address] but the return response is undefined. Can you please tell me what went wrong? There are two transactions made on feb 14th and 20th but still not posted to the network.

[pblab-dev]

Some erro here.

[AtomicLemon]

Having exactly this problem. #21167

[rjl493456442]

It's been fixed in v1.9.16

[coinwalletdev]

-- (moved to https://github.com/ethereum/go-ethereum/issues/21385#issuecomment-726620223)