Go-ethereum: eth_getTransactionCount does not include pending transactions when 'pending' qualifier given

[drewshaver]

As the title states, tx count is not getting incremented for pending transactions when giving second argument of 'pending'. It seems to be treated the same as 'latest'.

System information

Geth version: 1.4.10
OS & Version: OSX

> web3.eth.getTransactionCount("0xf82e...", "pending");
5
> web3.eth.sendTransaction({from: "0xf82...", to: "0xf1c...", value: 42000000000000000});
I0624 15:57:41.848826 eth/api.go:1193] Tx(0xbd094a59eb8f05653f35fa93a9254db95bc6b9b5bdd3b95aedda27bb781545f9) to: 0xf1c...
"0xbd094a59eb8f05653f35fa93a9254db95bc6b9b5bdd3b95aedda27bb781545f9"
> web3.eth.getTransactionCount("0xf82...", "pending");
5 (should be 6)
> I0624 15:57:59.315075 miner/worker.go:337] 🔨  Mined block (#4529 / 7788e873). Wait 5 blocks for confirmation
I0624 15:57:59.315639 miner/worker.go:555] commit new work on block 4530 with 1 txs & 0 uncles. Took 529.458µs
I0624 15:57:59.315664 miner/worker.go:433] 🔨 🔗  Mined 5 blocks back: block #4524
I0624 15:57:59.315989 miner/worker.go:555] commit new work on block 4530 with 1 txs & 0 uncles. Took 310.783µs
> web3.eth.getTransactionCount("0xf82...", "pending");
6

[fjl]

This is critical for the upcoming Go API as well. Maybe getTransactionCount(..., "pending") should return the transaction pool's latest known nonce instead of looking at the pending block.

[bas-vk]

Are you sure you used geth 1.4.10?

> eth.getTransactionCount(eth.accounts[0])
5
> eth.getTransactionCount(eth.accounts[0], "pending")
5
> eth.sendTransaction({from: eth.accounts[0], to: eth.accounts[1], value: 1})
"0xd3546397a3b0c1fb0a6c47364a66b1841f4ebffb15f58bb5c8d0e1f453108e59"
> eth.getTransactionCount(eth.accounts[0])
5
> eth.getTransactionCount(eth.accounts[0], "pending")
6
> miner.start(1)
true
... tx mined ...
> eth.getTransactionCount(eth.accounts[0])
6
> eth.getTransactionCount(eth.accounts[0], "pending")
6

Geth will grab the nonce from the pending state if you ask for it.

[murrekatt]

Same issue here. Looking at the code in eth/api.go here there is no mention of pending.

[bas-vk]

stateAndBlockByNumber is called which gets the pending state from the miner.

[fjl]

We will change it to look at the transaction pool because it knows about the latest nonce faster.

[murrekatt]

@fjl that sounds very good.

I'm testing out the Go bindings and in the example here there seems to be some races to be aware of due to the sleep:

address, tx, token, err := DeployToken(auth, backends.NewRPCBackend(conn), new(big.Int), "Contracts in Go!!!", 0, "Go!")
if err != nil {
    log.Fatalf("Failed to deploy new token contract: %v", err)
}
fmt.Printf("Contract pending deploy: 0x%x\n", address)
fmt.Printf("Transaction waiting to be mined: 0x%x\n\n", tx.Hash())

// Don't even wait, check its presence in the local pending state
time.Sleep(250 * time.Millisecond) // Allow it to be processed by the local node :P

name, err := token.Name(&bind.CallOpts{Pending: true})

I'm doing two transactions from the same account after each other and right now the nonce is the same for both which means once tx will be invalid once the other is mined.

How will the concurrency work after the change to use the transaction pool? Will multiple transactions from the same account increment the nonce without any tricks with sleep etc.?

Also when is this change planned to hit develop?

[fjl]

I cannot give you a timeline but it will be included in the next major release.

[murrekatt]

OK, thanks @fjl.

I think the Go bindings are really cool and was also wondering a bit how they were meant to be used in various situations e.g.:

1. single account calling multiple functions on a single contract (resulting in transactions)
2. single account calling multiple contracts (resulting in transactions)

In addition, the two above could be done concurrently from e.g. different goroutines. As the underlying mechanism from the calling go code right now calls out to the backend over RPC to get the transaction count or later the tx pool nonce there is a race. Hence, I assume concurrent transactions for the same account is not supported. Right?

However, for 1 & 2 it would be possible to keep track of the nonce in the client code, maybe in the TransactOpts by the underlying backend? Or something along those lines so that a caller using the same account would automatically get the nonce incremented without a need to ask over RPC. This opens of course up other points such as keeping the local value in sync. Has this been thought about? I'd be interested to understand what's been considered.

[Nexus7]

@fjl : Just checking if this change to look at the transaction pool for the nonce was made yet as obviously this issue is still currently open here...

[nickluijtgaarden]

I am seeing that this issue is still open. Does this mean this is not fixed?

[doppio]

@fjl any update on this? Is this change still going to happen?

[mcdee]

Any progress on this? It seems like a critical issue for those of us attempting to generate multiple transactions with Go, and still has a milestone tag of two major releases back.

[chatch]

There is a related issue in Parity (https://github.com/paritytech/parity/issues/5014) with "pending". Parity has a non-standard RPC call parity_nextNonce to get the nonce considering all pending transactions in the pool.

Maybe this could be added to the standard JSONRPC API. Something like eth_nextNonce. Or the suggestion on the issue of using "pending_mining" or "pending_inclusion" could work too.

Then this getTransactionCount with "pending" can remain as meaning "the currently mined block (including pending transactions)". ie. pending transactions in a block being mined.

[yondonfu]

At the moment I'm doing some local testing with this change to GetTransactionCount in internal/ethapi/api.go to ask the transaction pool for the nonce if the the transaction count is asked for a pending block:

// GetTransactionCount returns the number of transactions the given address has sent for the given block number
func (s *PublicTransactionPoolAPI) GetTransactionCount(ctx context.Context, address common.Address, blockNr rpc.BlockNumber) (*hexutil.Uint64, error) {
    if blockNr == rpc.PendingBlockNumber {
        nonce, err := s.b.GetPoolNonce(ctx, address)
        if err != nil {
            return nil, err
        }

        return (*hexutil.Uint64)(&nonce), nil
    } else {
        state, _, err := s.b.StateAndHeaderByNumber(ctx, blockNr)
        if state == nil || err != nil {
            return nil, err
        }

        nonce := state.GetNonce(address)

        return (*hexutil.Uint64)(&nonce), nil
    }
}

Thoughts on if this is a reasonable solution?

This issue seems to still be a problem for anyone using abigen to submit concurrent transactions since transact in https://github.com/ethereum/go-ethereum/blob/master/accounts/abi/bind/base.go#L176 uses PendingNonceAt to retrieve the pending nonce which I believe uses GetTransactionCount

[davux]

@fjl (2016-08-06)

I cannot give you a timeline but it will be included in the next major release.

When is the next major release? It's been a year and a half and the bug is serious and still present.

[dshulyak]

@yondonfu this is what @fjl suggested some time ago in this thread. In my opinion, you should submit PR with your change

[winteraz]

Any chance to have this fixed this year(2018) ?

[vernon99]

I wonder, how can it still open? How people deal with a need to send multiple transactions from the same account without storing the nonce (super fragile and can lead to transaction failures or overrides)? Do I miss something or this is a must have for any normal project?

[gavinmarkglynn]

This seems like to be too serious of bug to be left hanging? Is there any possibility of a fix? This issue will cause stability problems for any general application using Ethereum smart contracts.

[vernon99]

Btw, for now it seems we found a workaround by accessing node's pool using txpool api, fetching pending transactions and adding the number to eth_getTransactionCount. But yeah, this is a super crappy hack since you have to do extra fetch of a massive size on any transaction. Should be fixed on a node level!

[drewshaver]

@vernon99 just fyi, there's a potential race condition in that workaround if the node state changes between calls. (i think)

[vernon99]

@drewshaver yeah, agree that very suboptimal. Don't see a better way as of now. Cray.

[gavinmarkglynn]

Hi

Thanks for the help.... Is there any reason why the maintainers of geth
have not wanted to sort this issue? As an enterprise developer of over 20
years experience, this seems like a real stability issue.

Gavin :)

On Sat, Mar 10, 2018 at 1:30 PM, Mikhail Larionov [e-mail]
wrote:

@drewshaver https://github.com/drewshaver yeah, agree that very
suboptimal. Don't see a better way as of now. Cray.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ethereum/go-ethereum/issues/2880#issuecomment-371999117,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKq24Ix4Pdv7LInHDdsIfttvt8ehQ55Gks5tc0jmgaJpZM4JZIDD
.

--

Gavin Glynn
Senior Software Engineer
www.metavine.com

E * [email protected]
*T [phone]
S Metavine.Australia
M GPO Box 1793, Brisbane 4001, Queensland, Australia

This email may contain "Commercial-In-Confidence" information. If you are
not the intended recipient, please delete this email immediately and notify
the sender by return email. Thank-you for your assistance.

[wupeaking]

If this problem is not fixed, is there any other way to send multiple transactions at the same time using the same account?

[gavinmarkglynn]

Hi

In the end, I implemented a queuing system (based on an npm module called
"better-queue"), that ensures only one transactions is submitted per from
address at one time. The serially queued process would form the
transaction, using the eth.getTransactionCount(, 'pending)
command for the nonce, and wait for the transaction receipt. I also
included some retry in the mix, just in case the nonce is knocked out by a
transaction from the address in question, which is outside of my system.

This all worked for me and seemed to work fine when I slammed it with
transactions.

Regards

Gavin :)

On Tue, Apr 17, 2018 at 4:41 PM, wupengxin [e-mail] wrote:

If this problem is not fixed, is there any other way to send multiple
transactions at the same time using the same account?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ethereum/go-ethereum/issues/2880#issuecomment-381865459,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKq24FFOmf-qSwsr0AMB8yzfPfvGq74fks5tpY6XgaJpZM4JZIDD
.

--

Gavin Glynn
Senior Software Engineer
www.metavine.com

E * [email protected]
*T [phone]
S Metavine.Australia
M GPO Box 1793, Brisbane 4001, Queensland, Australia

This email may contain "Commercial-In-Confidence" information. If you are
not the intended recipient, please delete this email immediately and notify
the sender by return email. Thank-you for your assistance.

[gavinmarkglynn]

Hi

As I mentioned before, in seems like a deficiency in the geth code to not
handle this scenario.

Regards

Gavin Glynn :)

On Tue, Apr 17, 2018 at 4:49 PM, Gavin Glynn [e-mail] wrote:

Hi

In the end, I implemented a queuing system (based on an npm module called
"better-queue"), that ensures only one transactions is submitted per from
address at one time. The serially queued process would form the
transaction, using the eth.getTransactionCount(, 'pending)
command for the nonce, and wait for the transaction receipt. I also
included some retry in the mix, just in case the nonce is knocked out by a
transaction from the address in question, which is outside of my system.

This all worked for me and seemed to work fine when I slammed it with
transactions.

Regards

Gavin :)

On Tue, Apr 17, 2018 at 4:41 PM, wupengxin [e-mail]
wrote:

If this problem is not fixed, is there any other way to send multiple
transactions at the same time using the same account?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ethereum/go-ethereum/issues/2880#issuecomment-381865459,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKq24FFOmf-qSwsr0AMB8yzfPfvGq74fks5tpY6XgaJpZM4JZIDD
.

--

Gavin Glynn
Senior Software Engineer
www.metavine.com

E * [email protected]
*T [phone]
S Metavine.Australia
M GPO Box 1793, Brisbane 4001, Queensland, Australia

This email may contain "Commercial-In-Confidence" information. If you are
not the intended recipient, please delete this email immediately and notify
the sender by return email. Thank-you for your assistance.

--

Gavin Glynn
Senior Software Engineer
www.metavine.com

E * [email protected]
*T [phone]
S Metavine.Australia
M GPO Box 1793, Brisbane 4001, Queensland, Australia

This email may contain "Commercial-In-Confidence" information. If you are
not the intended recipient, please delete this email immediately and notify
the sender by return email. Thank-you for your assistance.

[nanmu42]

Parity has a parity_nextNonce method to resolve this problem.

Personally, I am really looking forward to seeing Geth get something similar.

[masoodtalha]

This is a serious bug. Its been 2 years already and issue is still open. Is there any person from Geth team who can tell if there is any specific reason that its been ignored for this long period of time?

[mlensment]

+1

[gavinmarkglynn]

HI All

As an update. My queuing strategy broke down when sending large number of transactions at a time (~2000 at once).

I don't program in go, but it sounds like I am going to have to break out the compiler and try and fix this issue myself. I have to say, as a professional developer of 20 years experience, it is extremely poor form of the maintainers of geth to leave this issue unresolved. It shows they are not serious about Ethereum becoming a serious, business grade platform.

Regards

Gavin :)

[mlensment]

@gavinmarkglynn This is concerning. I'm just tackling this issue myself and I was considering to use your strategy.

[mcdee]

I'm not sure that any system other than keeping track of the nonce yourself is going to be 100% successful. There is no guarantee that any given node will either accept or retain all of your pending transactions so could always return an incorrect value regardless of how the underlying code is changed.

[aleczadikian]

It breaks my sending most of the time. Send one unconfirmed transaction, and your nonce is wrong for a long time.

[gnuser]

Too disappointment for this issue still remains.

[aleczadikian]

For whatever reason, transactions don't even get broadcasted to the network if the chosen gas price is too low, so the nonce doesn't advance. Not sure if this is my Infura node or something else down the chain (I'm less familiar with Ethereum than with Bitcoin).

I've found that sending with the gas price recommended by https://ethgasstation.info makes this a non-issue. But it's sometimes very expensive, changing by 10X within a week, and it's very unfortunate that I can't just submit with a low gas price and have it go through later like how Electrum + Bitcoin does.

Edit: I've learned that this is all due to the nature of Ethereum's balance model, different from Bitcoin's UTXO model. There advantages to it, but here is one disadvantage. If you're having problems, I'd recommend reading up on that to understand why.

[juanfer]

It's a shame this issue is still opened after 2 years. Finally I faced the problem creating a queu of transactions in database to be able to send several transactions from the same address, one after the former was mined. Thanks @gavinmarkglynn for the idea; fortunately I don't have to send such number of Tx from the same address at once...

[ashispoddarac]

I'm quite sure, developers have solved it in client side where all the txn flow through single system (payments out)...Is there any design pattern for that?

[seik]

@ashispoddarac here is our approach (some things are simplified):

confirmed_transactions = w3.eth.getTransactionCount(from_address)

try:
    pending_transactions = len(w3.txpool.inspect.pending[from_address])
except KeyError:
    pending_transactions = 0

last_recorded_transaction = get_last_transaction(from_address)
theoric_nonce = confirmed_transactions + pending_transactions

if last_recorded_transaction and last_recorded_transaction.nonce == theoric_nonce:
    return theoric_nonce + 1
else:
    return theoric_nonce

The thing is, even with this, if a transaction does not go through, a gap in the nonce is created and at that point the system is just desynced.

[enVolt]

I'm on geth 1.8.4-unstable-45bd4fed.

For getting the Nonce, I'm using getTransactionCount(address, "pending"), if I try it on geth console, it is working as expected. But in my application I can see frequent error that "replacement transaction is underpriced" or seeing that transactions are getting dropped (probably getting replaced). For me it's an sporadic issue.

[zinxer]

Still getting this error in Version 1.8.9-stable-ff9b1461:

Error: Returned error: Known transaction

With following call in nodejs code:

web3.eth.getTransactionCount(<fromAddr>, "pending")

[alexyalunin]

I solved the problem adding globalNonce variable to code

var globalNonce = 0; #global variable
...
var nonce = web3.eth.getTransactionCount(owner, 'pending');

  if (globalNonce == 0) {
      globalNonce = nonce;
  } else if (nonce <= globalNonce) {
      globalNonce += 1;
      nonce = globalNonce;
  } else {
      globalNonce = nonce;
  }

[flute]

@alexyalunin This truly is a solution, but maybe two problems.

1. After your application restart, you will lose globalNonce status. Use redis may solve this.
2. Your code as below:

else if (nonce <= globalNonce) {
      globalNonce += 1;
      nonce = globalNonce;
}

If i have made three concurrent transactions, nonce as 1,2,3. GlobalNonce should be 3. then i make forth transaction, nonce will be 4, but if first transaction failed by some reasons. use getTransactionCountget nonce will be 1. You need to use nonce 1 directly, or your other transaction will always wait until nonce 1 used.

[aleczadikian]

Yes, you don't know whether your last transaction succeeded and can't just increment regardless because nodes won't mine transactions after a nonce gap until the gap is filled (why?). I haven't found a flawless way to handle this other than waiting for each transaction to be included in a block before submitting another transaction, which of course is very slow. Maybe you can make this fast by hosting your own full Ethereum node, which I don't understand well enough.

[oakcookies]

would this get 'pending' nonce still be correct if there are nodes behind a load balancer like infura ? Not sure if the nodes share the same mem pool or not.

[enVolt]

A pending transaction is immediately relayed to peer nodes (You can see your transaction as Pending almost immediately on Etherscan).

So it should work fine. You can ask on their Gitter channel

[pstehlik]

@oakcookies @enVolt Even though the nonce would be relayed "quickly" it would not really support a situation where you want to load a _lot_ of transactions at once. The different nodes behind the load balancer would have different states and you would quickly run into race conditions.
This would only work if you have a nonce in some kind of system like redis (like @flute says). Then put all the nodes behind the load balancer into a single system to keep the nonce on redis.
The general user-need is that they want to submit transactions in quick succession before they even hit the Ethereum network so some kind of shared state is required if you want to run multiple nodes in that (load balanced) setup.

[sgmheyhey]

$ geth version
Geth
Version: 1.8.11-stable
Git Commit: dea1ce052a10cd7d401a5c04f83f371a06fe293c
Architecture: amd64
Protocol Versions: [63 62]
Go Version: go1.10
Operating System: linux

> eth.getTransactionCount(eth.accounts[0], "pending")
10
> eth.getTransactionCount(eth.accounts[0])
6
> eth.getTransactionCount(eth.accounts[0], "pending")
10
> eth.getTransactionCount(eth.accounts[0])
10

Does it sometimes work? 8-o

[enVolt]

This is for ropsten testnet -

> admin.nodeInfo
{
  enode: "enode://df090c3d10b4e485e4e4f85eca81ed3acf7ba1f17a0e18d505aad84f4be4d044ffffac2a10d35791dd8b72a2d328bca5305ec7591e8aec98532ec485938318bf@0.0.0.0:30303",
  id: "df090c3d10b4e485e4e4f85eca81ed3acf7ba1f17a0e18d505aad84f4be4d044ffffac2a10d35791dd8b72a2d328bca5305ec7591e8aec98532ec485938318bf",
  ip: "0.0.0.0",
  listenAddr: "0.0.0.0:30303",
  name: "Geth/v1.8.8-stable-2688dab4/linux-amd64/go1.10.1",
  ports: {
    discovery: 30303,
    listener: 30303
  },
  protocols: {
    eth: {
      config: {
        byzantiumBlock: 1700000,
        chainId: 3,
        daoForkSupport: true,
        eip150Block: 0,
        eip150Hash: "0x41941023680923e0fe4d74a34bdac8141f2540e3ae90623718e47d66d1ca4a2d",
        eip155Block: 10,
        eip158Block: 10,
        ethash: {},
        homesteadBlock: 0
      },
      difficulty: 9696937350631166,
      genesis: "0x41941023680923e0fe4d74a34bdac8141f2540e3ae90623718e47d66d1ca4a2d",
      head: "0x52a714877fe45043a069597363c99a830d4460cb39e3ee2169a9f041be655250",
      network: 3
    }
  }
}
> eth.getTransaction("0xfe0878155927b58a77e8f32b3f2b360686e8897c6e45e5502a797f7251744753").from
"0x08f094d3bc043df21727919aad5a3a0d182fe32d"
> eth.getTransaction("0xfe0878155927b58a77e8f32b3f2b360686e8897c6e45e5502a797f7251744753").nonce
3462
> eth.getTransaction("0x708d1d0ba01b7ce82706bf7ffc03d14dbe768e518464644ef54b15f61eaeb956").nonce
3463
> eth.getTransaction("0x708d1d0ba01b7ce82706bf7ffc03d14dbe768e518464644ef54b15f61eaeb956").from
"0x08f094d3bc043df21727919aad5a3a0d182fe32d"
> eth.getTransactionCount('0x08f094d3bc043df21727919aad5a3a0d182fe32d')
3462
> eth.getTransactionCount('0x08f094d3bc043df21727919aad5a3a0d182fe32d', "pending")
3462

There are pending transaction, which are visible to my geth instance, but still getTransactionCount is not accounting these, when getting transactions with Pending status.

[fjl]

(Finally) fixed in https://github.com/ethereum/go-ethereum/pull/15794.