Parity-ethereum: [PoA] Transactions are not propagated when adding a new node to the network

Created on 10 Dec 2018  ·  17Comments  ·  Source: openethereum/parity-ethereum

_Before filing a new issue, please provide the following information._

  • Parity Ethereum version: Parity-Ethereum/v2.1.8-stable-3eae1d3-20181130/x86_64-linux-gnu/rustc1.30.1
  • Operating system: Linux
  • Installation: one-line installer
  • Fully synchronized: yes
  • Network: Private PoA
  • Restarted: yes

I'm trying the demo PoA tutorial. All seems to work, but when I add a new node to the network (this node isn't validator), the transactions from this node doesn't propagate to the other nodes in the network.

The initial network is composed of the nodes:

  • validator-01
  • validator-02

The updated network is with the node:

  • regular-02

All transactions related to this issue are from regular-02 node.

The genesis file for my network is:

{
  "name":"primer",
  "engine":{
    "authorityRound":{
      "params":{
        "stepDuration":"5",
        "blockReward":"0xDE0B6B3A7640000",
        "validators":{
          "list":[
            "0x001b3fc96e38936badeb3b43bf8b05e43c6128e3",
            "0x009dfd994dfe87af895bd9a6a3e5576182d52d9f"
          ]
        }
      }
    }
  },
  "params":{
    "gasLimitBoundDivisor":"0x400",
    "maximumExtraDataSize":"0x20",
    "minGasLimit":"0x1388",
    "networkID":"0x3144E",
    "eip155Transition":0,
    "validateChainIdTransition":0,
    "eip140Transition":0,
    "eip211Transition":0,
    "eip214Transition":0,
    "eip658Transition":0
  },
  "genesis":{
    "seal":{
      "authorityRound":{
        "step":"0x0",
        "signature":"0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
      }
    },
    "difficulty":"0x20000",
    "gasLimit":"0x5B8D80"
  },
  "accounts":{
    "0x0000000000000000000000000000000000000001":{
      "balance":"1",
      "builtin":{
        "name":"ecrecover",
        "pricing":{
          "linear":{
            "base":3000,
            "word":0
          }
        }
      }
    },
    "0x0000000000000000000000000000000000000002":{
      "balance":"1",
      "builtin":{
        "name":"sha256",
        "pricing":{
          "linear":{
            "base":60,
            "word":12
          }
        }
      }
    },
    "0x0000000000000000000000000000000000000003":{
      "balance":"1",
      "builtin":{
        "name":"ripemd160",
        "pricing":{
          "linear":{
            "base":600,
            "word":120
          }
        }
      }
    },
    "0x0000000000000000000000000000000000000004":{
      "balance":"1",
      "builtin":{
        "name":"identity",
        "pricing":{
          "linear":{
            "base":15,
            "word":3
          }
        }
      }
    },
    "0x89f9097168fd14bdf95155f5b268759d8550d43c":{
      "balance":"10000000000000000000000000"
    }
  }
}

The .toml configuration file from the regular node is:

[parity]                                                                                                                                                                                                           
chain = "/parity/primer.json"                                                                                                                                                                                      
base_path = "/parity/regular-02"                                                                                                                                                                                   
identity = "primer-regular-02"                                                                                                                                                                           
no_persistent_txqueue = true                                                                                                                                                                                       
[network]                                                                                                                                                                                                          
port = 30302                                                                                                                                                                                                       
[rpc]                                                                                                                                                                                                              
interface = "all"                                                                                                                                                                                                  
port = 8542                                                                                                                                                                                                        
cors = ["*"]                                                                                                                                                                                                       
apis = ["web3", "eth", "net", "personal", "parity", "parity_set", "traces", "rpc", "parity_accounts"]                                                                                                              
[ui]                                                                                                                                                                                                               
port = 8182                                                                                                                                                                                                        
[websockets]                                                                                                                                                                                                       
port = 8452                                                                                                                                                                                                        
[ipc]                                                                                                                                                                                                              
disable = true

The .toml config file from one of the validator nodes is (the other one is the same config but with different identity and base path):

[parity]                                                                                                                                                                                                           
chain = "/parity/primer.json"                                                                                                                                                                                      
base_path = "/parity/validator-02"                                                                                                                                                                                 
identity = "primer-validator-02"                                                                                                                                                                         
no_persistent_txqueue = true                                                                                                                                                                                       
[network]                                                                                                                                                                                                          
port = 30300                                                                                                                                                                                                       
min_peers = 50                                                                                                                                                                                                     
max_peers = 100                                                                                                                                                                                                    
[rpc]                                                                                                                                                                                                              
port = 8540                                                                                                                                                                                                        
cors = ["*"]                                                                                                                                                                                                       
apis = ["web3", "eth", "net", "personal", "parity", "parity_set", "traces", "rpc", "parity_accounts"]                                                                                                              
[ui]                                                                                                                                                                                                               
port = 8180                                                                                                                                                                                                        
[websockets]                                                                                                                                                                                                       
port = 8450                                                                                                                                                                                                        
[account]                                                                                                                                                                                                          
password = ["/parity/validator-02/node.pwds"]                                                                                                                                                                      
[mining]                                                                                                                                                                                                           
engine_signer = "0x001b3fc96e38936badeb3b43bf8b05e43c6128e3"                                                                                                                                                       
force_sealing = true                                                                                                                                                                                               
reseal_on_txs = "all"

When I send a new transaction from this node, a new transaction is created:

curl --data '{"jsonrpc":"2.0","method":"personal_sendTransaction","params":[{\
  "from":"0x00c498303595303a7533c47cf32f632087dc1c0c",\
  "to":"0x009dfd994dfe87af895bd9a6a3e5576182d52d9f",\
  "value":"0xDE0B6B3A7640000"}, "**************"],"id":0}' \
  -H "Content-Type: application/json" -X POST localhost:8542                                                                                             

{"jsonrpc":"2.0","result":"0x3135e61208a47af85e677be1e9f39efbd7a81098212f525f4cbcf6be082e8930","id":0}

When I want to retrieve the transaction info from this node, I obtain this result:

curl -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionByHash","params":["0x3135e61208a47af85e677be1e9f39efbd7a81098212f525f4cbcf6be082e8930"],"id":1}' -H "Content
-Type: application/json" localhost:8542

{                                                                                                                                                                                                                  
  "jsonrpc": "2.0",
  "result": {
    "blockHash": null,
    "blockNumber": null,
    "chainId": "0x3144e",
    "condition": null,
    "creates": null,
    "from": "0x00c498303595303a7533c47cf32f632087dc1c0c",
    "gas": "0x186a00",
    "gasPrice": "0x0",
    "hash": "0x3135e61208a47af85e677be1e9f39efbd7a81098212f525f4cbcf6be082e8930",
    "input": "0x",
    "nonce": "0x16",
    "publicKey": "0xa0263cf09db59c38b4c155aa72b059b06db26cb7123ef07a48025960254268e45496890d7aa258f1337ea43d9a0d1bb05ee99410f42205c6dd817276a311e512",
    "r": "0xefad4a024b155b4bb83c519ac3f5991184c7579d328f13ce1c0b5c922f7e5b75",
    "raw": "0xf86b168083186a0094009dfd994dfe87af895bd9a6a3e5576182d52d9f880de0b6b3a764000080830628bfa0efad4a024b155b4bb83c519ac3f5991184c7579d328f13ce1c0b5c922f7e5b75a06e97cd60d336c666ba702ba52abc49862f6158c2368
5350c89c79e1c6874560c",
    "s": "0x6e97cd60d336c666ba702ba52abc49862f6158c23685350c89c79e1c6874560c",
    "standardV": "0x0",
    "to": "0x009dfd994dfe87af895bd9a6a3e5576182d52d9f",
    "transactionIndex": null,
    "v": "0x628bf",
    "value": "0xde0b6b3a7640000"
  },
  "id": 1
}

From the Ethereum network status this is what I see:

parity-poa-network

I expect the transaction sent from the regular-02 node has been mined, but this never occurs.

What I'm missing?

When I use Parity 1.11.7 or 1.11.11, this network config work as expected, and all transactions are mined.

Related?

9329

F2-bug 🐞 M4-core ⛓ P5-sometimesoon 🌲
Source
picture fooock

Most helpful comment

@fooock thank you so much for the logs and results, it seems thought that your transaction pool is now empty and you don't have any transactions from 0x00c498303595303a7533c47cf32f632087dc1c0c, can you confirm that?

Would you mind reproducing the situation again and provide the same logs in the case where transactions are there?

picture tomusdrw on 18 Dec 2018
👍2

All 17 comments

Can you share the txqueue logs from your validator nodes? (-l txqueue=trace) when you try getTransactionByHash the result indicates that the transaction is still pending (blockHash is null, blockNumber is null) so that either means the transaction is either not reaching your validation nodes, or not being accepted by them once they reach the txqueue. Do you see the transaction if you try that rpc call on your validation nodes?

joshua-mir picture joshua-mir on 10 Dec 2018

txqueue logs from validator nodes here. When I execute the last command to see the transaction from my validator-02 node the response is nothing: 

$ curl -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionByHash","params":["0x3135e61208a47af85e677be1e9f39efbd7a81098212f525f4cbcf6be082e8930"],"id":1}' -
H "Content -Type: application/json" localhost:8540
$

Same for the validator-01 node.

But if I try to view all transactions from my regular node, all pending transactions appear:

$ curl --data '{"method":"parity_allTransactions","params":[],"id":1,"jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST localhost:8542 | jq
{
  "jsonrpc": "2.0",
  "result": [
    {}, ......
    {
      "blockHash": null,
      "blockNumber": null,
      "chainId": "0x3144e",
      "condition": null,
      "creates": null,
      "from": "0x00c498303595303a7533c47cf32f632087dc1c0c",
      "gas": "0x186a00",
      "gasPrice": "0x0",
      "hash": "0x2b96e93981db005758e66d04cc3b40e198e16ee87191d793ac9cadf7ddb1b507",
      "input": "0x",
      "nonce": "0x4",
      "publicKey": "0xa0263cf09db59c38b4c155aa72b059b06db26cb7123ef07a48025960254268e45496890d7aa258f1337ea43d9a0d1bb05ee99410f42205c6dd817276a311e512",
      "r": "0xff0e26b13567e5a5d683d88e89d6f0d273331a5e081d69e79e86c131cc2cd0ac",
      "raw": "0xf86b048083186a0094009dfd994dfe87af895bd9a6a3e5576182d52d9f880de0b6b3a764000080830628c0a0ff0e26b13567e5a5d683d88e89d6f0d273331a5e081d69e79e86c131cc2cd0aca0054970f35763fa4152934ad594aadc1e6f096da73
f02ed6735b63d23dd0d24be",
      "s": "0x54970f35763fa4152934ad594aadc1e6f096da73f02ed6735b63d23dd0d24be",
      "standardV": "0x1",
      "to": "0x009dfd994dfe87af895bd9a6a3e5576182d52d9f",
      "transactionIndex": null,
      "v": "0x628c0",
      "value": "0xde0b6b3a7640000"
    },
    {
      "blockHash": null,
      "blockNumber": null,
      "chainId": "0x3144e",
      "condition": null,
      "creates": null,
      "from": "0x00c498303595303a7533c47cf32f632087dc1c0c",
      "gas": "0x186a00",
      "gasPrice": "0x0",
      "hash": "0xeaf88ce8ace42ebecfe966225b13c82d5fe3e47aad465385da6d573a5fa769b9",
      "input": "0x",
      "nonce": "0x5",
      "publicKey": "0xa0263cf09db59c38b4c155aa72b059b06db26cb7123ef07a48025960254268e45496890d7aa258f1337ea43d9a0d1bb05ee99410f42205c6dd817276a311e512",
      "r": "0x9145d84b8fc9da4854d6f6a18f57c9e8b80ee325b1000dab869ab66487e993a9",
      "raw": "0xf86b058083186a0094009dfd994dfe87af895bd9a6a3e5576182d52d9f880de0b6b3a764000080830628c0a09145d84b8fc9da4854d6f6a18f57c9e8b80ee325b1000dab869ab66487e993a9a004dd2d87a058629ec3b7392c01380478ae237cb9a
c5e011f10bc94f688f32c4a",
      "s": "0x4dd2d87a058629ec3b7392c01380478ae237cb9ac5e011f10bc94f688f32c4a",
      "standardV": "0x1",
      "to": "0x009dfd994dfe87af895bd9a6a3e5576182d52d9f",
      "transactionIndex": null,
      "v": "0x628c0",
      "value": "0xde0b6b3a7640000"
    },
    {},.....
    {
      "blockHash": null,
      "blockNumber": null,
      "chainId": "0x3144e",
      "condition": null,
      "creates": null,
      "from": "0x00c498303595303a7533c47cf32f632087dc1c0c",
      "gas": "0x186a00",
      "gasPrice": "0x0",
      "hash": "0xc31e2886ac200c325e20a36036d58d6d832c757e027bd17c0e62a818b07db71e",
      "input": "0x",
      "nonce": "0xa",
      "publicKey": "0xa0263cf09db59c38b4c155aa72b059b06db26cb7123ef07a48025960254268e45496890d7aa258f1337ea43d9a0d1bb05ee99410f42205c6dd817276a311e512",
      "r": "0xb0e745ec14a47a62f548df4dbfa8c148d9ed08b0fb28e4dde8d876e41c3d6aa9",
      "raw": "0xf86b0a8083186a0094009dfd994dfe87af895bd9a6a3e5576182d52d9f880de0b6b3a764000080830628c0a0b0e745ec14a47a62f548df4dbfa8c148d9ed08b0fb28e4dde8d876e41c3d6aa9a00d4264a679f14f6a5150706be81bc1b6b33841b91
c59de7e3e75ffb54e31e8ef",
      "s": "0xd4264a679f14f6a5150706be81bc1b6b33841b91c59de7e3e75ffb54e31e8ef",
      "standardV": "0x1",
      "to": "0x009dfd994dfe87af895bd9a6a3e5576182d52d9f",
      "transactionIndex": null,
      "v": "0x628c0",
      "value": "0xde0b6b3a7640000"
    },
    {},.......
    {
      "blockHash": null,
      "blockNumber": null,
      "chainId": "0x3144e",
      "condition": null,
      "creates": null,
      "from": "0x00c498303595303a7533c47cf32f632087dc1c0c",
      "gas": "0x186a00",
      "gasPrice": "0x0",
      "hash": "0xe8ac167d33ec8d5aed947d2bea63cc5cf8308c0337b620ec530339ccd3c872ea",
      "input": "0x",
      "nonce": "0x15",
      "publicKey": "0xa0263cf09db59c38b4c155aa72b059b06db26cb7123ef07a48025960254268e45496890d7aa258f1337ea43d9a0d1bb05ee99410f42205c6dd817276a311e512",
      "r": "0x4520d4894b1af5c20ba09897edd39d0a4cb19e8b37e6fb54d51d4affd0b058c4",
      "raw": "0xf86b158083186a0094009dfd994dfe87af895bd9a6a3e5576182d52d9f880de0b6b3a764000080830628bfa04520d4894b1af5c20ba09897edd39d0a4cb19e8b37e6fb54d51d4affd0b058c4a00e59571009ee79de483e904f635506e6097f0e454
f8d65e2a6d0afdb0739c91d",
      "s": "0xe59571009ee79de483e904f635506e6097f0e454f8d65e2a6d0afdb0739c91d",
      "standardV": "0x0",
      "to": "0x009dfd994dfe87af895bd9a6a3e5576182d52d9f",
      "transactionIndex": null,
      "v": "0x628bf",
      "value": "0xde0b6b3a7640000"
    },
    {
      "blockHash": null,
      "blockNumber": null,
      "chainId": "0x3144e",
      "condition": null,
      "creates": null,
      "from": "0x00c498303595303a7533c47cf32f632087dc1c0c",
      "gas": "0x186a00",
      "gasPrice": "0x0",
      "hash": "0x3135e61208a47af85e677be1e9f39efbd7a81098212f525f4cbcf6be082e8930",
      "input": "0x",
      "nonce": "0x16",
      "publicKey": "0xa0263cf09db59c38b4c155aa72b059b06db26cb7123ef07a48025960254268e45496890d7aa258f1337ea43d9a0d1bb05ee99410f42205c6dd817276a311e512",
      "r": "0xefad4a024b155b4bb83c519ac3f5991184c7579d328f13ce1c0b5c922f7e5b75",
      "raw": "0xf86b168083186a0094009dfd994dfe87af895bd9a6a3e5576182d52d9f880de0b6b3a764000080830628bfa0efad4a024b155b4bb83c519ac3f5991184c7579d328f13ce1c0b5c922f7e5b75a06e97cd60d336c666ba702ba52abc49862f6158c23
685350c89c79e1c6874560c",
      "s": "0x6e97cd60d336c666ba702ba52abc49862f6158c23685350c89c79e1c6874560c",
      "standardV": "0x0",
      "to": "0x009dfd994dfe87af895bd9a6a3e5576182d52d9f",
      "transactionIndex": null,
      "v": "0x628bf",
      "value": "0xde0b6b3a7640000"
    }
  ],
  "id": 1
}
fooock picture fooock on 10 Dec 2018

Yeah, you're certainly correct that the transactions don't seem to be being propagated in the first place. Your netstats also show your nodes are all in sync and on the same chain, that rules out connectivity issues. I'm seeing other users with similar issues elsewhere.

It looks like you're running with --no-persistent-txqueue. Have the transactions (with lower nonces) before those on the txqueue been confirmed already? This is likely a case of nodes not accepting transactions because they don't have the dependencies of the transaction.

joshua-mir picture joshua-mir on 10 Dec 2018
👎1

No transactions from any validator node:

parity-validator-01

$ curl --data '{"method":"parity_allTransactions","params":[],"id":1,"jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST localhost:8540
{"jsonrpc":"2.0","result":[],"id":1}

Using the address from my validator-01 node the client return 3 pending transactions:

$ curl -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["0x009dfd994dfe87af895bd9a6a3e5576182d52d9f","pending"],"id":1}' -H "Content-Type
: application/json" localhost:8540

{"jsonrpc":"2.0","result":"0x3","id":1}

Same number of transactions using the latest param.

parity-validator-02

$ curl --data '{"method":"parity_allTransactions","params":[],"id":1,"jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST localhost:8540
{"jsonrpc":"2.0","result":[],"id":1}

No transactions sent using this validator.

What do you mean by dependencies of the transaction? The transactions I created are to move funds between accounts and no external dependencies are involved

fooock picture fooock on 10 Dec 2018

ie, transaction nonces

joshua-mir picture joshua-mir on 10 Dec 2018

No transactions confirmed from the regular node. The pending transactions shown in one of my last comments is shorted to not occupy much space, currently, nonces from these transactions are from 0x0 to 0x16

fooock picture fooock on 12 Dec 2018

@fooock:

  1. could you run the the regular node with -lsync=trace,txpool=trace?
  2. Seems that you are sending more than 16 transactions from the same address, can you check if setting --txqueue-per-sender 1024 on the regular node changes anything (related #9909 )
  3. Is regular node connected to validators and on the same block?
  4. Could you run parity_nextNonce("0x00c498303595303a7533c47cf32f632087dc1c0c") and eth_getTransactionCount("0x00c498303595303a7533c47cf32f632087dc1c0c", "latest") on the regular node and also on validator nodes?
  5. The validator logs never have anything in the pool, which seems suspicious - probably the transactions are not sent at all.
tomusdrw picture tomusdrw on 17 Dec 2018

1 - Logs for the regular node can be found here
3 - Yes, regular node is connected to two nodes
4 - Results:

regular-02

  • Method parity_nextNonce:
$ curl -X POST --data '{"jsonrpc":"2.0","method":"parity_nextNonce","params":["0x00c498303595303a7533c47cf32f632087dc1c0c"],"id":1}' -H "Content-Type: application/json" localhost:8542
{"jsonrpc":"2.0","result":"0x0","id":1}
  • Method eth_getTransactionCount:
$ curl -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["0x00c498303595303a7533c47cf32f632087dc1c0c", "latest"],"id":1}' -H "Content-Type: application/json" localhost:8542                                                             
{"jsonrpc":"2.0","result":"0x0","id":1}

validator-01

  • Method parity_nextNonce:
$ curl -X POST --data '{"jsonrpc":"2.0","method":"parity_nextNonce","params":["0x00c498303595303a7533c47cf32f632087dc1c0c"],"id":1}' -H "Content-Type: application/json" localhost:8540                                                                                   
{"jsonrpc":"2.0","result":"0x0","id":1}
  • Method eth_getTransactionCount:
$ curl -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["0x00c498303595303a7533c47cf32f632087dc1c0c", "latest"],"id":1}' -H "Content-Type: application/json" localhost:8540                                                             
{"jsonrpc":"2.0","result":"0x0","id":1}

validator-02

  • Method parity_nextNonce:
$ curl -X POST --data '{"jsonrpc":"2.0","method":"parity_nextNonce","params":["0x00c498303595303a7533c47cf32f632087dc1c0c"],"id":1}' -H "Content-Type: application/json" localhost:8540                                                                                   
{"jsonrpc":"2.0","result":"0x0","id":1}
  • Method eth_getTransactionCount:
$ curl -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["0x00c498303595303a7533c47cf32f632087dc1c0c", "latest"],"id":1}' -H "Content-Type: application/json" localhost:8540                                                             
{"jsonrpc":"2.0","result":"0x0","id":1}

I will update the configuration to add the --txqueue-per-sender 1024 flag into the regular node.

fooock picture fooock on 17 Dec 2018

Using --tx-queue-per-sender 1024 doesn't changes anything

fooock picture fooock on 17 Dec 2018

@fooock thank you so much for the logs and results, it seems thought that your transaction pool is now empty and you don't have any transactions from 0x00c498303595303a7533c47cf32f632087dc1c0c, can you confirm that?

Would you mind reproducing the situation again and provide the same logs in the case where transactions are there?

tomusdrw picture tomusdrw on 18 Dec 2018
👍2

@joshua-mir, you have seen these symptoms in other installations. Could you kindly cross-reference Issues for those and similar, to compare configurations and logs?
#9329 was never replicated or root caused, and this feels way beyond an 'annoyance'.

wkarshat picture wkarshat on 4 Feb 2019

"I'm seeing other users with similar issues" usually refers to gitter. Unfortunately, I can't get everyone to share their logs or report/update issues here.

Bumped priority, I'm still seeing this incidentally, but there has been improvements due to better transaction importing #10051 @wkarshat are you still seeing this yourself and are you willing to put in time to help investigate?

joshua-mir picture joshua-mir on 11 Feb 2019

@tomusdrw I can't reproduce the error...

fooock picture fooock on 15 Feb 2019

@joshua-mir I am running something like this

  • 3 validator nodes
  • 1 regular node.
  • regular node is connected to one of the validators nodes.

Any transaction that is submitted in the regular node are propagated to the network and included in the next block.

SidhMj picture SidhMj on 30 Sep 2019

@SidhMj does that imply that you aren't seeing this issue?

joshua-mir picture joshua-mir on 30 Sep 2019

@joshua-mir yes

SidhMj picture SidhMj on 30 Sep 2019

OP also mentioned they can't replicate so will close unless someone states otherwise

joshua-mir picture joshua-mir on 30 Sep 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

famfamfam picture famfamfam  ·  3Comments
BillSantos picture BillSantos  ·  3Comments
tzapu picture tzapu  ·  3Comments
bryaan picture bryaan  ·  3Comments
barakman picture barakman  ·  3Comments