Parity-ethereum: PoA nodes are no longer transmitting transactions to other (mining)-nodes

[roderik]

I'm running:

• Which Parity version?: 2.0.1
• Which operating system?: Docker on Linux
• How installed?: Offcial Docker image
• Are you fully synchronized?: yes
• Which network are you connected to?: PoA network, details below
• Did you try to restart the node?: yes

Network info

• Spec file: https://github.com/settlemint/mintnet/blob/master/mintnet.json
• Stats: https://stats.mintnet.settlemint.com/
• Explorer: https://explorer.mintnet.settlemint.com/

Network setup:

• 2 mining nodes in 2 regions in Digital Ocean (with firewall around it, but due to debugging allows all traffic) - Bootnodes config
• 2 edge nodes (non-mining) with a load balancer in front with an https connection URL (like you use infura) - Bootnodes config
• Each project or dev machine runs a client docker node to read and write to - Client config

Bootnodes:

• Commandline: https://gist.github.com/roderik/97dd56161db1a75f0022a071b0abaaa3
• Config: https://gist.github.com/roderik/67a63642d3ffae8661deefc7061f5c70
• Dockerfile: https://gist.github.com/roderik/5867da2fb08ca43af58ac3bbb658fc73
• Docker container: not public

Clients:

• Commandline: https://github.com/settlemint/mintnet/blob/master/entrypoint.sh
• Config: https://github.com/settlemint/mintnet/blob/master/mintnet.toml
• Dockerfile: https://github.com/settlemint/mintnet/blob/master/Dockerfile
• Docker container: https://hub.docker.com/r/settlemint/mintnet/

Actual and Expected behavior with v2.0.0 and below

This setup ran smoothly. Transactions were sent to either the client or edge nodes and processed by the mining nodes every 5 seconds.

Actual behavior after upgrading everything to v2.0.1

• Sending a transaction to a client node, will import the TX nicely but does not transmit it to the rest of the nodes. So the miners never pick up on them.
• Sending a transaction to the load balancer or an edge node directly, same behavior

• Sending a transaction to a mining node, will mine a block with the transaction in it.

• Changes made: https://github.com/settlemint/mintnet/compare/2.0.0...master (lots of debugging tries so do not mind the commit mess)

• Edge node logs with a lot of tracing enabled: https://gist.github.com/roderik/17897483ca68be04f659e97e224a63c6 (look for tx id 0x9e792fd35d277627c268119113d3168be5eb599417d821e28012fbb2467af594)
• Client node logs with less tracing enabled: https://gist.github.com/roderik/ba6078f14d0f412ca478ea0159e0d4cb (look for tx id 0xe48675d399ac2b5d3d9e05ea0f326a3187da13ccf6041a03dc4a4edce7b16f98)

What is weird:

1. Four nodes, 10 peers? Peer list from an edge node and from a miner node. Starting a client makes it go to 11. Why are there a bunch of 1.11.3 nodes connection or a non-standard port with a custom network id?
2. Sending a tx to an edge node ends up with a stuck tx with e.g txid 0xe48675d399ac2b5d3d9e05ea0f326a3187da13ccf6041a03dc4a4edce7b16f98 on the edge node (so 1 pending). Sending that same transaction (executing another migration.sol deployment) to the mining node, which ends up with the same txid, goes in, gets mined, and then the client does drop the pending one. So there is communication between the nodes, just not "sending".

I've got a workaround, I pointed my load balancer to the mining node, but that is not in the least decentralized nor resilient. I've got no further ideas on what it could be, the only thing I have not tried is rolling everything back to 2.0.0, but since I've found no issues relating to the same problem, I guess it's more a config issue on my end than a software issue.

[Tbaut]

Do you have any way to reproduce on a minimal setup for us to reproduce quickly? This would tremendously help :)

[roderik]

Use this docker-compose file https://github.com/settlemint/mintnet/blob/master/docker-compose.yml and a client node will spin up and start syncing. After the sync, you can send a tx to it.

The easiest way to send a TX is by running ETHEREUM_DEPLOYER_SEED='robot robot robot robot robot robot robot robot robot robot robot robot' truffle migrate --network mintnet on https://github.com/DataBrokerDAO/dtx-crowdsale-contracts

Adding more logging to the container can be done by cloning https://github.com/settlemint/mintnet , changing either the toml or the entrypoint.sh file and modifying the docker-compose file as follows

version: "3.3"

services:
  ethereum:
    build: .
    ports:
      - "8545:8545"
      - "8546:8546"
      - "30303:30303"
      - "30303:30303/udp"
  net-intelligence-api:
    image: settlemint/eth-net-intelligence-api
    environment:
      CONTACT_DETAILS: [email protected]
      INSTANCE_NAME: My awesome Node (add a company name or location)
      LISTENING_PORT: '30311'
      RPC_HOST: ethereum
      RPC_PORT: '8545'
      WS_SECRET: settlemint
      WS_SERVER: wss://stats.mintnet.settlemint.com
    links:
      - ethereum:ethereum

then run docker-compose build; docker-compose up.

[Tbaut]

Four nodes, 10 peers? Peer list from an edge node and from a miner node. Starting a client makes it go to 11. Why are there a bunch of 1.11.3 nodes connection or a non-standard port with a custom network id?

This is the open web, anything is possible... Your configuration is very open. Be sure to not use things like unsafe-expose, jsonrpc-cors all or no firewall as soon as you are able to. (find here our list of security best practices)

Sending a tx to an edge node ends up with a stuck tx

I can't think of anything that was changed between 2.0 and 2.0.1 except maybe #9148 which should have a minimal impact.
We need to understand why this tx is stuck. FWIW here are more info on the transactions queue.

• How full is the queue of the client node?
• What is the block gas limit, the number of tx per block ? (you can use a block explorer for that)

[roderik]

This is the open web, anything is possible... Your configuration is very open. Be sure to not use things like unsafe-expose, jsonrpc-cors all or no firewall as soon as you are able to. (find here our list of security best practices)

Good point, just making sure it is not setup issue in that respect.

How full is the queue of the client node?

0 transactions, and after adding one, 1 transaction

What is the block gas limit, the number of tx per block ? (you can use a block explorer for that)

Blocks usually have 0 or 1 tx per block.
Gas limit is 87000000 gas
FYI: The explorer and netstats links are in my initial ticket to verify

[Tbaut]

I couldn't reproduce. I've deployed a dummy contract at 0x6a1c59ac3cd85c4b3f45ef50c4b65331f6ee2ee6 using RPC personal_sendTransaction

The logs showed the following:

ethereum_1              | 2018-08-14 15:36:15 UTC IO Worker #3 DEBUG txqueue  Re-computing pending set for block: 1793896
ethereum_1              | 2018-08-14 15:36:19 UTC IO Worker #0 INFO import     2/25 peers      1 MiB chain   99 MiB db  0 bytes queue    4 MiB sync  RPC:  0 conn,    0 req/s,  185 µs
ethereum_1              | 2018-08-14 15:36:20 UTC Verifier #0 INFO import  Imported #1793897 0xf41d…b221 (0 txs, 0.00 Mgas, 2 ms, 0.58 KiB)
ethereum_1              | 2018-08-14 15:36:20 UTC IO Worker #1 DEBUG txqueue  Removed 0 stalled transactions. Pool: 0/131072 (0 senders; 0/18014398509481983 kB) [minGasPrice: 0 Mwei, maxGas: 87000000]
ethereum_1              | 
ethereum_1              | 2018-08-14 15:36:21 UTC IO Worker #2 DEBUG txqueue  Re-computing pending set for block: 1793897
ethereum_1              | 2018-08-14 15:36:22 UTC  TRACE own_tx  Importing transaction: PendingTransaction { transaction: SignedTransaction { transaction: UnverifiedTransaction { unsigned: Transaction { nonce: 0x0, gas_price: 0x0, gas: 0xe57e0, action: Create, value: 0x0, data: [96, 128, 96, 64, 82, 52, 128, 21, 97, 0, 16, 87, 96, 0, 128, 253, 91, 80, 96, 172, 128, 97, 0, 31, 96, 0, 57, 96, 0, 243, 0, 96, 128, 96, 64, 82, 96, 4, 54, 16, 96, 63, 87, 96, 0, 53, 124, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 144, 4, 99, 255, 255, 255, 255, 22, 128, 99, 12, 73, 195, 108, 20, 96, 68, 87, 91, 96, 0, 128, 253, 91, 52, 128, 21, 96, 79, 87, 96, 0, 128, 253, 91, 80, 96, 86, 96, 116, 86, 91, 96, 64, 81, 128, 130, 96, 0, 25, 22, 96, 0, 25, 22, 129, 82, 96, 32, 1, 145, 80, 80, 96, 64, 81, 128, 145, 3, 144, 243, 91, 96, 0, 96, 42, 96, 1, 2, 144, 80, 144, 86, 0, 161, 101, 98, 122, 122, 114, 48, 88, 32, 226, 114, 105, 33, 116, 136, 100, 217, 29, 190, 75, 20, 254, 133, 26, 10, 225, 138, 31, 230, 67, 230, 115, 171, 62, 104, 49, 120, 182, 236, 22, 211, 0, 41] }, v: 18026, r: 0x4593d289d014bb7a4caead30ef5b366d69d20d037aa28ffefef2f3a011dcebe0, s: 0xa3d4944df1cc3a07431f10adbb2d90dd401bed98b77397eece723d4e5984d65, hash: 0x9dbfcf62c29e48fc086851dcf77b73880d946e11423a470fb17da6baf49b1b30 }, sender: 0x0023dffdff54e3381271c97e6ed16d438002779f, public: Some(0x9f5e1d2459c7f4445d7045d62fa5ded886e668d06b590e7e3c932c59c3f334c52e173a32cf70de7ec11d8aa4d348833c5e40e17e26a05c72eb597d9aa00cbf7b) }, condition: None }
ethereum_1              | 2018-08-14 15:36:22 UTC  DEBUG own_tx  Imported to the pool (hash 0x9dbfcf62c29e48fc086851dcf77b73880d946e11423a470fb17da6baf49b1b30)
ethereum_1              | 2018-08-14 15:36:22 UTC  DEBUG txqueue  [0x9dbfcf62c29e48fc086851dcf77b73880d946e11423a470fb17da6baf49b1b30] Added to the pool.
ethereum_1              | 2018-08-14 15:36:22 UTC  DEBUG txqueue  [0x9dbfcf62c29e48fc086851dcf77b73880d946e11423a470fb17da6baf49b1b30] Sender: 0x0023…779f, nonce: 0, gasPrice: 0, gas: 940000, value: 0, dataLen: 203))

Transaction receipt shows a status of 0x1 and my contract was deployed.

[roderik]

To what rpc endpoint? A locally running node or https://mintnet.settlemint.com? That last one points to a mining node at the moment.

[Tbaut]

I launched a local node using your Docker and deployed a contract with the above RPC pointed to this local node.

[roderik]

Sorry for the silence, i was on holiday. I'm fairly sure that did not work.

The transaction from your log is not found in the chain: https://explorer.mintnet.settlemint.com/tx/0x9dbfcf62c29e48fc086851dcf77b73880d946e11423a470fb17da6baf49b1b30

I tried it myself again (deploying Migrations.sol from Truffle)

ethereum_1                | 
ethereum_1                | 2018-08-28 13:47:26 UTC IO Worker #2 DEBUG txqueue  Re-computing pending set for block: 2027875
ethereum_1                | 2018-08-28 13:47:27 UTC  WARN parity_rpc::v1::impls::eth  Fallback to `BlockId::Latest`
ethereum_1                | 2018-08-28 13:47:27 UTC  TRACE own_tx  Importing transaction: PendingTransaction { transaction: SignedTransaction { transaction: UnverifiedTransaction { unsigned: Transaction { nonce: 0x0, gas_price: 0x0, gas: 0x6691b7, action: Create, value: 0x0, data: [96, 128, 96, 64, 82, 52, 128, 21, 97, 0, 16, 87, 96, 0, 128, 253, 91, 80, 96, 0, 128, 84, 96, 1, 96, 160, 96, 2, 10, 3, 25, 22, 51, 23, 144, 85, 97, 2, 63, 128, 97, 0, 50, 96, 0, 57, 96, 0, 243, 0, 96, 128, 96, 64, 82, 96, 4, 54, 16, 97, 0, 97, 87, 99, 255, 255, 255, 255, 124, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 96, 0, 53, 4, 22, 99, 9, 0, 240, 16, 129, 20, 97, 0, 102, 87, 128, 99, 141, 165, 203, 91, 20, 97, 0, 150, 87, 128, 99, 251, 219, 173, 60, 20, 97, 0, 212, 87, 128, 99, 253, 172, 213, 118, 20, 97, 0, 251, 87, 91, 96, 0, 128, 253, 91, 52, 128, 21, 97, 0, 114, 87, 96, 0, 128, 253, 91, 80, 97, 0, 148, 115, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 96, 4, 53, 22, 97, 1, 19, 86, 91, 0, 91, 52, 128, 21, 97, 0, 162, 87, 96, 0, 128, 253, 91, 80, 97, 0, 171, 97, 1, 200, 86, 91, 96, 64, 128, 81, 115, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 144, 146, 22, 130, 82, 81, 144, 129, 144, 3, 96, 32, 1, 144, 243, 91, 52, 128, 21, 97, 0, 224, 87, 96, 0, 128, 253, 91, 80, 97, 0, 233, 97, 1, 228, 86, 91, 96, 64, 128, 81, 145, 130, 82, 81, 144, 129, 144, 3, 96, 32, 1, 144, 243, 91, 52, 128, 21, 97, 1, 7, 87, 96, 0, 128, 253, 91, 80, 97, 0, 148, 96, 4, 53, 97, 1, 234, 86, 91, 96, 0, 128, 84, 115, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 22, 51, 20, 97, 1, 56, 87, 96, 0, 128, 253, 91, 129, 144, 80, 128, 115, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 22, 99, 253, 172, 213, 118, 96, 1, 84, 96, 64, 81, 130, 99, 255, 255, 255, 255, 22, 124, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 129, 82, 96, 4, 1, 128, 130, 129, 82, 96, 32, 1, 145, 80, 80, 96, 0, 96, 64, 81, 128, 131, 3, 129, 96, 0, 135, 128, 59, 21, 128, 21, 97, 1, 172, 87, 96, 0, 128, 253, 91, 80, 90, 241, 21, 128, 21, 97, 1, 192, 87, 61, 96, 0, 128, 62, 61, 96, 0, 253, 91, 80, 80, 80, 80, 80, 80, 86, 91, 96, 0, 84, 115, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 22, 129, 86, 91, 96, 1, 84, 129, 86, 91, 96, 0, 84, 115, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 22, 51, 20, 97, 2, 14, 87, 96, 0, 128, 253, 91, 96, 1, 85, 86, 0, 161, 101, 98, 122, 122, 114, 48, 88, 32, 143, 61, 89, 224, 204, 154, 204, 166, 199, 116, 125, 81, 242, 96, 127, 12, 180, 103, 71, 176, 239, 79, 89, 174, 9, 62, 82, 22, 56, 142, 237, 213, 0, 41] }, v: 27, r: 0xb16ca331c588fe1d2e062d4e22f20734c1359253fd3091b40ab81131b6c12680, s: 0x60a6079208b1d7da4fd0bfaa2e23b94cc165c6c582c89e2c88c9f349251d0466, hash: 0xd7624a6fbc4295f046a2abd5c8639732acaa99af18f64e8f97884c11a5cc184d }, sender: 0xaeb1777cf13472cf0df25a7d8f688451a15609c9, public: Some(0x055b655588d5f6105028f55bb6e00e9341b1bb6d81f36f67bada7fbbfcf36108f4a2577aafc11bbd257dca850933b0603a374017f3d577f2224622a88c24d0c8) }, condition: None }
ethereum_1                | 2018-08-28 13:47:27 UTC  DEBUG own_tx  Imported to the pool (hash 0xd7624a6fbc4295f046a2abd5c8639732acaa99af18f64e8f97884c11a5cc184d)
ethereum_1                | 2018-08-28 13:47:27 UTC  DEBUG txqueue  [0xd7624a6fbc4295f046a2abd5c8639732acaa99af18f64e8f97884c11a5cc184d] Added to the pool.
ethereum_1                | 2018-08-28 13:47:27 UTC  DEBUG txqueue  [0xd7624a6fbc4295f046a2abd5c8639732acaa99af18f64e8f97884c11a5cc184d] Sender: 0xaeb1…09c9, nonce: 0, gasPrice: 0, gas: 6721975, value: 0, dataLen: 625))
ethereum_1                | 2018-08-28 13:47:27 UTC  DEBUG txqueue  Re-computing pending set for block: 2027875
ethereum_1                | 2018-08-28 13:47:30 UTC Verifier #1 DEBUG txqueue  Re-computing pending set for block: 2027876
ethereum_1                | 2018-08-28 13:47:30 UTC IO Worker #0 DEBUG txqueue  Removed 0 stalled transactions. Pool: 1/131072 (1 senders; 0/18014398509481983 kB) [minGasPrice: 0 Mwei, maxGas: 87000000]
ethereum_1                | 
ethereum_1                | 2018-08-28 13:47:30 UTC Verifier #1 INFO import  Imported #2027876 0xd3b2…ac28 (0 txs, 0.00 Mgas, 0 ms, 0.58 KiB)
ethereum_1                | 2018-08-28 13:47:35 UTC Verifier #2 DEBUG txqueue  Re-computing pending set for block: 2027877
ethereum_1                | 2018-08-28 13:47:35 UTC IO Worker #2 DEBUG txqueue  Removed 0 stalled transactions. Pool: 1/131072 (1 senders; 0/18014398509481983 kB) [minGasPrice: 0 Mwei, maxGas: 87000000]
ethereum_1                | 
ethereum_1                | 2018-08-28 13:47:35 UTC Verifier #2 INFO import  Imported #2027877 0x6362…0a69 (0 txs, 0.00 Mgas, 0 ms, 0.58 KiB)
ethereum_1                | 2018-08-28 13:47:40 UTC Verifier #3 DEBUG txqueue  Re-computing pending set for block: 2027878
ethereum_1                | 2018-08-28 13:47:40 UTC Verifier #3 INFO import  Imported #2027878 0xd0dc…bc42 (0 txs, 0.00 Mgas, 0 ms, 0.58 KiB)
ethereum_1                | 2018-08-28 13:47:40 UTC IO Worker #3 DEBUG txqueue  Removed 0 stalled transactions. Pool: 1/131072 (1 senders; 0/18014398509481983 kB) [minGasPrice: 0 Mwei, maxGas: 87000000]
ethereum_1              

Doing the same migration on the mining node shows me this on my local docker node

ethereum_1                | 2018-08-28 13:55:10 UTC Verifier #3 INFO import  Imported #2027968 0x052f…29a3 (0 txs, 0.00 Mgas, 0 ms, 0.58 KiB)
ethereum_1                | 2018-08-28 13:55:11 UTC IO Worker #3 DEBUG txqueue  Re-computing pending set for block: 2027968
ethereum_1                | 2018-08-28 13:55:14 UTC IO Worker #1 DEBUG txqueue  [0x17fcc2c8935c25c21619fa439b914e265ec258869c495fa1880e48dff9056102] Added to the pool.
ethereum_1                | 2018-08-28 13:55:14 UTC IO Worker #1 DEBUG txqueue  [0x17fcc2c8935c25c21619fa439b914e265ec258869c495fa1880e48dff9056102] Sender: 0x6e12…9f36, nonce: 2368, gasPrice: 0, gas: 6721975, value: 0, dataLen: 625))
ethereum_1                | 2018-08-28 13:55:15 UTC Verifier #2 INFO import  Imported #2027969 0xdca4…e1da (0 txs, 0.00 Mgas, 0 ms, 0.58 KiB)
ethereum_1                | 2018-08-28 13:55:15 UTC IO Worker #2 DEBUG txqueue  Removed 0 stalled transactions. Pool: 1/131072 (1 senders; 0/18014398509481983 kB) [minGasPrice: 0 Mwei, maxGas: 87000000]
ethereum_1                | 
ethereum_1                | 2018-08-28 13:55:15 UTC IO Worker #1 DEBUG txqueue  Re-computing pending set for block: 2027969
ethereum_1                | 2018-08-28 13:55:20 UTC IO Worker #0 DEBUG txqueue  [0x17fcc2c8935c25c21619fa439b914e265ec258869c495fa1880e48dff9056102] Mined.
ethereum_1                | 2018-08-28 13:55:20 UTC IO Worker #0 DEBUG txqueue  Removed 1 stalled transactions. Pool: 0/131072 (0 senders; 0/18014398509481983 kB) [minGasPrice: 0 Mwei, maxGas: 87000000]
ethereum_1                | 
ethereum_1                | 2018-08-28 13:55:20 UTC Verifier #0 INFO import  Imported #2027970 0x50f9…197d (1 txs, 0.22 Mgas, 1 ms, 1.27 KiB)

So the migration works for sure, the nodes are in connection with each other since it receives both transactions from the mining nodes pool and new blocks.

From the node to the mining nodes pool, however, that is a mystery... Nothing appears in the mining node log when adding a tx in the local nodes txqueue.

[roderik]

It feels like it has to be a networking thing, can you advise what logging and level I could use to sho what parity does after receiving the tx and wants to send it on?

[roderik]

We can also just drop the local docker nodes from the discussion for the sake of simplicity, since deploying directly to the mining nodes has the following effect:

The mining nodes only process the transactions sent to themselves. That is why you see a TX to amsterdam, nothing for frankfurt, and again a tx to amsterdam. And the same but reversed if I use frankfurt.

[roderik]

I downgraded everything to v2.0.0, and it did not change the difference. So the exact version is not the issue

[roderik]

It seems weird, i made sure all mining settings were out of the non mining nodes, and as soon as i push in a TX, it starts up the internal miner.

ethereum_1              | 2018-08-29 10:45:19 UTC  TRACE own_tx  Importing transaction: PendingTransaction { transaction: SignedTransaction { transaction: UnverifiedTransaction { unsigned: Transaction { nonce: 0x111, gas_price: 0x0, gas: 0x6691b7, action: Create, value: 0x0, data: [96, 128, 96, 64, 82, 52, 128, 21, 97, 0, 16, 87, 96, 0, 128, 253, 91, 80, 96, 0, 128, 84, 96, 1, 96, 160, 96, 2, 10, 3, 25, 22, 51, 23, 144, 85, 97, 3, 79, 128, 97, 0, 50, 96, 0, 57, 96, 0, 243, 0, 96, 128, 96, 64, 82, 96, 4, 54, 16, 97, 0, 97, 87, 99, 255, 255, 255, 255, 124, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 96, 0, 53, 4, 22, 99, 9, 0, 240, 16, 129, 20, 97, 0, 102, 87, 128, 99, 141, 165, 203, 91, 20, 97, 0, 150, 87, 128, 99, 251, 219, 173, 60, 20, 97, 0, 212, 87, 128, 99, 253, 172, 213, 118, 20, 97, 0, 251, 87, 91, 96, 0, 128, 253, 91, 52, 128, 21, 97, 0, 114, 87, 96, 0, 128, 253, 91, 80, 97, 0, 148, 115, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 96, 4, 53, 22, 97, 1, 19, 86, 91, 0, 91, 52, 128, 21, 97, 0, 162, 87, 96, 0, 128, 253, 91, 80, 97, 0, 171, 97, 2, 80, 86, 91, 96, 64, 128, 81, 115, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 144, 146, 22, 130, 82, 81, 144, 129, 144, 3, 96, 32, 1, 144, 243, 91, 52, 128, 21, 97, 0, 224, 87, 96, 0, 128, 253, 91, 80, 97, 0, 233, 97, 2, 108, 86, 91, 96, 64, 128, 81, 145, 130, 82, 81, 144, 129, 144, 3, 96, 32, 1, 144, 243, 91, 52, 128, 21, 97, 1, 7, 87, 96, 0, 128, 253, 91, 80, 97, 0, 148, 96, 4, 53, 97, 2, 114, 86, 91, 96, 0, 128, 84, 115, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 22, 51, 20, 97, 1, 192, 87, 96, 64, 128, 81, 127, 8, 195, 121, 160, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 129, 82, 96, 32, 96, 4, 130, 1, 82, 96, 46, 96, 36, 130, 1, 82, 127, 116, 104, 101, 32, 115, 101, 110, 100, 111, 114, 32, 111, 102, 32, 116, 104, 101, 32, 116, 114, 97, 110, 115, 97, 99, 116, 105, 111, 110, 32, 105, 115, 96, 68, 130, 1, 82, 127, 32, 110, 111, 116, 32, 116, 104, 101, 32, 111, 119, 110, 101, 114, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 96, 100, 130, 1, 82, 144, 81, 144, 129, 144, 3, 96, 132, 1, 144, 253, 91, 129, 144, 80, 128, 115, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 22, 99, 253, 172, 213, 118, 96, 1, 84, 96, 64, 81, 130, 99, 255, 255, 255, 255, 22, 124, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 129, 82, 96, 4, 1, 128, 130, 129, 82, 96, 32, 1, 145, 80, 80, 96, 0, 96, 64, 81, 128, 131, 3, 129, 96, 0, 135, 128, 59, 21, 128, 21, 97, 2, 52, 87, 96, 0, 128, 253, 91, 80, 90, 241, 21, 128, 21, 97, 2, 72, 87, 61, 96, 0, 128, 62, 61, 96, 0, 253, 91, 80, 80, 80, 80, 80, 80, 86, 91, 96, 0, 84, 115, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 22, 129, 86, 91, 96, 1, 84, 129, 86, 91, 96, 0, 84, 115, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 22, 51, 20, 97, 3, 30, 87, 96, 64, 128, 81, 127, 8, 195, 121, 160, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 129, 82, 96, 32, 96, 4, 130, 1, 82, 96, 46, 96, 36, 130, 1, 82, 127, 116, 104, 101, 32, 115, 101, 110, 100, 111, 114, 32, 111, 102, 32, 116, 104, 101, 32, 116, 114, 97, 110, 115, 97, 99, 116, 105, 111, 110, 32, 105, 115, 96, 68, 130, 1, 82, 127, 32, 110, 111, 116, 32, 116, 104, 101, 32, 111, 119, 110, 101, 114, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 96, 100, 130, 1, 82, 144, 81, 144, 129, 144, 3, 96, 132, 1, 144, 253, 91, 96, 1, 85, 86, 0, 161, 101, 98, 122, 122, 114, 48, 88, 32, 239, 116, 234, 141, 150, 80, 154, 103, 36, 128, 216, 187, 80, 39, 157, 228, 200, 62, 221, 3, 108, 112, 210, 223, 13, 54, 210, 153, 35, 119, 0, 119, 0, 41] }, v: 27, r: 0xab8b6f2d61fc1fc8e41d53d2426bcea0f954f51e9bce4bd3878f2aadaefecc85, s: 0x6adde923f95943e388eaa3f3ef18367d965c7c437a16feba12693fdfcbabe52f, hash: 0x79c86612bc63adadc143b17d8733611db61078f2c3f4525a60624c6b1b7f23b0 }, sender: 0x31401412f6902e0cd41822eeced276c80134e916, public: Some(0xc498b808217bcf9b8f8782b545474610e3cfd4416ae06e437c05d33a5471b8e558c1224a81ba09925179438cdc9d2725eb306e8bb92e5dbcb4447595ea0cb5e9) }, condition: None }
ethereum_1              | 2018-08-29 10:45:19 UTC  DEBUG own_tx  Imported to the pool (hash 0x79c86612bc63adadc143b17d8733611db61078f2c3f4525a60624c6b1b7f23b0)
ethereum_1              | 2018-08-29 10:45:19 UTC  DEBUG txqueue  [0x79c86612bc63adadc143b17d8733611db61078f2c3f4525a60624c6b1b7f23b0] Added to the pool.
ethereum_1              | 2018-08-29 10:45:19 UTC  DEBUG txqueue  [0x79c86612bc63adadc143b17d8733611db61078f2c3f4525a60624c6b1b7f23b0] Sender: 0x3140…e916, nonce: 273, gasPrice: 0, gas: 6721975, value: 0, dataLen: 897))
ethereum_1              | 2018-08-29 10:45:19 UTC  TRACE miner  prepare_pending_block: entering
ethereum_1              | 2018-08-29 10:45:19 UTC  TRACE miner  prepare_pending_block: have_work=false
ethereum_1              | 2018-08-29 10:45:19 UTC  TRACE miner  prepare_block: No existing work - making new block
ethereum_1              | 2018-08-29 10:45:19 UTC  DEBUG txqueue  Re-computing pending set for block: 2042961
ethereum_1              | 2018-08-29 10:45:19 UTC  DEBUG miner  Attempting to push 1 transactions.
ethereum_1              | 2018-08-29 10:45:19 UTC  DEBUG miner  Adding tx 0x79c86612bc63adadc143b17d8733611db61078f2c3f4525a60624c6b1b7f23b0 took 2 ms
ethereum_1              | 2018-08-29 10:45:19 UTC  DEBUG miner  Pushed 1 transactions in 2 ms
ethereum_1              | 2018-08-29 10:45:19 UTC  TRACE miner  prepare_work: Checking whether we need to reseal: orig=None last=None, this=0x2d4e5c540732384cd4d7f2e1512d8de63f3910053e6046cd3d8b35172c6dc31c
ethereum_1              | 2018-08-29 10:45:19 UTC  TRACE miner  prepare_work: Pushing a new, refreshed or borrowed pending 0x2d4e…c31c...
ethereum_1              | 2018-08-29 10:45:19 UTC  TRACE miner  prepare_work: leaving (last=Some(0x2d4e5c540732384cd4d7f2e1512d8de63f3910053e6046cd3d8b35172c6dc31c))
ethereum_1              | 2018-08-29 10:45:19 UTC  TRACE miner  prepare_pending_block: Miner received request (was 0, now 2042961) - waking up.

It seems like each node wants to mine its own transactions.

FYI, the configs:

Config non-mining node: https://gist.github.com/roderik/7b4f9ca2d819da534af231268e8750bb
Commandline non-mining node: https://gist.github.com/roderik/5a48fc1ee9d2f1beb8ab0618e0cb0fb0

Config mining node: https://gist.github.com/roderik/e75b62fdbf6e4f06bbada1ca7d5143fa
Commandline mining node: https://gist.github.com/roderik/bdaf7b01bc19be5fbe52b418e6fdd3fe

[roderik]

After a full day of banging my head reverted everything to v1.11.6 and it is working again. Would love to get to the bottom of this together, so ping me when anyone has the time to go through everything for a few hours. I can provide access to the servers, repos, etc.

[Tbaut]

Summarizing our conversation on Gitter:
I spin up a node on 2.0.1 and made a Tx (value transfer) from an address I created to an address found in the chain spec, it was directed to my node. It's visible here: https://explorer.mintnet.settlemint.com/tx/0x1a91e05ae2397532d96d0d7b50f2ff86fc1e81544a7c78a54f6dbcf03fd33e58

The node shows:

 2018-08-30 12:52:00 UTC Imported #2059429 0x3bb7…33fe (0 txs, 0.00 Mgas, 3 ms, 0.57 KiB)
ethereum_1              | 2018-08-30 12:52:05 UTC Transaction mined (hash 0x1a91e05ae2397532d96d0d7b50f2ff86fc1e81544a7c78a54f6dbcf03fd33e58)
ethereum_1              | 2018-08-30 12:52:05 UTC Imported #2059430 0xd103…ae9a (1 txs, 0.02 Mgas, 5 ms, 0.67 KiB)

I made the Tx manually, so the problem could come from truffle. I suggest you try out manually also. e.g. curl --data '{"method":"personal_sendTransaction","params":[{"from":"0xc205a69f31df47533c9474c345daefa1da062b2a","to":"0x0065c7c895af2f707c6cfaafca8d9cec6538e08f","value":"0x"},"your_password"],"id":1,"jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST localhost:8545

[roderik]

Upgraded all the nodes again, and now everything works. 🤯

[wkarshat]

So, just like that, @roderik fiddles with the install till it appears to work and punts on finding a way to replicate and @Tbaut cares not about the root cause?

[Tbaut]

If you read the thread, I have never been able to replicate the problem, there is no evidence that the problem indeed comes from Parity Ethereum. @roderik has closed the issue, we're always happy to reopen if you or anyone encounters the problem. In that case adding anything constructive to the discussion is welcome :).