Jormungandr: Fragment propagation on the network fails

[dkijania]

Describe the bug

Given network scenario:

Fails.

1) One observation is that when sending transaction from unassigned1 to delegated1 via Node1 creates transaction which is pending forever

INFO ][[email protected]:8001]: GET 'fragment/logs'
[INFO ][[email protected]:8001]: fragment logs (1)
[INFO ][[email protected]:8001]: Fragment '512fddc692ada5e8ca28027a3a45407b9dd18b33b21ecc8c4cf4f6ed1a76d7ba' is still pending
[INFO ][[email protected]:8001]: GET 'fragment/logs'
[INFO ][[email protected]:8001]: fragment logs (1)
[INFO ][[email protected]:8001]: Fragment '512fddc692ada5e8ca28027a3a45407b9dd18b33b21ecc8c4cf4f6ed1a76d7ba' is still pending
[INFO ][[email protected]:8001]: GET 'fragment/logs'
[INFO ][[email protected]:8001]: fragment logs (1)
[INFO ][[email protected]:8001]: Fragment '512fddc692ada5e8ca28027a3a45407b9dd18b33b21ecc8c4cf4f6ed1a76d7ba' is still pending
[INFO ][[email protected]:8001]: GET 'fragment/logs'
[INFO ][[email protected]:8001]: fragment logs (1)

2) Also we can observe network errors in logs:

[INFO ][[email protected]:8001]: Sep 04 11:25:29.007 WARN GetBlocks response stream failed: Error { code: Unknown, source: Status { code: Unknown, message: "grpc-status header missing, mapped from HTTP status code 200" } }, node_id: 22741290214948620289923328334713000481, peer_addr: 127.0.0.1:12000, task: network

As a result funds are not transferred between accounts.

full logs.txt

Mandatory Information

1. jcli --full-version output;
   jormungandr 0.3.3 (master-f39cb72, release, windows [x86_64]) - [rustc 1.36.0 (a53f9df32 2019-07-03)]
2. jormungandr --full-version output:
   jormungandr 0.3.3 (master-f39cb72, release, windows [x86_64]) - [rustc 1.36.0 (a53f9df32 2019-07-03)]
3. jormungandr-scenario-test seed:
   90a4097425e406b355a3b90c7cb9ca089edf8e1201dfd8e3bc2fe00e40703df8

To Reproduce
Steps to reproduce the behavior:

1. checkout branch https://github.com/input-output-hk/jormungandr/pull/744
2. run command: jormungandr-scenario-tests --root-dir L2001 --document
3. Observe logs
4. Run jcli commands (get account/message logs) to check status of transaction

Expected behavior
No issues in logs. Funds are being transferred between accounts which is verifiable using jcli (message logs/accounts get/tip)

[mzabaluev]

Transaction propagation is not implemented yet (@NicolasDP @CodeSandwich should we have an issue for that?). The GetBlocks error is interesting on its own, I need to look into it.

[NicolasDP]

Yes. This is a fragment error. Not the whole network.

[NicolasDP]

we will use this issue to monitor the work @CodeSandwich is on at the moment. Is that Okay with you @CodeSandwich ?

[mzabaluev]

The GetBlocks issue may be caused by https://github.com/tower-rs/tower-grpc/issues/202

[NicolasDP]

@dkijania this is supposed to have been fixed now.

[dkijania]

I've restested on:

λ jormungandr --full-version
jormungandr 0.3.1415 (master-df9e278, release, windows [x86_64]) - [rustc 1.36.0 (a53f9df32 2019-07-03)]

There is an improvement: fragment is propagatated from passive node to leader node and from leader logs we can read that fragment is put into block. However, passive node still show it as Pending:

λ jcli rest v0 message logs -h http://127.0.0.1:8001/api
---
- fragment_id: aed0daa4c317cd84986b581f601a9d062f18b84539076e41bbbf2bb25fd7a6b6
  last_updated_at: "2019-09-19T07:40:39.006057+00:00"
  received_at: "2019-09-19T07:40:18.124107700+00:00"
  received_from: Network
  status:
    InABlock:
      date: "0.36"

C:\Work\iohk\jormungandr (master -> origin)
λ jcli rest v0 message logs -h http://127.0.0.1:8000/api
---
- fragment_id: aed0daa4c317cd84986b581f601a9d062f18b84539076e41bbbf2bb25fd7a6b6
  last_updated_at: "2019-09-19T07:40:18.119108300+00:00"
  received_at: "2019-09-19T07:40:18.119108300+00:00"
  received_from: Rest
  status: Pending

127.0.0.1:8000 - passive node which accepted transaction
127.0.0.1:8000 - leader node which created block with incoming transaction

[CodeSandwich]

The root cause is that when block is received, its fragments aren't removed from the fragment pool. This should be harmless in greater scheme of things, but the obsolete fragments should definitely be purged to prevent clutter and memory leak. I'll fix that tomorrow.

[NicolasDP]

I've created a new issue for that. Please follow up in there.

[CodeSandwich]

@dkijania The fix is on branch purge_pool. The logs show that every fragment eventually goes into the blockchain. The test seems to never stop, is that ok? Could you confirm if the issue is fixed?

[NicolasDP]

@CodeSandwich the issue described in is that fragment are not propagated through the network. It is now. The issue with updating the mempool upon receiving blocks is documented in #795 and is implemented in #798