Jormungandr: Bootstrap finishes at the tip taken at the time when it was started, possibly missing new blocks

[stanfieldr]

Describe the bug
I have two nodes (different machines) setup on the same internal network. Node A is the pool and Node B is passive. Node A has no trusted peers except Node B. When I successfully bootstrap A from B, node A doesn't have the current block height at the time of finishing the bootstrap, it contains the block height when it started. This is a problem because sometimes 2 or 3 blocks could have been added to the chain and it doesn't seem to catch up. I've noticed my nodes seem to be more fragile the less connections I have open. Perhaps the code should work in such a way that Node A continues to get the new blocks from B for a set period of time (maybe 1 - 2 min) to allow the connections to ramp up so that it is capable of keep the block height by itself.

Mandatory Information

1. jcli 0.8.5 (thread-pool-for-server-tasks-6be7d49c, release, linux [x86_64]) - [rustc 1.40.0 (73528e339 2019-12-16)]
2. jormungandr 0.8.5 (tmp-520e9407, release, linux [x86_64]) - [rustc 1.40.0 (73528e339 2019-12-16)] (basically everything in my custom branch has been merged into master except #1461 )

To Reproduce
Steps to reproduce the behavior:

1. On Node B run jcli rest v0 node stats get --host http://127.0.0.1:8300/api just before or right when the bootstrap starts
2. Wait for Node A to finish bootstrapping
3. On Node A run same command as step 1. The block height will be exactly the same as witnessed in step 1. Depending on timing that may or may not be current block height.

Expected behavior
When the node finishes bootstrapping, it should be at the current height of the node it is bootstrapping from.

[rinor]

When I successfully bootstrap A from B, node A doesn't have the current block height at the time of finishing the bootstrap, it contains the block height when it started.

That is correct. This is how it works right now.
While it will/can be improved, let's check when this causes issues.

This is a problem because sometimes 2 or 3 blocks could have been added to the chain and it doesn't seem to catch up

This shouldn't be a problem in most of the cases (assuming normal network state and node B is not far behind), since once initial bootstrap completed, the node should start subscribing for gossip/block/fragment to the nodes propagated to you from node B (also if node A is publicly reachable, depending on different factors, if node A data are already available/hot on the network gossip, other nodes may also connect to you), and when you receive a new block announcement if your tip is not the parent, the node will request the missing blocks and receive them. Smth like this should be visible also in the logs:

Jan 10 09:35:07.329 INFO received block announcement, hash: 90e760c5e9290333392c7f2318ebc0231cb70dcf8772adffa017b4ceb5bd587e, direction: in, stream: block_events, node_id: 18bf81a75e5b15a49b843a66f61602e14d4261fb5595b5f5, peer_addr: 52.8.15.52:3000, task: network
Jan 10 09:35:07.329 INFO received block announcement from network, from_node_id: 18bf81a75e5b15a49b843a66f61602e14d4261fb5595b5f5, date: 27.25768, parent: ffb2e2e8a598fcaae410c686e44fc3fc12d9b7de78af7333560a8c8125a6e9a8, hash: 90e760c5e9290333392c7f2318ebc0231cb70dcf8772adffa017b4ceb5bd587e, task: block
Jan 10 09:35:07.330 DEBG block is missing a locally stored parent, from_node_id: 18bf81a75e5b15a49b843a66f61602e14d4261fb5595b5f5, date: 27.25768, parent: ffb2e2e8a598fcaae410c686e44fc3fc12d9b7de78af7333560a8c8125a6e9a8, hash: 90e760c5e9290333392c7f2318ebc0231cb70dcf8772adffa017b4ceb5bd587e, task: block
Jan 10 09:35:07.398 DEBG pulling headers, to: 90e760c5e9290333392c7f2318ebc0231cb70dcf8772adffa017b4ceb5bd587e, from: [e9875c17a440396d7b30b1d0c7ba071598670ddca3c8c10a92e0da994e49bdd4, 520d6695e7ac2bea4fb0206e86a0874ef1e01540dbf89a7720ff710b00e2071d, 1738681e9e0674b5895af6b9465ef79810b29b6fa0a4a038554f365a7d10a8bf, 1902e50257c5dc444df27df4b82c382faeb6030bafd6d28f855080934a02dda2, 2898d705aae65ba15d8c65862c029f50c6ad879a3edcb5e7c2afe55a05cf0be7, 77bc8f5b0c584920aa76e59dc6573084c9398b1d413f072b83e99e5b697e0914, 624cf457a5476cbdcc11e308932d2fe49fe1677d790fea8d4a02a91f049db1ed, 6480e411242c6008bfbceb85634453e18e7ed3a381537d2357bda9503f42f2f9, 4ec0cb36c64230e98992e5d8dd2e494b3e5b159e83fe02a8f2565603a02c4c36], node_id: 18bf81a75e5b15a49b843a66f61602e14d4261fb5595b5f5, task: network
...
Jan 10 09:35:07.862 DEBG propagating block, hash: 90e760c5e9290333392c7f2318ebc0231cb70dcf8772adffa017b4ceb5bd587e, task: network

and your node should catch up properly.

This means that indeed node A will be behind X blocks, until the first valid/interesting block announcement is received and among other factors the number of connected nodes plays a role, hence your observation:

I've noticed my nodes seem to be more fragile the less connections I have open.

matches and makes sense.

Now, if node A is not an active Leader (meaning won't produce blocks before it catches up, or was a passive one), the fact that stays behind for some time until block announcement is received as mentioned before, would't be much of a problem (still is smth that needs improvement).

---

The story changes if node A while in that X blocks behind state is elected as slot leader to produce/sign block(s), since now indeed we may have a problem that may cause a fork that may cause our produced block(s) to be discarded later (since we build upon actual tip as parent - simplified explanation). _This case is one of the biggest problems_ (related to this reported issue) and frequently active leader nodes restarts reported/done from the pool operators may be affected and causing further network issues/forks.

---

There are other cases when the node may not catch up, causing a node to become a zombie, but will skip them for now since some are mostly related to the actual network/bootstrap stability state. Can elaborate further if needed.

Thank you for the report.

[thiago-tsnnst]

if node A data are already available/hot on the network gossip, other nodes may also connect to you)
@rinor

"if node A data are already available/hot on the network gossip"

If i drop my gossip to 3s for example, it will help my other node to bootstrap, or improve the chances to get the last blocks that are missing after the bootstrap?

[rinor]

@thiago-tsnnst related to:

If i drop my gossip to 3s for example, it will help my other node to bootstrap, or improve the chances to get the last blocks that are missing after the bootstrap?

it depends, while it may help, at expence of higher bandwidth and other resources usage, it may not be always the case. Will check and explain more a bit later.

just to avoid any missunderstanding related to:

also if node A is publicly reachable, depending on different factors, if node A data are already available/hot on the network gossip

is related to the case of a node, publicly reachable, that was up and running (so the ip:port/id are part of the gossip and are broadcasted to the network), and the operator decides to restart so that node goes again through the bootstap process and once you reach at a point like ex:

Jan 10 13:00:37.094 INFO listening and accepting gRPC connections, local_addr: 127.0.0.1:9001, task: network

other nodes, those having within their gossip cache your ip:port/id (assuming you did not change them), may reconnect to you, hence you have some reports from users that have nodes that fit the profile, having some success when they restart the nodes even without any trusted peer configured or when they fail.

[stanfieldr]

Thanks @rinor for the detailed responses. Both nodes are publicly accessible (distinct public ips) and have this setting

"topics_of_interest": {
  "blocks": "high",
  "messages": "high"
}

I've also tested with various gossip_intervals like 500ms, 2s and 10s. I didn't realize blocks could be requested if it was behind, that's nice. Wouldn't that mean Node A should have been updated to the same block height as Node B within 10s using the default gossip_interval? I have observed Node A fall behind in block height > 10s and had pooltool open on another monitor and it wasn't reporting any forks. Is this potentially a bug with Node B not firing the block announcements? I'll do some more testing later tonight. Thank you for the response.

[mzabaluev]

This is partly a server-side problem, as the tip hash in the implementation of PullBlocksToTip gets fixed at the start of the processing; updates to the tip while the response is being streamed are not included.

A comparatively simple way to fix it on the client side is to have the bootstrapping node subscribe to block announcements from the trusted peer it connects to, and fetch any blocks announced while the stream is being processed.

Going forward, I think we should rework bootstrapping to use the same algorithm as pulling missing blocks online (once that is stable enough).

[thiago-tsnnst]

A comparatively simple way to fix it on the client side is to have the bootstrapping node subscribe to block announcements from the trusted peer it connects to, and fetch any blocks announced while the stream is being processed.
@mzabaluev

is there any other config we can do on config file to achieve this subscribe option, or just the node be listed as trusted peer is enough?

[mzabaluev]

is there any other config we can do on config file to achieve this subscribe option

No, this was meant to be an implementation note for the future fix; this behavior during subscribe is not implemented yet.

[michaeljfazio]

I just got a dud bootstrap of an official IOHK peer. Dud meaning that the particular peer which I was unfortunately enough to select only had a fraction of the total blockchain (a few epochs). This resulted in my node syncing in-kind to just a few epochs at which point it assumed I was close enough to the actual "tip" to start receiving blocks and get up to date. Of course I was too far behind and the node just lingered listening to much newer blocks but not being able to actually sync.

Perhaps if we can detect this state, the node can just drop back into bootstrap again until it can keep up from block announcements alone.

[mzabaluev]

As discussed with @rinor, a subscription during bootstrap is not necessary for the simplest viable fix; instead, Tip can be queried from one of the trusted peers after processing the stream of blocks, and if the tip is found to have advanced, another pull can be initiated until the local tip catches up with the tip on the peer.