Jormungandr: Leadership schedule may miss tip branch switch and create blocks in dimissied branch

[NicolasDP]

I have been receiving comments regarding some issues stake pools are experiencing when following the creation of consecutive blocks from different stake pools.

Description of the problem

What is being observed

In the current implementation of the leadership schedule it is possible a node creates a block on the parent of the tip instead of the tip itself.

let's say we have a 2 nodes, scheduled to respectively create a Block A and a Block B on 2 consecutive slots.

What we would expect is:

    Parent Block <------ Block A <--------- Block B

However from the PoV of the node creating the block B it may be possible that we have the following:

    Parent Block <--------- Block A
             \
              `------------------------------------- Block B

1. the node creating block B sees the Parent Block as the current tip;
2. the node start creating block B
3. the node create block B with Parent Block as its block ancestor
4. the node receives Block A whilst still creating block B and switch its tip to block A.
5. block B was created still with Block A using Parent Block as parent

what are the consequences

For the blockchain in general, the consequence is that a fork has been created where it could have been avoided. For the stake pools involved, it means one of the 2 blocks won't be created and they will miss on rewards.

Is this a problem? can we mitigate this?

Under discussion

[ilap]

I observed similar issues, at least two times when I monitored my nodes. which seems to me a bug in jormungandr.
Using the similar tree:

                        His time
-------.....-----------|43-------------44---------------45=============46============47
20.25382 (ce3...) <--- |20.25383 (395b...)
             \                          ^ my node starts receiving 8 blocks 25383
               \                        announcement at least 2 in 100ms
                `---------------------------------------|20.25384 (5530.....) My node

Despite I received at least 8 25383 announcement till 45sec, my node created a block /w 25382 as its parent instead of 25383.

<13>1 2020-01-03T19:19:42.395544+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: f2b809764c539f2ff5854d835738f9640bdd466505bb1269, date: 20.25382, parent: 160b84f3684c053e8d84ffa34cc8eb58b4f072f2db0e20a4b39d1fa36ae4bd0f, hash: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, task: block
<13>1 2020-01-03T19:19:42.858247+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 42316fefb65067ec5bac70a5a89902065ff75ed54a8c3ff1, date: 20.25382, parent: 160b84f3684c053e8d84ffa34cc8eb58b4f072f2db0e20a4b39d1fa36ae4bd0f, hash: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, task: block
<13>1 2020-01-03T19:19:42.965248+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: f89b8b04e82fbfdcb5baa67231b25ac10d05a81b1b1a58ae, date: 20.25382, parent: 160b84f3684c053e8d84ffa34cc8eb58b4f072f2db0e20a4b39d1fa36ae4bd0f, hash: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, task: block
<13>1 2020-01-03T19:19:43.460238+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 99cb10f53185fbef110472d45a36082905ee12df8a000006, date: 20.25382, parent: 160b84f3684c053e8d84ffa34cc8eb58b4f072f2db0e20a4b39d1fa36ae4bd0f, hash: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, task: block
<13>1 2020-01-03T19:19:43.544570+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 18be2d9ecae69a101b86eb155819eed53b4ee50fb73ba84d, date: 20.25382, parent: 160b84f3684c053e8d84ffa34cc8eb58b4f072f2db0e20a4b39d1fa36ae4bd0f, hash: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, task: block
<13>1 2020-01-03T19:19:43.747623+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 378a18c72e3aba6d60a7c0a907c23e135da2a12115e4e9d4, date: 20.25382, parent: 160b84f3684c053e8d84ffa34cc8eb58b4f072f2db0e20a4b39d1fa36ae4bd0f, hash: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, task: block
<13>1 2020-01-03T19:19:44.045418+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 751f40ae3136689c28615989dcc597141081f5a3a6624c92, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:44.045477+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: f3d407779f32e4df9a4fc7c924f1dfbb0d369f4d51b5d527, date: 20.25382, parent: 160b84f3684c053e8d84ffa34cc8eb58b4f072f2db0e20a4b39d1fa36ae4bd0f, hash: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, task: block
<13>1 2020-01-03T19:19:44.340540+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: f3d407779f32e4df9a4fc7c924f1dfbb0d369f4d51b5d527, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:44.340566+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: f2b809764c539f2ff5854d835738f9640bdd466505bb1269, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:44.344377+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 574c31e3fa70c59ae10437d83e19c3fa63033129967995fb, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:44.472294+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 42316fefb65067ec5bac70a5a89902065ff75ed54a8c3ff1, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:44.592341+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: cc2af8ab8acf6bc7bbcd2664ee90f8d89417727dce950464, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:44.592588+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 31551925d1ccdd71e7208c064522eccfc973b3cbe864834d, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:44.660441+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: a2bbc371c12bed233f5f3803732ee20f272a7212af9edae4, date: 20.25382, parent: 160b84f3684c053e8d84ffa34cc8eb58b4f072f2db0e20a4b39d1fa36ae4bd0f, hash: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, task: block
<13>1 2020-01-03T19:19:44.663604+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 34c08cd169c3acc847d16212db69e53580ad807eb3795dcb, date: 20.25382, parent: 160b84f3684c053e8d84ffa34cc8eb58b4f072f2db0e20a4b39d1fa36ae4bd0f, hash: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, task: block
<13>1 2020-01-03T19:19:44.664019+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: fda2c52d158c5aae3222f4e18bf48776ca80369a027e4352, date: 20.25382, parent: 160b84f3684c053e8d84ffa34cc8eb58b4f072f2db0e20a4b39d1fa36ae4bd0f, hash: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, task: block
<13>1 2020-01-03T19:19:44.779818+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 6bd656312b458ba3652de93daa1685105cda21ca49011e1d, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:44.780072+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 21222f52c295ff45315049240fc5552a39a0046c8ba39ab6, date: 20.25382, parent: 160b84f3684c053e8d84ffa34cc8eb58b4f072f2db0e20a4b39d1fa36ae4bd0f, hash: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, task: block
<13>1 2020-01-03T19:19:44.780277+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 42ebd140de000500e1a78f14a11ceb78c59c58fa7934a081, date: 20.25382, parent: 160b84f3684c053e8d84ffa34cc8eb58b4f072f2db0e20a4b39d1fa36ae4bd0f, hash: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, task: block
<13>1 2020-01-03T19:19:44.780467+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 1644fb576fbb33e6a87e31b56237520257ba76902050b120, date: 20.25382, parent: 160b84f3684c053e8d84ffa34cc8eb58b4f072f2db0e20a4b39d1fa36ae4bd0f, hash: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, task: block
<13>1 2020-01-03T19:19:44.983390+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 05360d74a238aab5090ef95d6fd008ae0dc9971ab40e26cf, date: 20.25382, parent: 160b84f3684c053e8d84ffa34cc8eb58b4f072f2db0e20a4b39d1fa36ae4bd0f, hash: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, task: block


<13>1 2020-01-03T19:19:45.250151+10:00 pool jormungandr.0.8.5 21938 - -  Leader event started, event_remaining_time: 1s 750ms 8us 759ns, event_end: 2020-01-03T09:19:47+00:00, event_start: 2020-01-03T09:19:45+00:00, event_date: 20.25384, leader_id: 1, task: leadership
<13>1 2020-01-03T19:19:45.256571+10:00 pool jormungandr.0.8.5 21938 - -  receiving block from leadership service, date: 20.25384, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 55306d868f2dfed63b942a8c8fb59803681b2246a8e639bcd46ce9c6b85715de, task: block
<13>1 2020-01-03T19:19:45.257572+10:00 pool jormungandr.0.8.5 21938 - -  block from leader event successfully stored, date: 20.25384, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 55306d868f2dfed63b942a8c8fb59803681b2246a8e639bcd46ce9c6b85715de, task: block
<13>1 2020-01-03T19:19:45.257657+10:00 pool jormungandr.0.8.5 21938 - -  update current branch tip, date: 20.25384, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 55306d868f2dfed63b942a8c8fb59803681b2246a8e639bcd46ce9c6b85715de, task: block

<13>1 2020-01-03T19:19:45.314767+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 9e9fb6e6513da577514d887af39a5da97c975d58d9826818, date: 20.25382, parent: 160b84f3684c053e8d84ffa34cc8eb58b4f072f2db0e20a4b39d1fa36ae4bd0f, hash: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, task: block
<13>1 2020-01-03T19:19:45.316004+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 6f8e69f47ebdecf7f64cf17a28f98d3e5c1566a23a753537, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:45.442025+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: c7cad14c7f800720bf78db89966c98420e38faa7588ae204, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:45.442604+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 384a01e0c91800d1baddfe5d33f004cbf607d86842895023, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:45.963683+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 1e8e6fe86503bd95bdc0584117fd665199ce89b34f235cee, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:45.963848+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 9bc88fda7dc25e1b166f0c753c2ebc7d65f55f589241f4af, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:45.963894+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 9d096f523f215f35ee20ce026313a4333f54f638cf5524d9, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:45.963966+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 40d73edbcc6c950fa4f9834636dacb9dc0ec4e18a2da40fb, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:46.529246+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: f89b8b04e82fbfdcb5baa67231b25ac10d05a81b1b1a58ae, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:46.608426+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 212021290a60b5fa70a026712e5d081c945764db41e4d72d, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:46.609231+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 5765311e6b8358d2c4c699597d5056753b72dcfbec714378, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:46.800020+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: c26ebd7961e9106f0b127230bc7fc6fbd52b920cba670475, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:46.800101+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 51f4b6074da705361999ff256eb734343b9b36a671403161, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:46.800175+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 16512c1492de13972e33787ec820f71bbd1361e1ad23bc4d, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:46.901582+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 572e00f5bb34c69a14bac188693d82a6fd7b7f7c85423519, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:46.991056+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: e9edc8ddd38359e6648026cd5c9b125f0a93477a4678345d, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block
<13>1 2020-01-03T19:19:46.991575+10:00 pool jormungandr.0.8.5 21938 - -  received block announcement from network, from_node_id: 378a18c72e3aba6d60a7c0a907c23e135da2a12115e4e9d4, date: 20.25383, parent: ce3893ec9db6f9eb8c90ebfd9454e0ad395539b4d428b6129d716b595d27c5b5, hash: 395bc613b4608c0ccfc5e6eac3784c17579807f447c7809281909ca19b44fc9f, task: block

[hodlonaut]

Similar problem here - announcement received about a new block with parent X, then two seconds later my own leader event was actioned and created a block with parent X (i.e. fork):

Jan 07 11:57:02.813 WARN received block announcement from network, from_node_id: 1a866537825726db248b5f596fd6e0cd181aaa636f1e389a, date: 24.28301, parent: 3de39c7d43478a3649b24d9bf221cfeb816fe3fd38b7e198d98928972a114695, hash: 992f626af4077cb231972317e262e7ca12ee57e908cc29cd9a6267cb5111d0cc, task: block

Jan 07 11:57:05.008 WARN Leader event started, event_remaining_time: 1s 997ms 674us 404ns, event_end: 2020-01-07T10:57:07+00:00, event_start: 2020-01-07T10:57:05+00:00, event_date: 24.28304, leader_id: 1, task: leadership

Jan 07 11:57:05.015 WARN processing block from leader event, date: 24.28304, parent: 3de39c7d43478a3649b24d9bf221cfeb816fe3fd38b7e198d98928972a114695, hash: 2b2025c798c4e323b8165699a8ec5d48bb6d7a25016da1f19927b98c9d806ad1, task: block

[Linicks]

I'm wondering if I didn't just experience this issue. While watching my nodes tip value and comparing it to the tip on pool tools during my slot time, I noticed that my block height was exactly in sync. However, there was another large fork that was surging behind the tip. I built a block during my slot time, but it was invalid. I've turned my logs down for performance reasons, but have the leader logs for the invalid block:

```

• created_at_time: "2020-01-09T04:46:20.533345198+00:00"
  enclave_leader_id: 1
  finished_at_time: "2020-01-09T06:48:37.017396485+00:00"
  scheduled_at_date: "26.20850"
  scheduled_at_time: "2020-01-09T06:48:37+00:00"
  status:
  Block:
  block: 65168d6b2973e53bf43f01e6858e6a0900f055922534632d67bf5a517f6aa6f1
  chain_length: 83427
  wake_at_time: "2020-01-09T06:48:37.001280461+00:00"

[feqifei]

on epoch 35 my pool faced this behavior 5 times in 5 hours destroying its performance ratio. I think it should be taken seriously in consideration due to consequences on pool performances

edit: 9 times in one epoch

[adaplanet]

I'm also seeing a large percentage of block creating opportunities being reported as created and in the chain by the leader logs of the stakepool, yet they never show up in the chain. The stake node is showing no problems at the time of creation and is in sync with the tip per the community reported tip information and other relay tips in a private cluster. This seems to be happening much more frequently than it was several days ago, possibly due to changing testnet network conditions and/or using v0.8.6. Would be great to get this resolved since as a stakepool operator, the stake pool is online and stable with reliable uptime, yet performance is getting terrible for this reason of minted blocks now rarely making it into the chain for reasons that are not clearly described in the logs anywhere that I can easily see.

[cffls]

Seeing the same in v8.6.0.

Three examples:

Jan 18 20:10:45.004 INFO receiving block from leadership service, date: 36.1714, parent: 1c3553e51ccf740840ae74b64a36c4bbf59019249a1962b4a3661d598e010e3c, hash: e08dd93f9475b78ca1ec6a813ad0712bec4688046f5e463aa418e8fddb8bd20c, task: block
Jan 18 20:10:45.008 INFO block from leader event successfully stored, date: 36.1714, parent: 1c3553e51ccf740840ae74b64a36c4bbf59019249a1962b4a3661d598e010e3c, hash: e08dd93f9475b78ca1ec6a813ad0712bec4688046f5e463aa418e8fddb8bd20c, task: block
Jan 18 20:47:25.001 INFO Leader event started, event_remaining_time: 1s 998ms 327us 741ns, event_end: 2020-01-18T20:47:27+00:00, event_start: 2020-01-18T20:47:25+00:00, event_date: 36.2814, leader_id: 1, task: leadership
Jan 18 20:47:25.003 INFO receiving block from leadership service, date: 36.2814, parent: 688a1f58b80abf7486882c12e2e9eaa43840e6c4d5f430cc40587f9374999ad2, hash: 6fc5e373397edb289512aa654d971c3c2682134baaea399b7944998039b74187, task: block
Jan 18 20:47:25.007 INFO block from leader event successfully stored, date: 36.2814, parent: 688a1f58b80abf7486882c12e2e9eaa43840e6c4d5f430cc40587f9374999ad2, hash: 6fc5e373397edb289512aa654d971c3c2682134baaea399b7944998039b74187, task: block
Jan 18 21:09:01.001 INFO Leader event started, event_remaining_time: 1s 998ms 861us 136ns, event_end: 2020-01-18T21:09:03+00:00, event_start: 2020-01-18T21:09:01+00:00, event_date: 36.3462, leader_id: 1, task: leadership
Jan 18 21:09:01.003 INFO receiving block from leadership service, date: 36.3462, parent: b416ba88ccda8c95145a200b6fb99f75d835f05201ccdf678ecdf3e4542b89bf, hash: 8ef12b5ebbdbdbdfb66df125245de5eb712ad07c313b80a0ab2a6bf9fc1015c6, task: block
Jan 18 21:09:01.008 INFO block from leader event successfully stored, date: 36.3462, parent: b416ba88ccda8c95145a200b6fb99f75d835f05201ccdf678ecdf3e4542b89bf, hash: 8ef12b5ebbdbdbdfb66df125245de5eb712ad07c313b80a0ab2a6bf9fc1015c6, task: block

Take the last missed block as an example. My node considered b416ba88ccda8c95145a200b6fb99f75d835f05201ccdf678ecdf3e4542b89bf (36.3451) as the parent block, but it did received the correct block from the next slot (36.3452), as I can see in the log:

Jan 18 21:08:53.910 INFO received block announcement from network, from_node_id: 0e2054f00067027fec90e6e2d2feb60dff1a02ed37b25a9c, date: 36.3452, parent: b416ba88ccda8c95145a200b6fb99f75d835f05201ccdf678ecdf3e4542b89bf, hash: 1482709b4a9acaba0b282c7c53610f41b4ee75cbc04de5458e114cd51a21ad56, task: block

However, my node still took b416ba88ccda8c95145a200b6fb99f75d835f05201ccdf678ecdf3e4542b89bf as the parent block.
My case is slightly different from the issue in the way that the node is aware of the latest tip (because we can see it received block announcement), but it still chose to use previous block as parent. Is this considered as a bug?

[stakehodlers]

Have been looking through some of these "vaporizing" block issues... wondering if this was also the cause of issue #1608? I had logging set to WARN, so not sure if we have sufficient data to make that determination, but the symptoms seem similar.

[stakehodlers]

1472 is another...

[zmeel]

Probably seeing the same. I have a block scheduled at date 40.21999:
```- created_at_time: "2020-01-22T23:59:20.801286083+00:00"
enclave_leader_id: 1
finished_at_time: "2020-01-23T07:26:55.022032277+00:00"
scheduled_at_date: "40.21999"
scheduled_at_time: "2020-01-23T07:26:55+00:00"
status:
Block:
block: 6b9601a459e08f12b22ddb201afab34ca28cff3d4c5e60dea950766e1e41c445
chain_length: 123476
wake_at_time: "2020-01-23T07:26:55.000437474+00:00"

From my logs:
```Jan 23 07:26:55.000 INFO Leader event started, event_remaining_time: 1s 999ms 513us 490ns, event_end: 2020-01-23T07:26:57+00:00, event_start: 2020-01-23T07:26:55+00:00, event_date: 40.21999, leader_id: 1, task: leadership
Jan 23 07:26:55.012 INFO receiving block from leadership service, date: 40.21999, parent: 7f4675a55d4dc472289ae958d089913b6f9f5c9e558f86710fbf972ff0d2447e, hash: 6b9601a459e08f12b22ddb201afab34ca28cff3d4c5e60dea950766e1e41c445, task: block
Jan 23 07:26:55.067 INFO block from leader event successfully stored, date: 40.21999, parent: 7f4675a55d4dc472289ae958d089913b6f9f5c9e558f86710fbf972ff0d2447e, hash: 6b9601a459e08f12b22ddb201afab34ca28cff3d4c5e60dea950766e1e41c445, task: block
Jan 23 07:26:55.067 INFO update current branch tip, date: 40.21999, parent: 7f4675a55d4dc472289ae958d089913b6f9f5c9e558f86710fbf972ff0d2447e, hash: 6b9601a459e08f12b22ddb201afab34ca28cff3d4c5e60dea950766e1e41c445, task: block

The parent does exist in the chain. Epoch 40, slot 21898. Created at 07:23:33. So, I create my block (with chain length 123476 ) 3 minutes and 22 seconds later. My block is not accepted and there appears to be a block 53d41be708cc8a1eb7544a46cfb642877831a757936597bea04628df752cca41 with the same chain length (123476) in slot 21909. This slot has the same parent as my node received at 07:26:55.

The next block announcement my node gets is: also exists in the chain at slot 22020:

Jan 23 07:27:38.936 INFO received block announcement, hash: 5010560af90b1b669f21441e8ea522659c1b3bd8b5752bd1942b04837a7d3b86, direction: in, stream: block_events, node_id: 2347c2e7d5d42648b005c0183051eb961ff26f5b077dc776, sub_task: server, task: network Jan 23 07:27:38.937 INFO received block announcement from network, from_node_id: 2347c2e7d5d42648b005c0183051eb961ff26f5b077dc776, date: 40.22020, parent: d0dc41597e73b930fe62594ff7649add0a738ee0e13c9fd23599ee7c28f10afe, hash: 5010560af90b1b669f21441e8ea522659c1b3bd8b5752bd1942b04837a7d3b86, task: block Jan 23 07:27:39.037 INFO receiving header stream from network, task: block Jan 23 07:27:39.149 INFO receiving block stream from network, task: block Jan 23 07:27:39.444 INFO switching to new candidate branch, task: block

[zmeel]

Another one:
```enclave_leader_id: 1
finished_at_time: "2020-01-23T08:01:53.030698086+00:00"
scheduled_at_date: "40.23048"
scheduled_at_time: "2020-01-23T08:01:53+00:00"
status:
Block:
block: 39a604a81a8816710f92119134967a4294b319ade0cb99e7b9bb45f218489788
chain_length: 123560
wake_at_time: "2020-01-23T08:01:53.001529830+00:00"

```Jan 23 08:01:52.051 INFO failed to connect to peer, reason: Connection refused (os error 111), node_id: 7d18d0522c6f39bd9120a247c40a3480b9f6cd984d9a030a, peer_addr: 139.162.159.152:3000, task: network
Jan 23 08:01:53.001 INFO Leader event started, event_remaining_time: 1s 998ms 421us 637ns, event_end: 2020-01-23T08:01:55+00:00, event_start: 2020-01-23T08:01:53+00:00, event_date: 40.23048, leader_id: 1, task: leadership
Jan 23 08:01:53.030 INFO receiving block from leadership service, date: 40.23048, parent: 8a647871eb2e2a7e89945272ec426bfaad02ebce51eee16e501cae68a2bbd3eb, hash: 39a604a81a8816710f92119134967a4294b319ade0cb99e7b9bb45f218489788, task: block
Jan 23 08:01:53.097 INFO block from leader event successfully stored, date: 40.23048, parent: 8a647871eb2e2a7e89945272ec426bfaad02ebce51eee16e501cae68a2bbd3eb, hash: 39a604a81a8816710f92119134967a4294b319ade0cb99e7b9bb45f218489788, task: block
Jan 23 08:01:53.097 INFO update current branch tip, date: 40.23048, parent: 8a647871eb2e2a7e89945272ec426bfaad02ebce51eee16e501cae68a2bbd3eb, hash: 39a604a81a8816710f92119134967a4294b319ade0cb99e7b9bb45f218489788, task: block

The parent does exist in the chain. Epoch 40, slot 23041. Created at 08:01:39. So, I create my block (with chain length 123560) 14 seconds later in slot 23048. My block is again not accepted and there appears to be a block 99436812c21bb8053fcf1765c627ec14952d236ecaf6a79976cfca1105285a89 with the same chain length (123560) and parent 8a647871eb2e2a7e89945272ec426bfaad02ebce51eee16e501cae68a2bbd3eb in slot 23044.

It seems that I'm not competing for slots but for block height. Is this behaviour explained by just being out of sync? How can I win this race if my block is already created in a previous slot?

Before 0.8.6 I had no problems creating valid blocks. Once and while I missed a block due to dual assignment. These blocks were always in the chain in the same slot my block was assigned. With 0.8.6 uptime is fabulous and my node appears to be in sync but my blocks hardly make it to the chain. As a consequence stakers are leaving because I perform so badly.

[Straightpool]

Definately a major problem! Is there any progress on this yet? From my point of view many nodes are currently struggling to propagate blocks. This is especially hard on smaller pools like mine. We have dozens of issues with the same main problems all very likely linked to this root cause.

[metastrix]

Well, this is a huge issue for pools with 2-6m stake, bc. if we miss our 1-3 blocks per epoch the ROI and performance metric is destroyedin no minute and the small amount of stakers who gave us there trust is gone. I havnt had any issue before but since 3Days no block was accepted by the network and I created 3 of them per Day.

There was no downtime issue nore a fork issue, I have a completely fine tuned setup with multiple nodes and I am always on top of the chain tip.

blockRecvCnt: 5089 lastBlockContentSize: 0 lastBlockDate: "45.1724" lastBlockFees: 0 lastBlockHash: a906453a9ba0740498f24a523c77d14168b606b6c4dd6b3f8b45fd1ec7db8afe lastBlockHeight: "137804" lastBlockSum: 0 lastBlockTime: "2020-01-27T20:11:05+00:00" lastBlockTx: 0 lastReceivedBlockTime: "2020-01-27T20:11:09+00:00" peerAvailableCnt: "96363" peerQuarantinedCnt: "31559" peerUnreachableCnt: "4448" state: Running txRecvCnt: 5262 uptime: 87299 version: jormungandr 0.8.7-364cd84

Edit:

still having issues in 0.8.9, this epoch, 7 out of 10 blocks disappeared in the air, it all started this week and I cant find anything what could cause the issue its like I am connected to bad peers that do not forward the blocks, my blocks are definitely sent out

`

• created_at_time: "2020-01-30T20:01:23.829574394+00:00"
  enclave_leader_id: 1
  finished_at_time: "2020-01-30T23:15:39.006257098+00:00"
  scheduled_at_date: "48.7261"
  scheduled_at_time: "2020-01-30T23:15:39+00:00"
  status:
  Block:
  block: 88f7244ce1c27954dbc6ee4affa94e85a8d687b92d1d86d75bcc79db31ca1ccd
  chain_length: 147632
  wake_at_time: "2020-01-30T23:15:39.001649377+00:00"

• created_at_time: "2020-01-30T20:01:23.829590383+00:00"
  enclave_leader_id: 1
  finished_at_time: "2020-01-30T23:33:51.003470924+00:00"
  scheduled_at_date: "48.7807"
  scheduled_at_time: "2020-01-30T23:33:51+00:00"
  status:
  Block:
  block: cc83eecf11a31cba92d27caeb775834d15db288f14637b9e32157ceca5417eb2
  chain_length: 147679

• created_at_time: "2020-01-30T20:01:23.829597016+00:00"
  enclave_leader_id: 1
  finished_at_time: "2020-01-31T07:41:53.210604750+00:00"
  scheduled_at_date: "48.22448"
  scheduled_at_time: "2020-01-31T07:41:53+00:00"
  status:
  Block:
  block: c47e13afa29a0b4b693a7fc0d414954f88d6cb40982bf56acb397c906dfe3075
  chain_length: 148838
  wake_at_time: "2020-01-31T07:41:53.208791615+00:00"

• created_at_time: "2020-01-30T20:01:23.829599043+00:00"
  enclave_leader_id: 1
  finished_at_time: "2020-01-31T14:19:45.195568536+00:00"
  scheduled_at_date: "48.34384"
  scheduled_at_time: "2020-01-31T14:19:45+00:00"
  status:
  Block:
  block: d4ec7c943418daa14fb20792953c7fd737dfb90b498c236f6835753ad9a4858c
  chain_length: 149760

• created_at_time: "2020-01-30T20:01:23.829600603+00:00"
  enclave_leader_id: 1
  finished_at_time: "2020-01-31T15:20:27.623061215+00:00"
  scheduled_at_date: "48.36205"
  scheduled_at_time: "2020-01-31T15:20:27+00:00"
  status:
  Block:
  block: f96479f5fc97f727c4628b9bf2534856c1ea62406adfdc5f064b1f43893f3064
  chain_length: 149908

• created_at_time: "2020-01-31T16:37:17.497276096+00:00"
  enclave_leader_id: 1
  finished_at_time: "2020-01-31T17:09:49.019645563+00:00"
  scheduled_at_date: "48.39486"
  scheduled_at_time: "2020-01-31T17:09:49+00:00"
  status:
  Block:
  block: e4eb24c2a06e25a9565070ab549e2933dcced856555ef878b54ac63cc1f6e548
  chain_length: 150170
  wake_at_time: "2020-01-31T17:09:49.001249306+00:00"

• created_at_time: "2020-01-31T16:37:17.497342386+00:00"
  enclave_leader_id: 1
  finished_at_time: "2020-01-31T17:17:59.002552026+00:00"
  scheduled_at_date: "48.39731"
  scheduled_at_time: "2020-01-31T17:17:59+00:00"
  status:
  Block:
  block: 68d8a0a4cb58ca459d0dc1253d170a725055b57ca5c26467101d7ebe99c54b94
  chain_length: 150187
  wake_at_time: "2020-01-31T17:17:59.001419411+00:00"`

`

[metastrix]

1681

1694

[zmeel]

Scheduled slot 52.8624

Feb 04 00:01:05.020 INFO receiving block from leadership service, date: 52.8624, parent: 237d8ba092d12b225d05651fbee21842c1faaebe6a1aebc22d50297089ad7fb8, hash: ae07c3acea93ca24c438ed694f3f499cfc7c20c33838ac367949ce103b284e70, task: block
Feb 04 00:01:05.032 INFO block from leader event successfully stored, date: 52.8624, parent: 237d8ba092d12b225d05651fbee21842c1faaebe6a1aebc22d50297089ad7fb8, hash: ae07c3acea93ca24c438ed694f3f499cfc7c20c33838ac367949ce103b284e70, task: block
Feb 04 00:01:05.033 INFO update current branch tip: 237d8ba0-000276cc-52.8617 -> ae07c3ac-000276cd-52.8624, date: 52.8624, parent: 237d8ba092d12b225d05651fbee21842c1faaebe6a1aebc22d50297089ad7fb8, hash: ae07c3acea93ca24c438ed694f3f499cfc7c20c33838ac367949ce103b284e70, task: block

More than 1 minute later:

Feb 04 00:02:07.992 INFO received block announcement from network, from_node_id: d294daaaf1a73ced99425bde068ae847535cf076f8ac7349, date: 52.8655, parent: 237d8ba092d12b225d05651fbee21842c1faaebe6a1aebc22d50297089ad7fb8, hash: 453a2e2617bfa9434083ee1303faf2739e0f290844d8d7ebb08f892062066d16, task: block
Feb 04 00:02:08.721 INFO received block announcement from network, from_node_id: 2a172cb0ab4f3960338a41d3c94b8e6eac36e790d333707e, date: 52.8655, parent: 237d8ba092d12b225d05651fbee21842c1faaebe6a1aebc22d50297089ad7fb8, hash: 453a2e2617bfa9434083ee1303faf2739e0f290844d8d7ebb08f892062066d16, task: block
Feb 04 00:02:09.602 INFO received block announcement from network, from_node_id: 7760716995c8cefdba1e1d0260b85951590254e29af8c928, date: 52.8655, parent: 237d8ba092d12b225d05651fbee21842c1faaebe6a1aebc22d50297089ad7fb8, hash: 453a2e2617bfa9434083ee1303faf2739e0f290844d8d7ebb08f892062066d16, task: block
Feb 04 00:02:09.670 INFO received block announcement from network, from_node_id: ab2521cca8322d98bfe2abc3439a64a7d345d59ab384f0f1, date: 52.8655, parent: 237d8ba092d12b225d05651fbee21842c1faaebe6a1aebc22d50297089ad7fb8, hash: 453a2e2617bfa9434083ee1303faf2739e0f290844d8d7ebb08f892062066d16, task: block
Feb 04 00:02:22.091 INFO received block announcement from network, from_node_id: f9cca885452763a024f889555a0ee9b47c2c5c979fcea6fe, date: 52.8655, parent: 237d8ba092d12b225d05651fbee21842c1faaebe6a1aebc22d50297089ad7fb8, hash: 453a2e2617bfa9434083ee1303faf2739e0f290844d8d7ebb08f892062066d16, task: block

This later block is accepted in slot 8655, 31 slots later than my scheduled slot!

I have the feeling this issue is increasing in frequency. When it started around halve of my blocks were vaporized this way. Currently, 75% of my blocks vaporize this way. And it is always taken by one of the big pools like the 1PCT cluster.

Looking at the competitive fork page on Pooltools.io it seems to intensify too:
competitive blockheights: 162101, 162103, 162110, 162134, 162146, 162148, 162155, 162162, 162163, 162164 and 162172.

Yesterday, two of my competitive blockheights weren't even listed on Pooltool. So, there is even more.

In conclusion: currently, I only make 25% of assigned slots, the other 75% is lost due to slot or block height competition. This remains a serious problem! Is this still a Proof of Stake protocol? I have the stake so my blocks should be accepted if created correctly (which the log shows). It shouldn't be a battle of resources making it a kind of Proof of Work. If smaller pools keep losing these battles they will drop out and only a few large clusters of stake pools will remain. That's not decentralisation.

Up till now there has been many reports but no feedback from devs. Why? Do they think this is irrelevant? This issue (1513) was opened by a dev 28 days ago. Nothing after that.

[zmeel]

From the same Epoch, 1 block later:
Scheduled slot 52.9283

Feb 04 00:23:03.591 INFO Leader event started, event_remaining_time: 1s 408ms 178us 754ns, event_end: 2020-02-04T00:23:05+00:00, event_start: 2020-02-04T00:23:03+00:00, event_date: 52.9283, leader_id: 1, task: leadership
Feb 04 00:23:03.619 INFO receiving block from leadership service, date: 52.9283, parent: a38cf16a3a02269f3b946decd5b7db0d6f2354575fa135c267d4a2526b1b600b, hash: 28a12578c3dce9d656b5b6711c859f60985f89305fe28a0862fa8f363d429e3e, task: block
Feb 04 00:23:03.629 INFO block from leader event successfully stored, date: 52.9283, parent: a38cf16a3a02269f3b946decd5b7db0d6f2354575fa135c267d4a2526b1b600b, hash: 28a12578c3dce9d656b5b6711c859f60985f89305fe28a0862fa8f363d429e3e, task: block
Feb 04 00:23:03.630 INFO update current branch tip: a38cf16a-000276f0-52.9273 -> 28a12578-000276f1-52.9283, date: 52.9283, parent: a38cf16a3a02269f3b946decd5b7db0d6f2354575fa135c267d4a2526b1b600b, hash: 28a12578c3dce9d656b5b6711c859f60985f89305fe28a0862fa8f363d429e3e, task: block

More than 1 minute later:

Feb 04 00:24:13.601 INFO received block announcement from network, from_node_id: 0e053e48b33f1fe104a53183e5256d844c601bbe0294dcab, date: 52.9318, parent: a38cf16a3a02269f3b946decd5b7db0d6f2354575fa135c267d4a2526b1b600b, hash: 59035505dc8cf94a4f39bcc76251e625134b0d2b3516ea834fb02ac30c20bc01, task: block
Feb 04 00:24:14.334 INFO received block announcement from network, from_node_id: 6000000000edbcadbeffbacdaabccdeefbace00000000004, date: 52.9318, parent: a38cf16a3a02269f3b946decd5b7db0d6f2354575fa135c267d4a2526b1b600b, hash: 59035505dc8cf94a4f39bcc76251e625134b0d2b3516ea834fb02ac30c20bc01, task: block
Feb 04 00:24:14.346 INFO received block announcement from network, from_node_id: 23b3ca09c644fe8098f64c24d75d9f79c8e058642e63a28c, date: 52.9318, parent: a38cf16a3a02269f3b946decd5b7db0d6f2354575fa135c267d4a2526b1b600b, hash: 59035505dc8cf94a4f39bcc76251e625134b0d2b3516ea834fb02ac30c20bc01, task: block
Feb 04 00:24:14.471 INFO received block announcement from network, from_node_id: 153e5d2fbe64b3c0822313196dffb4a3a3de3c8c104af313, date: 52.9318, parent: a38cf16a3a02269f3b946decd5b7db0d6f2354575fa135c267d4a2526b1b600b, hash: 59035505dc8cf94a4f39bcc76251e625134b0d2b3516ea834fb02ac30c20bc01, task: block
Feb 04 00:24:14.970 INFO received block announcement from network, from_node_id: cdd6e41a4de53e33fa9cc9998b526157e6fc2f8b8a87c970, date: 52.9318, parent: a38cf16a3a02269f3b946decd5b7db0d6f2354575fa135c267d4a2526b1b600b, hash: 59035505dc8cf94a4f39bcc76251e625134b0d2b3516ea834fb02ac30c20bc01, task: block
Feb 04 00:24:15.587 INFO received block announcement from network, from_node_id: 40e75526a4e619de570fb474df4b1b4bcac32fb01e89f02b, date: 52.9318, parent: a38cf16a3a02269f3b946decd5b7db0d6f2354575fa135c267d4a2526b1b600b, hash: 59035505dc8cf94a4f39bcc76251e625134b0d2b3516ea834fb02ac30c20bc01, task: block
Feb 04 00:24:16.288 INFO received block announcement from network, from_node_id: 71c901b4605bb496b18ce45b79a81778c9675ece9873649f, date: 52.9318, parent: a38cf16a3a02269f3b946decd5b7db0d6f2354575fa135c267d4a2526b1b600b, hash: 59035505dc8cf94a4f39bcc76251e625134b0d2b3516ea834fb02ac30c20bc01, task: block

This later block is accepted in slot 9318, 35 slots later also by a big pool (SPS)

[metastrix]

Update from my comment above. - 100% of my sheduled blocks where accepted by the network again

A short story:
It got so worse that i lost 90% of my created blocks, ( I looked at this for 10days now ) then yesterday I bought a new public IP address and created a new CentOS VM and started the stakepool from this VM and the new IP.

Well just for the sacke of testing I let both stakepools run under the same node_secret, but as I thought the first stackepool under the old IP still wasnt able to propergate the block fast enough, but the new one was, so this is what I have done.

It would be nice to see If this would be the case for others to.

[zmeel]

It would be nice to see If this would be the case for others to.

Interesting observation!
I seems to have a similar observation: 2 days ago I moved my node to the Digital Ocean platform and thus ran the pool with a new IP. The first 7 blocks were all minted without any battles (could be luck) and then the next 2 were lost due to height battles.

[pigofapool]

Information isn't being propagated through the network fast enough. Either that or the network is segregated somehow.

I had ~160 connections to the node and it was 10 blocks behind at the time the leader event kicked in. CPU usage, memory and network were all low.
Eventually it caught up to the tip, but it was too late by then.

[bergr01]

Not sure what the last comment has to do with this github issue to be honest.

Back to the original problem - my main question is not why different nodes create blocks with the same parent hash, but how a node that does so 30 seconds after the first node that performed such an action (and propagated its block to at least a few other peers, as evidenced by logs) manages to win the fork that ensues... I mean, I'd have thought the block created at t - 30 seconds should have made its way around the topology adequately to survive? Or am I missing some poldercast / Praos nuances here? e.g.

Mar 08 07:57:29.028 INFO received block announcement from network, from_node_id: xxxxx, date: 85.21116, parent: 4409fced85c1070c406df2d74ad6dbf1083cebba386a197e1ea4cd3bb0103fd4, hash: db76c78cebc43bdd5286a5222669ad083dcb86a82ec3f6a5baf4c2efb1c2bd4a, task: block
Mar 08 07:57:29.055 INFO receiving block stream from network, task: block
Mar 08 07:57:29.064 INFO update current branch tip: 4409fced-00044fc9-85.21104 -> db76c78c-00044fca-85.21116, task: block

then over 30 seconds later:

Mar 08 07:58:05.134 INFO received block announcement from network, from_node_id: xxxxx, date: 85.21134, parent: 4409fced85c1070c406df2d74ad6dbf1083cebba386a197e1ea4cd3bb0103fd4, hash: 60cab7133e9e47a21526257b60bdee5a3f759a3cd88a1625ef3b95a8d25017b7, task: block
Mar 08 07:58:05.144 INFO receiving block stream from network, task: block
Mar 08 07:58:05.155 INFO receiving block stream from network, task: block
Mar 08 07:58:05.158 INFO create new branch with tip 60cab713-00044fca-85.21134 | current-tip db76c78c-00044fca-85.21116, task: block

yet somehow it's the 85.21134 block that ends up surviving, credit with the reward, shows up in explorer etc... please explain?