Jormungandr: node panicked - "cannot process leadership block" - cluster with 2 nodes on local pc; v0.7.0-rc4

[dorin100]

Describe the bug
Node panicked when the second node started to create blocks.

Mandatory Information

jcli 0.7.0-rc4 (HEAD-56906548, release, windows [x86_64]) - [rustc 1.38.0 (625451e37 2019-09-23)]
jormungandr 0.7.0-rc4 (HEAD-56906548, release, windows [x86_64]) - [rustc 1.38.0 (625451e37 2019-09-23)]

To Reproduce
Steps to reproduce the behavior:

1. delete the storage files for both nodes
2. start node1 - jormungandr ---genesis-block block-0.bin --config node_config.yaml --secret node_secret.yaml
3. start node2 - jormungandr --config node_config.yaml --secret node_secret.yaml --genesis-block-hash c5a7f2b5832a0e1715880ef2c8eadb2d8e6f0ca2fd60bf83d3aa164539df6da6
4. send funds from faucet to accout that would delegate to node2 - bash send-money.sh ca1sku4yc68q9cuwlsg296l6yrk6ccth2pzqu26r4s5r8enymyfeyptgl5vxk4 28000000000000 9001 ed25519e_sk18r7nd20gaxjfgmahyqu2vngv98leqefcdcft2nevcakpf999spx55t4ph8ryqslp6ac7uryekjcqsqzl63rjpmh0k92dvquesweq38cc8a0wc
5. delegate accout from step 3 to node2's stake pool - bash delegate-account.sh 223fe74ad117c5a012d4e08eb585803787766f784a3b5addbe0d753d39ab0966 9002 ed25519e_sk1xpdwnr5wx54mh4fpexxsyrc9n0zejfnf7pczr0zr9c3ym98wgpde2drs8t6lkshgeg5z4n2l7lkgkj4kyw97a62mljsrljze9j0c6zgh5j28d
6. wait for 2 epochs to pass
7. at the begining of the third epoch, node1 is crashing; node2 continues to create blocks

Expected behavior
The nodes should not crash and each one should create valid blocks based on the staked percent.

Additional context

• cluster of 2 nodes on the same win 10 machine
• node1 acts as faucet; node2 acts as stake pool; both stake pool certificates are set in genesis file at startup
• "consensus_genesis_praos_active_slot_coeff": 1
• "slot_duration": 2
• "slots_per_epoch": 50
• node 1 files
  node1.zip
  node1_logs.txt
• node 2 files
  node2.zip

[dorin100]

PS: with the same configuration/steps, sometimes I receive the above crash, other times I get the error explained here - https://github.com/input-output-hk/jormungandr/issues/1056#issuecomment-549101002

[rinor]

This (NonMonotonicDate) is reported/similar also in https://github.com/input-output-hk/jormungandr/issues/1024 and https://github.com/input-output-hk/jormungandr/issues/975

[dorin100]

Yeap. @rinor is right. I added it separately as this should be easier to reproduce and maybe to investigate (as there are only 2 nodes).

[NicolasDP]

Not a duplicate of #1024 (it is more related to a local time change). However the exciting thing is that this is certainly linked, helping to pin point what may have gone wrong here.