Cardano-node: [BUG] - chain stopped for more than 4 minutes on producer + relays

Created on 26 Aug 2020 · 8Comments · Source: input-output-hk/cardano-node

Internal/External

External

Summary
The chain tip stopped moving for more than 240s on all of my nodes on different systems.

Steps to reproduce

N/A

Expected behavior
The chain should probably not stop for so long

System info (please complete the following information):

OS: Debian
Version [Buster]
Node version: 1.19.0

Screenshots and attachments
The five nodes of my pool (ANFRA) got stuck on the same slot (and were automatically restarted independently as a result):

Block: 4,606,358 - Slot: 6,825,594 - Epoch: 213 - Epoch's slot: 172,794 - TxsProcessed: 24,775 - Uptime: 4d 10h 34' 01" - Mem: 1,722,560,512

Here are the logs of one of my relays:
chainStuck-slot6825594-ANFRA-relay2.log.zip

bug

Source

2072

Most helpful comment

We've identified this is a performance problem and the fix should be in the next release (1.20). Please confirm if this is resolved next week (after the release)

vix-io on 15 Sep 2020

👍4

All 8 comments

Ok so checking the explorer we can see that this really happened globally :

block 4606358 made on 2020/08/25 21:44:45 UTC
block 4606359 made on 2020/08/25 21:49:27 UTC

2072 on 26 Aug 2020

Related JIRA ticket: https://jira.iohk.io/browse/CAD-1747

johnalotoski on 31 Aug 2020

👍1

This condition was observed again today. It appears to happen exactly 48hrs after each epoch boundary. Took place across all nodes in my pool. Outage lasts for about 2min. During this time chain density metric from Prometheus also shows a steep vertical drop. Sometimes also accompanied by a burst of TX through mempool ~ 25.

cybershaman73 on 15 Sep 2020

We've identified this is a performance problem and the fix should be in the next release (1.20). Please confirm if this is resolved next week (after the release)

vix-io on 15 Sep 2020

👍4

From what I understand still happening with cardano-node v1.20