Cardano-node: [BUG] - cardano.node.DnsSubscription:Error

[vae520283995]

Internal/Exernal
External

Summary
Unable to synchronize properly

Steps to reproduce
Steps to reproduce the behavior:
docker run latest
cardano-node-mainnet inputoutput/cardano-node latest e8eb3a838c50 778 MB

Expected behavior
Normal synchronization

System info (please complete the following information):

• OS: Ubuntu
• Version 16.04.10
• Node version latest

Screenshots and attachments
cardano-node-mainnet inputoutput/cardano-node latest e8eb3a838c50 778 MB

cardano-node-mainnet | [4341dab7:cardano.node.ErrorPolicy:Notice:67] [2020-08-18 04:13:26.70 UTC] [String "ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = \"Parallel connection cancelled\", seStack = []}))) 1s",String "ErrorPolicyTrace",String "3.129.199.184:3001"]
cardano-node-mainnet | [4341dab7:cardano.node.DnsSubscription:Error:3856] [2020-08-18 04:13:27.17 UTC] [String "Application Exception: 18.159.88.186:3001 HeaderError (At (Block {blockPointSlot = SlotNo {unSlotNo = 5789249}, blockPointHash = 68914f093daebff961def6df4830a21b2fffccec69442e8e6b780e51c40d5217})) (HeaderProtocolError (HardForkValidationErrFromEra S (Z (WrapValidationErr {unwrapValidationErr = ChainTransitionError [OverlayFailure (VRFLeaderValueTooBig (OutputVRF {getOutputVRFBytes = \"\NUL\NUL\157\185\199\189l\210Y\130\175\220\248\140\242\129\&4!\138\193\175\235\238ch(\\\187\156Y3\239\204lYMF\160\239F.>S\248\193l\CAN\162\181\204\147\241\EOT\183\182P{Rz\\200vM\"}) (0 % 1) (ActiveSlotCoeff {unActiveSlotVal = UnsafeUnitInterval (1 % 20), unActiveSlotLog = -512932943875505334261962382072846}))]})))) (Tip (SlotNo {unSlotNo = 5789244}) 9c925a40b21eb2768f2b07733568f7bd9dbfe74848f0b29755b9e8a230b4f43e (BlockNo {unBlockNo = 4555199})) (Tip (SlotNo {unSlotNo = 6157711}) 9bc99fa3047b1a4476e3bb061bfc04b7b217f83fa403d315cb257a3f146ff93f (BlockNo {unBlockNo = 4573383}))",String "SubscriptionTrace",String "\"relays-new.cardano-mainnet.iohk.io\""] cardano-node-mainnet | [4341dab7:cardano.node.ErrorPolicy:Warning:67] [2020-08-18 04:13:27.17 UTC] [String "ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (HeaderError (At (Block {blockPointSlot = SlotNo {unSlotNo = 5789249}, blockPointHash = 68914f093daebff961def6df4830a21b2fffccec69442e8e6b780e51c40d5217})) (HeaderProtocolError (HardForkValidationErrFromEra S (Z (WrapValidationErr {unwrapValidationErr = ChainTransitionError [OverlayFailure (VRFLeaderValueTooBig (OutputVRF {getOutputVRFBytes = \"\\NUL\\NUL\\157\\185\\199\\189l\\210Y\\130\\175\\220\\248\\140\\242\\129\\&4!\\138\\193\\175\\235\\238c*h(\\\\\\187\\156Y3\\239\\204lYMF\\160\\239F.>S\\248\\193l\\CAN\\162\\181\\204\\147\\241\\EOT\\183\\182P{Rz\200vM\"}) (0 % 1) (ActiveSlotCoeff {unActiveSlotVal = UnsafeUnitInterval (1 % 20), unActiveSlotLog = -512932943875505334261962382072846}))]})))) (Tip (SlotNo {unSlotNo = 5789244}) 9c925a40b21eb2768f2b07733568f7bd9dbfe74848f0b29755b9e8a230b4f43e (BlockNo {unBlockNo = 4555199})) (Tip (SlotNo {unSlotNo = 6157711}) 9bc99fa3047b1a4476e3bb061bfc04b7b217f83fa403d315cb257a3f146ff93f (BlockNo {unBlockNo = 4573383}))))) 200s 200s",String "ErrorPolicyTrace",String "18.159.88.186:3001"]
cardano-node-mainnet | [4341dab7:cardano.node.DnsSubscription:Error:3855] [2020-08-18 04:13:27.36 UTC] [String "Application Exception: 3.133.45.145:3001 HeaderError (At (Block {blockPointSlot = SlotNo {unSlotNo = 5789249}, blockPointHash = 68914f093daebff961def6df4830a21b2fffccec69442e8e6b780e51c40d5217})) (HeaderProtocolError (HardForkValidationErrFromEra S (Z (WrapValidationErr {unwrapValidationErr = ChainTransitionError [OverlayFailure (VRFLeaderValueTooBig (OutputVRF {getOutputVRFBytes = \"\NUL\NUL\157\185\199\189l\210Y\130\175\220\248\140\242\129\&4!\138\193\175\235\238ch(\\\187\156Y3\239\204lYMF\160\239F.>S\248\193l\CAN\162\181\204\147\241\EOT\183\182P{Rz\\200vM\"}) (0 % 1) (ActiveSlotCoeff {unActiveSlotVal = UnsafeUnitInterval (1 % 20), unActiveSlotLog = -512932943875505334261962382072846}))]})))) (Tip (SlotNo {unSlotNo = 5789244}) 9c925a40b21eb2768f2b07733568f7bd9dbfe74848f0b29755b9e8a230b4f43e (BlockNo {unBlockNo = 4555199})) (Tip (SlotNo {unSlotNo = 6157711}) 9bc99fa3047b1a4476e3bb061bfc04b7b217f83fa403d315cb257a3f146ff93f (BlockNo {unBlockNo = 4573383}))",String "SubscriptionTrace",String "\"relays-new.cardano-mainnet.iohk.io\""] cardano-node-mainnet | [4341dab7:cardano.node.ErrorPolicy:Warning:67] [2020-08-18 04:13:27.36 UTC] [String "ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (HeaderError (At (Block {blockPointSlot = SlotNo {unSlotNo = 5789249}, blockPointHash = 68914f093daebff961def6df4830a21b2fffccec69442e8e6b780e51c40d5217})) (HeaderProtocolError (HardForkValidationErrFromEra S (Z (WrapValidationErr {unwrapValidationErr = ChainTransitionError [OverlayFailure (VRFLeaderValueTooBig (OutputVRF {getOutputVRFBytes = \"\\NUL\\NUL\\157\\185\\199\\189l\\210Y\\130\\175\\220\\248\\140\\242\\129\\&4!\\138\\193\\175\\235\\238c*h(\\\\\\187\\156Y3\\239\\204lYMF\\160\\239F.>S\\248\\193l\\CAN\\162\\181\\204\\147\\241\\EOT\\183\\182P{Rz\200vM\"}) (0 % 1) (ActiveSlotCoeff {unActiveSlotVal = UnsafeUnitInterval (1 % 20), unActiveSlotLog = -512932943875505334261962382072846}))]})))) (Tip (SlotNo {unSlotNo = 5789244}) 9c925a40b21eb2768f2b07733568f7bd9dbfe74848f0b29755b9e8a230b4f43e (BlockNo {unBlockNo = 4555199})) (Tip (SlotNo {unSlotNo = 6157711}) 9bc99fa3047b1a4476e3bb061bfc04b7b217f83fa403d315cb257a3f146ff93f (BlockNo {unBlockNo = 4573383}))))) 200s 200s",String "ErrorPolicyTrace",String "3.133.45.145:3001"]

[bmatthewshea]

Same issue.
No docker. Compiled 1.18.1/git-tag myself on relay and producer. same error on both when it gets to that block.
Followed instructions @ https://docs.cardano.org/projects/cardano-node/en/latest/ precisely.

 cardano-cli --version
 cardano-cli 1.18.1 - linux-x86_64 - ghc-8.6
 git rev a4b6dae699fa21dc3c025c8a83d1718475cb3afc

[cardanor:cardano.node.DnsSubscription:Error:170172] [2020-08-18 15:11:04.91 UTC] [String "Application Exception: 54.151.113.34:3001 HeaderError (At (Block {blockPointSlot = SlotNo {unSlotNo = 5789249}, blockPointHash = 68914f093daebff961def6df4830a21b2fffccec69442e8e6b780e51c40d5217})) (HeaderProtocolError (HardForkValidationErrFromEra S (Z (WrapValidationErr {unwrapValidationErr = ChainTransitionError [OverlayFailure (VRFLeaderValueTooBig (OutputVRF {getOutputVRFBytes = \"\\NUL\\NUL\\157\\185\\199\\189l\\210Y\\130\\175\\220\\248\\140\\242\\129\\&4!\\138\\193\\175\\235\\238c*h(\\\\\\187\\156Y3\\239\\204lYMF\\160\\239F.>S\\248\\193l\\CAN\\162\\181\\204\\147\\241\\EOT\\183\\182P{R`z\\200vM\"}) (0 % 1) (ActiveSlotCoeff {unActiveSlotVal = UnsafeUnitInterval (1 % 20), unActiveSlotLog = -512932943875505334261962382072846}))]})))) (Tip (SlotNo {unSlotNo = 5789044}) db29e1e0e9080c96bb41e78b119ffae1097192214c21a6f2d257fb98c8e80933 (BlockNo {unBlockNo = 4555190})) (Tip (SlotNo {unSlotNo = 6197155}) 8fa87b94d8c34ac6eaed669173306d74dc2467a4ed7694657157f7de6361aa51 (BlockNo {unBlockNo = 4575327}))",String "SubscriptionTrace",String "\"relays-new.cardano-mainnet.iohk.io\""]

[mark-stopka]

Known issue in 1.18.1, downgrade to 1.18.0 or wait for 1.18.2 due for release this week... The issue is VRFLeaderValueTooBig, the DNS issue is just a symptom, a smokescreen of sorts.

https://github.com/input-output-hk/cardano-node/issues/1676

[rphair]

We saw this issue today, probably on the basis of having run 1.18.1 for a day or so, even though we downgraded to 1.18.0 a while ago. I posted it here first without noticing that this issue has been closed: https://github.com/input-output-hk/cardano-node/issues/1676#issuecomment-676064337

I haven't found any other way to resolve the problem, so as a last resort fix we are rebuilding the DB on all our nodes.

[cyber-russ]

We saw this issue today, probably on the basis of having run 1.18.1 for a day or so, even though we downgraded to 1.18.0 a while ago. I posted it here first without noticing that this issue has been closed: #1676 (comment)

I haven't found any other way to resolve the problem, so as a last resort fix we are rebuilding the DB on all our nodes.

Saw the same issue. Required 'uncorrupted' DB to be copied from node that had only run 1.18.0.

[mrBliss]

BTW, I think it's enough if you delete the ledger folder. This will require rebuilding the ledger from genesis, which will take a while (but this will be much faster with 1.19.0 than 1.18.0!). If you delete the entire database, including the immutable folder, you'll have to resync the whole chain, which takes much longer of course.

[kevinhammond]

Closing since this relates to a rolled-back version (1.18.1).