Cardano-node: [BUG] - The return of the epoch + 48 hour rewards calculation freeze

[Straightpool]

Internal/External
External otherwise.

Area
Other Stakepool node operation

Summary
After epoch start + 48hours when rewards calculation takes place the network slows to a halt for about 2,5 minutes. This was fixed with 1.21.1 but returned with 1.23.0.

Steps to reproduce
Steps to reproduce the behavior:

1. Run pool on 1.23.0
2. Wait on epoch switch + 48 hours
   In the worst case have a slot in this time window + 2,5 mintues as I did:
   12 | leader | - | 15897752 | 172952 | 2020-12-08 22:47:23 CET
3. Look at CPU graph and block height graph
   In worst case notice missing block, BP will produce it but it will fail to propagate in time and get lost
4. See error

Expected behavior
A node should sail through reward calculation on 1.23.0 as it did on 1.21.1 without stop of slot production.

System info (please complete the following information):

• OS: Ubunto
• Version: 20.04.1 LTS
• Node version:
  cardano-node 1.23.0 - linux-x86_64 - ghc-8.10
  git rev eed250546fa9acec4c9de557b3e3551c1f682a30

Screenshots and attachments
Shows bp did wake up, compares non-leader to leader count:

CPU spike on all nodes (BPs and Relays)

Stuck blockchain for 2.5 minutes:

Additional context
See old issue which was fixed before

[glitch40]

Have also noticed loss of incoming peers and pauses in transactions at the 48h mark as we experienced previously. Nodes that have been updated to 1.23 experience increasing instability on relays after updating to 1.23 the longer they run 15h or longer from my experience after this time nodes have started to "lag" and lose incoming connections which continues to increase until they are restarted.

[AndrewWestberg]

[papacarp]

About 20% of reporters dropped out (200+ reporters right now)