Prysm: ERROR sync: Failed to set stream deadline error=stream closed

[q9f]

🐞 Bug Report

Description

A Prysm Node connected to the Schlesi network throws a lot of sync errors about stream closed and cannot write to closed stream.

Has this worked before in a previous version?

This didn't happen previously. So it's either a newly introduced issue or _maybe_ some issue with new clients joining the testnet.

🔬 Minimal Reproduction

Connect a Prysm node from the latest v0.12 branch to the Schlesi testnet

🔥 Error

[2020-05-06 11:41:48]  INFO blockchain: Synced new block block=0xf1c16538... epoch=2022 finalizedEpoch=2020 finalizedRoot=0xf81ba27f... slot=64709
[2020-05-06 11:41:48]  INFO blockchain: Finished applying state transition attestations=1 attesterSlashings=0 deposits=0 proposerSlashings=0 voluntaryExits=0
[2020-05-06 11:41:53] ERROR sync: Failed to set stream deadline error=stream closed
[2020-05-06 11:41:56] ERROR sync: Failed to set stream deadline error=stream closed
[2020-05-06 11:41:56] ERROR sync: Failed to send a chunked response error=cannot write to closed stream
[2020-05-06 11:41:56]  WARN sync: Failed to handle p2p RPC error=cannot write to closed stream peer=16Uiu2HAm3MTyX8oo3KQgoosn3oVkgJgT57tkPKwEBTk7RPWYo2Ng topic=/eth2/beacon_chain/req/beacon_blocks_by_range/1/ssz_snappy
[2020-05-06 11:42:00] ERROR sync: Failed to set stream deadline error=stream closed
[2020-05-06 11:42:02] ERROR sync: Failed to set stream deadline error=stream closed
[2020-05-06 11:42:02] ERROR sync: Failed to send a chunked response error=cannot write to closed stream
[2020-05-06 11:42:02]  WARN sync: Failed to handle p2p RPC error=cannot write to closed stream peer=16Uiu2HAm3MTyX8oo3KQgoosn3oVkgJgT57tkPKwEBTk7RPWYo2Ng topic=/eth2/beacon_chain/req/beacon_blocks_by_range/1/ssz_snappy
[2020-05-06 11:42:07] ERROR sync: Failed to set stream deadline error=stream closed
[2020-05-06 11:42:09] ERROR sync: Failed to set stream deadline error=stream closed
[2020-05-06 11:42:09] ERROR sync: Failed to send a chunked response error=cannot write to closed stream
[2020-05-06 11:42:09]  WARN sync: Failed to handle p2p RPC error=cannot write to closed stream peer=16Uiu2HAm3MTyX8oo3KQgoosn3oVkgJgT57tkPKwEBTk7RPWYo2Ng topic=/eth2/beacon_chain/req/beacon_blocks_by_range/1/ssz_snappy
[2020-05-06 11:42:12]  INFO blockchain: Synced new block block=0xa6c4a4c1... epoch=2022 finalizedEpoch=2020 finalizedRoot=0xf81ba27f... slot=64711
[2020-05-06 11:42:12]  INFO blockchain: Finished applying state transition attestations=2 attesterSlashings=0 deposits=0 proposerSlashings=0 voluntaryExits=0

🌍 Your Environment

Operating System: Arch Linux

What version of Prysm are you running? (Which release) Branch v0.12 @ 4cc7f171f9d08a3f05951d20877f0f6ead444970

[xrow]

I found similar issues in my log

  | time="2020-05-06 14:35:14" level=error msg="Failed to write to stream" error="session shutdown" handler=status prefix=sync
  | time="2020-05-06 14:35:14" level=error msg="Failed to close stream" error="session shutdown" prefix=sync
  | time="2020-05-06 14:35:14" level=warning msg="Failed to handle p2p RPC" error="session shutdown" peer=16Uiu2HAmBxM2za6JNoKgT7patj5ue16wA2yRwqCpe9agZVpkw3jX prefix=sync topic="/eth2/beacon_chain/req/status/1/ssz_snappy"
  | time="2020-05-06 14:35:24" level=warning msg="Resolving fork due to new attestation: head at slot 133344 with weight 82029 is not eligible, finalizedEpoch 4165 != 4165, justifiedEpoch 4166 != 4167" prefix=blockchain
  | time="2020-05-06 14:35:24" level=warning msg="Resolving fork due to new attestation: head at slot 133344 with weight 82042 is not eligible, finalizedEpoch 4165 != 4165, justifiedEpoch 4166 != 4167" prefix=blockchain
  | time="2020-05-06 14:35:24" level=warning msg="Resolving fork due to new attestation: head at slot 133344 with weight 82042 is not eligible, finalizedEpoch 4165 != 4165, justifiedEpoch 4166 != 4167" prefix=blockchain
  | time="2020-05-06 14:35:24" level=info msg="Verified and saved pending attestations to pool" blockRoot=dcbccf74f2ca pendingAttsCount=10 prefix=sync
  | time="2020-05-06 14:35:24" level=warning msg="Resolving fork due to new attestation: head at slot 133344 with weight 82064 is not eligible, finalizedEpoch 4165 != 4165, justifiedEpoch 4166 != 4167" prefix=blockchain
  | time="2020-05-06 14:35:24" level=warning msg="Resolving fork due to new attestation: head at slot 133344 with weight 82064 is not eligible, finalizedEpoch 4165 != 4165, justifiedEpoch 4166 != 4167" prefix=blockchain
  | time="2020-05-06 14:35:24" level=warning msg="Resolving fork due to new attestation: head at slot 133344 with weight 82064 is not eligible, finalizedEpoch 4165 != 4165, justifiedEpoch 4166 != 4167" prefix=blockchain
  | time="2020-05-06 14:35:24" level=warning msg="Resolving fork due to new attestation: head at slot 133344 with weight 82064 is not eligible, finalizedEpoch 4165 != 4165, justifiedEpoch 4166 != 4167" prefix=blockchain
  | time="2020-05-06 14:35:25" level=warning msg="Resolving fork due to new attestation: head at slot 133344 with weight 82064 is not eligible, finalizedEpoch 4165 != 4165, justifiedEpoch 4166 != 4167" prefix=blockchain
  | time="2020-05-06 14:35:25" level=warning msg="Resolving fork due to new attestation: head at slot 133344 with weight 82064 is not eligible, finalizedEpoch 4165 != 4165, justifiedEpoch 4166 != 4167" prefix=blockchain
  | time="2020-05-06 14:35:25" level=warning msg="Resolving fork due to new attestation: head at slot 133344 with weight 82064 is not eligible, finalizedEpoch 4165 != 4165, justifiedEpoch 4166 != 4167" prefix=blockchain
  | time="2020-05-06 14:35:25" level=warning msg="Resolving fork due to new attestation: head at slot 133344 with weight 82064 is not eligible, finalizedEpoch 4165 != 4165, justifiedEpoch 4166 != 4167" prefix=blockchain
  | time="2020-05-06 14:35:25" level=warning msg="Resolving fork due to new attestation: head at slot 133344 with weight 82064 is not eligible, finalizedEpoch 4165 != 4165, justifiedEpoch 4166 != 4167" prefix=blockchain
  | time="2020-05-06 14:35:25" level=warning msg="Resolving fork due to new attestation: head at slot 133344 with weight 82064 is not eligible, finalizedEpoch 4165 != 4165, justifiedEpoch 4166 != 4167" prefix=blockchain
  | time="2020-05-06 14:35:25" level=info msg="Starting next epoch" epoch=4168 finalizedEpoch=4166 justifiedEpoch=4167 prefix=blockchain previousJustifiedEpoch=4166

[0xKiwi]

Thanks for reporting @q9f ! Maybe @farazdagi or @nisdas can look into these errors?

[rauljordan]

Please take a look @farazdagi

[rauljordan]

Can we consider this resolved @farazdagi ? I don't think we've seen for a long time

[rauljordan]

No longer observed for a quite a long time. Closing unless this is still a problem