Prysm: beacon-chain memory leak alpha-6

Created on 7 May 2020 · 4Comments · Source: prysmaticlabs/prysm

🐞 Bug Report

Description

alpha-6 beacon-chain contains a memory leak

Has this worked before in a previous version?

alpha-5 memory leak was not nearly as bad, it would usually take a few days to reach 75% memory usage (on 4GB Raspberry Pi)

🔬 Minimal Reproduction

Run alpha-6 arm64 beacon-chain and observe memory usage over time.

🔥 Error

## 🌍 Your Environment **Operating System:**

  
Raspberry Pi 4 with Ubuntu 20.04 LTS

What version of Prysm are you running? (Which release)

  
alpha-6

Anything else relevant (validator index / public key)?

Monit logs which show the process reaches 75% memory usage about every 3 hours, then monit restarts the process


[UTC May  6 02:22:12] info     : 'ubuntu' Monit 5.26.0 started
[UTC May  6 07:38:16] warning  : 'prysm.sh' mem usage of 75.0% matches resource limit [mem usage > 75.0%]
[UTC May  6 07:40:16] error    : 'prysm.sh' mem usage of 75.0% matches resource limit [mem usage > 75.0%]
[UTC May  6 07:40:16] info     : 'prysm.sh' trying to restart
[UTC May  6 07:40:16] info     : 'prysm.sh' stop: '/usr/sbin/service beacon-chain stop'
[UTC May  6 07:40:23] info     : 'prysm.sh' start: '/usr/sbin/service beacon-chain start'
[UTC May  6 07:42:33] info     : 'prysm.sh' mem usage check succeeded [current mem usage = 19.5%]
[UTC May  6 14:46:38] error    : 'prysm.sh' mem usage of 75.0% matches resource limit [mem usage > 75.0%]
[UTC May  6 14:46:38] info     : 'prysm.sh' trying to restart
[UTC May  6 14:46:38] info     : 'prysm.sh' stop: '/usr/sbin/service beacon-chain stop'
[UTC May  6 14:46:46] info     : 'prysm.sh' start: '/usr/sbin/service beacon-chain start'
[UTC May  6 14:49:01] info     : 'prysm.sh' mem usage check succeeded [current mem usage = 17.9%]
[UTC May  6 17:09:02] error    : 'prysm.sh' mem usage of 75.1% matches resource limit [mem usage > 75.0%]
[UTC May  6 17:09:02] info     : 'prysm.sh' trying to restart
[UTC May  6 17:09:02] info     : 'prysm.sh' stop: '/usr/sbin/service beacon-chain stop'
[UTC May  6 17:09:10] info     : 'prysm.sh' start: '/usr/sbin/service beacon-chain start'
[UTC May  6 17:11:26] info     : 'prysm.sh' mem usage check succeeded [current mem usage = 18.2%]
[UTC May  6 20:33:28] error    : 'prysm.sh' mem usage of 75.8% matches resource limit [mem usage > 75.0%]
[UTC May  6 20:33:28] info     : 'prysm.sh' trying to restart
[UTC May  6 20:33:28] info     : 'prysm.sh' stop: '/usr/sbin/service beacon-chain stop'
[UTC May  6 20:33:36] info     : 'prysm.sh' start: '/usr/sbin/service beacon-chain start'
[UTC May  6 20:35:46] info     : 'prysm.sh' mem usage check succeeded [current mem usage = 18.4%]
[UTC May  6 23:11:48] error    : 'prysm.sh' mem usage of 76.7% matches resource limit [mem usage > 75.0%]
[UTC May  6 23:11:48] info     : 'prysm.sh' trying to restart
[UTC May  6 23:11:48] info     : 'prysm.sh' stop: '/usr/sbin/service beacon-chain stop'
[UTC May  6 23:11:55] info     : 'prysm.sh' start: '/usr/sbin/service beacon-chain start'
[UTC May  6 23:14:06] info     : 'prysm.sh' mem usage check succeeded [current mem usage = 19.3%]

For reference, here are the service configs I created to automatically monitor and restart: https://github.com/prysmaticlabs/prysm/compare/master...mshean:systemd

These warnings occurred when the 75% threshold was passed around 23:11:


May 06 23:08:12 ubuntu bash[49643]: time="2020-05-06 23:08:12" level=warning msg="Subscription next failed" error="subscription cancelled by calling sub.Cancel()" prefix=sync topic="/eth2/f071c66c/committee_index5_beacon_attestation/ssz_snappy"
May 06 23:08:29 ubuntu bash[49643]: time="2020-05-06 23:08:29" level=error msg="Failed to write to stream" error="session shutdown" handler=status prefix=sync
May 06 23:08:29 ubuntu bash[49643]: time="2020-05-06 23:08:29" level=warning msg="Failed to handle p2p RPC" error="stream reset" peer=16Uiu2HAm31kWC17rDHetTv8muX6swwcxktaTqcJsdZBWG1EnMbG5 prefix=sync topic="/eth2/beacon_chain/req/status/1/ssz_snappy"
May 06 23:09:11 ubuntu bash[49643]: time="2020-05-06 23:09:11" level=warning msg="Resolving fork due to new attestation: head at slot 136512 with weight 83357 is not eligible, finalizedEpoch 4264 != 4264, justifiedEpoch 4265 != 4266" prefix=blockchain
May 06 23:09:12 ubuntu bash[49643]: time="2020-05-06 23:09:12" level=warning msg="Resolving fork due to new attestation: head at slot 136512 with weight 83366 is not eligible, finalizedEpoch 4264 != 4264, justifiedEpoch 4265 != 4266" prefix=blockchain
May 06 23:09:12 ubuntu bash[49643]: time="2020-05-06 23:09:12" level=warning msg="Resolving fork due to new attestation: head at slot 136512 with weight 83366 is not eligible, finalizedEpoch 4264 != 4264, justifiedEpoch 4265 != 4266" prefix=blockchain
May 06 23:09:12 ubuntu bash[49643]: time="2020-05-06 23:09:12" level=warning msg="Resolving fork due to new attestation: head at slot 136512 with weight 83376 is not eligible, finalizedEpoch 4264 != 4264, justifiedEpoch 4265 != 4266" prefix=blockchain
May 06 23:09:12 ubuntu bash[49643]: time="2020-05-06 23:09:12" level=warning msg="Resolving fork due to new attestation: head at slot 136512 with weight 83382 is not eligible, finalizedEpoch 4264 != 4264, justifiedEpoch 4265 != 4266" prefix=blockchain
May 06 23:09:12 ubuntu bash[49643]: time="2020-05-06 23:09:12" level=warning msg="Resolving fork due to new attestation: head at slot 136512 with weight 83395 is not eligible, finalizedEpoch 4264 != 4264, justifiedEpoch 4265 != 4266" prefix=blockchain
May 06 23:09:12 ubuntu bash[49643]: time="2020-05-06 23:09:12" level=warning msg="Resolving fork due to new attestation: head at slot 136512 with weight 83395 is not eligible, finalizedEpoch 4264 != 4264, justifiedEpoch 4265 != 4266" prefix=blockchain
May 06 23:09:12 ubuntu bash[49643]: time="2020-05-06 23:09:12" level=warning msg="Resolving fork due to new attestation: head at slot 136512 with weight 83395 is not eligible, finalizedEpoch 4264 != 4264, justifiedEpoch 4265 != 4266" prefix=blockchain
May 06 23:09:12 ubuntu bash[49643]: time="2020-05-06 23:09:12" level=warning msg="Resolving fork due to new attestation: head at slot 136512 with weight 83395 is not eligible, finalizedEpoch 4264 != 4264, justifiedEpoch 4265 != 4266" prefix=blockchain
May 06 23:09:12 ubuntu bash[49643]: time="2020-05-06 23:09:12" level=warning msg="Resolving fork due to new attestation: head at slot 136512 with weight 83395 is not eligible, finalizedEpoch 4264 != 4264, justifiedEpoch 4265 != 4266" prefix=blockchain
May 06 23:09:12 ubuntu bash[49643]: time="2020-05-06 23:09:12" level=warning msg="Resolving fork due to new attestation: head at slot 136512 with weight 83402 is not eligible, finalizedEpoch 4264 != 4264, justifiedEpoch 4265 != 4266" prefix=blockchain
May 06 23:09:12 ubuntu bash[49643]: time="2020-05-06 23:09:12" level=warning msg="Resolving fork due to new attestation: head at slot 136512 with weight 83411 is not eligible, finalizedEpoch 4264 != 4264, justifiedEpoch 4265 != 4266" prefix=blockchain
May 06 23:09:12 ubuntu bash[49643]: time="2020-05-06 23:09:12" level=warning msg="Resolving fork due to new attestation: head at slot 136512 with weight 83411 is not eligible, finalizedEpoch 4264 != 4264, justifiedEpoch 4265 != 4266" prefix=blockchain
May 06 23:09:12 ubuntu bash[49643]: time="2020-05-06 23:09:12" level=warning msg="Resolving fork due to new attestation: head at slot 136512 with weight 83415 is not eligible, finalizedEpoch 4264 != 4264, justifiedEpoch 4265 != 4266" prefix=blockchain
May 06 23:09:12 ubuntu bash[49643]: time="2020-05-06 23:09:12" level=warning msg="Resolving fork due to new attestation: head at slot 136512 with weight 83415 is not eligible, finalizedEpoch 4264 != 4264, justifiedEpoch 4265 != 4266" prefix=blockchain
May 06 23:11:15 ubuntu bash[49643]: time="2020-05-06 23:11:15" level=warning msg="Subscription to new head notifier failed" error="websocket: close 1006 (abnormal closure): unexpected EOF" prefix=powchain

And these warnings showed up near the 20:33 restart:


May 06 20:29:12 ubuntu bash[49054]: time="2020-05-06 20:29:12" level=warning msg="Resolving fork due to new attestation: head at slot 135712 with weight 80461 is not eligible, finalizedEpoch 4239 != 4239, justifiedEpoch 4240 != 4241" prefix=blockchain
May 06 20:29:12 ubuntu bash[49054]: time="2020-05-06 20:29:12" level=warning msg="Resolving fork due to new attestation: head at slot 135712 with weight 80461 is not eligible, finalizedEpoch 4239 != 4239, justifiedEpoch 4240 != 4241" prefix=blockchain
May 06 20:29:12 ubuntu bash[49054]: time="2020-05-06 20:29:12" level=warning msg="Resolving fork due to new attestation: head at slot 135712 with weight 80461 is not eligible, finalizedEpoch 4239 != 4239, justifiedEpoch 4240 != 4241" prefix=blockchain
May 06 20:29:12 ubuntu bash[49054]: time="2020-05-06 20:29:12" level=warning msg="Resolving fork due to new attestation: head at slot 135712 with weight 80461 is not eligible, finalizedEpoch 4239 != 4239, justifiedEpoch 4240 != 4241" prefix=blockchain
May 06 20:29:12 ubuntu bash[49054]: time="2020-05-06 20:29:12" level=warning msg="Resolving fork due to new attestation: head at slot 135712 with weight 80461 is not eligible, finalizedEpoch 4239 != 4239, justifiedEpoch 4240 != 4241" prefix=blockchain
May 06 20:29:12 ubuntu bash[49054]: time="2020-05-06 20:29:12" level=warning msg="Resolving fork due to new attestation: head at slot 135712 with weight 80461 is not eligible, finalizedEpoch 4239 != 4239, justifiedEpoch 4240 != 4241" prefix=blockchain
May 06 20:30:05 ubuntu bash[49054]: time="2020-05-06 20:30:05" level=error msg="Failed to close stream" error="session shutdown" prefix=sync
May 06 20:30:05 ubuntu bash[49054]: time="2020-05-06 20:30:05" level=warning msg="Failed to handle p2p RPC" error="session shutdown" peer=16Uiu2HAkwFbFYvHe7VHcpemg4pwB2XyP1kCMkR53XfB2HTeeNgdS prefix=sync topic="/eth2/beacon_chain/req/status/1/ssz_snappy"
May 06 20:30:06 ubuntu bash[49054]: time="2020-05-06 20:30:06" level=error msg="Failed to write to stream" error="stream reset" handler=status prefix=sync
May 06 20:30:06 ubuntu bash[49054]: time="2020-05-06 20:30:06" level=warning msg="Failed to handle p2p RPC" error="stream reset" peer=16Uiu2HAkwFbFYvHe7VHcpemg4pwB2XyP1kCMkR53XfB2HTeeNgdS prefix=sync topic="/eth2/beacon_chain/req/status/1/ssz_snappy"
May 06 20:30:17 ubuntu bash[49054]: time="2020-05-06 20:30:17" level=error msg="Failed to write to stream" error="session shutdown" handler=status prefix=sync
May 06 20:30:17 ubuntu bash[49054]: time="2020-05-06 20:30:17" level=warning msg="Failed to handle p2p RPC" error="stream reset" peer=16Uiu2HAmFgW6wCSvYS5eYYEbejCaojk5DW2WU7H4pYP45TwG1g6y prefix=sync topic="/eth2/beacon_chain/req/status/1/ssz_snappy"

The 17:09 restart didn't have anything too alarming:


May 06 17:08:03 ubuntu bash[48599]: time="2020-05-06 17:08:03" level=info msg="Synced new block" block=0xa1fe9e62... epoch=4210 finalizedEpoch=4208 finalizedRoot=0xb11a1859... prefix=blockchain slot=134740
May 06 17:08:03 ubuntu bash[48599]: time="2020-05-06 17:08:03" level=info msg="Finished applying state transition" attestations=32 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0
May 06 17:08:16 ubuntu bash[48599]: time="2020-05-06 17:08:16" level=info msg="Synced new block" block=0x0533a1ad... epoch=4210 finalizedEpoch=4208 finalizedRoot=0xb11a1859... prefix=blockchain slot=134741
May 06 17:08:16 ubuntu bash[48599]: time="2020-05-06 17:08:16" level=info msg="Finished applying state transition" attestations=31 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0
May 06 17:08:26 ubuntu bash[48599]: time="2020-05-06 17:08:26" level=info msg="Synced new block" block=0xbc62cbb5... epoch=4210 finalizedEpoch=4208 finalizedRoot=0xb11a1859... prefix=blockchain slot=134742
May 06 17:08:26 ubuntu bash[48599]: time="2020-05-06 17:08:26" level=info msg="Finished applying state transition" attestations=20 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0
May 06 17:08:38 ubuntu bash[48599]: time="2020-05-06 17:08:38" level=info msg="Synced new block" block=0xd0cc4447... epoch=4210 finalizedEpoch=4208 finalizedRoot=0xb11a1859... prefix=blockchain slot=134743
May 06 17:08:38 ubuntu bash[48599]: time="2020-05-06 17:08:38" level=info msg="Finished applying state transition" attestations=12 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0
May 06 17:08:41 ubuntu bash[48599]: time="2020-05-06 17:08:41" level=warning msg="Failed to decode stream message" error="stream reset" peer=16Uiu2HAm3tcqJpyXCyHQv6diAwWYPvZuPXY1sWmiTcH19P3EWRMv prefix=sync topic="/eth2/beacon_chain/req/ping/1/ssz_snappy"
May 06 17:08:50 ubuntu bash[48599]: time="2020-05-06 17:08:50" level=info msg="Synced new block" block=0x6f48ee21... epoch=4210 finalizedEpoch=4208 finalizedRoot=0xb11a1859... prefix=blockchain slot=134744
May 06 17:08:50 ubuntu bash[48599]: time="2020-05-06 17:08:50" level=info msg="Finished applying state transition" attestations=21 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0
May 06 17:09:02 ubuntu bash[48599]: time="2020-05-06 17:09:02" level=info msg="Synced new block" block=0xf76c461f... epoch=4210 finalizedEpoch=4208 finalizedRoot=0xb11a1859... prefix=blockchain slot=134745
May 06 17:09:02 ubuntu bash[48599]: time="2020-05-06 17:09:02" level=info msg="Finished applying state transition" attestations=23 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0

14:46 restart


May 06 14:43:24 ubuntu bash[47262]: time="2020-05-06 14:43:24" level=warning msg="Resolving fork due to new attestation: head at slot 133983 with weight 84297 is not eligible, finalizedEpoch 4184 != 4185, justifiedEpoch 4185 != 4187" prefix=blockchain
May 06 14:43:24 ubuntu bash[47262]: time="2020-05-06 14:43:24" level=warning msg="Resolving fork due to new attestation: head at slot 133983 with weight 84297 is not eligible, finalizedEpoch 4184 != 4185, justifiedEpoch 4185 != 4187" prefix=blockchain
May 06 14:43:24 ubuntu bash[47262]: time="2020-05-06 14:43:24" level=warning msg="Resolving fork due to new attestation: head at slot 133983 with weight 84297 is not eligible, finalizedEpoch 4184 != 4185, justifiedEpoch 4185 != 4187" prefix=blockchain
May 06 14:43:25 ubuntu bash[47262]: time="2020-05-06 14:43:25" level=info msg="Starting next epoch" epoch=4188 finalizedEpoch=4186 justifiedEpoch=4187 prefix=blockchain previousJustifiedEpoch=4186
May 06 14:43:25 ubuntu bash[47262]: time="2020-05-06 14:43:25" level=info msg="Validator registry information" activeValidators=28369 averageBalance="32.26754 ETH" prefix=blockchain totalValidators=28392
May 06 14:43:26 ubuntu bash[47262]: time="2020-05-06 14:43:26" level=info msg="Synced new block" block=0xfb483ec6... epoch=4188 finalizedEpoch=4186 finalizedRoot=0x0cdd0b57... prefix=blockchain slot=134016
May 06 14:43:26 ubuntu bash[47262]: time="2020-05-06 14:43:26" level=info msg="Finished applying state transition" attestations=22 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0
May 06 14:43:31 ubuntu bash[47262]: time="2020-05-06 14:43:31" level=info msg="Synced new block" block=0xeeaf3d8b... epoch=4188 finalizedEpoch=4186 finalizedRoot=0x0cdd0b57... prefix=blockchain slot=134017
May 06 14:43:31 ubuntu bash[47262]: time="2020-05-06 14:43:31" level=info msg="Finished applying state transition" attestations=83 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0
May 06 14:43:35 ubuntu bash[47262]: time="2020-05-06 14:43:35" level=info msg="Synced new block" block=0xeeaf3d8b... epoch=4188 finalizedEpoch=4186 finalizedRoot=0x0cdd0b57... prefix=blockchain slot=134017
May 06 14:43:35 ubuntu bash[47262]: time="2020-05-06 14:43:35" level=info msg="Finished applying state transition" attestations=83 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0
May 06 14:43:35 ubuntu bash[47262]: time="2020-05-06 14:43:35" level=info msg="Verified and saved pending attestations to pool" blockRoot=eeaf3d8be929 pendingAttsCount=117 prefix=sync
May 06 14:43:38 ubuntu bash[47262]: time="2020-05-06 14:43:38" level=info msg="Synced new block" block=0x3752d9d2... epoch=4188 finalizedEpoch=4186 finalizedRoot=0x0cdd0b57... prefix=blockchain slot=134018
May 06 14:43:38 ubuntu bash[47262]: time="2020-05-06 14:43:38" level=info msg="Finished applying state transition" attestations=31 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0
May 06 14:43:41 ubuntu bash[47262]: time="2020-05-06 14:43:41" level=error msg="Failed to write to stream" error="session shutdown" handler=status prefix=sync
May 06 14:43:41 ubuntu bash[47262]: time="2020-05-06 14:43:41" level=error msg="Failed to close stream" error="session shutdown" prefix=sync
May 06 14:43:41 ubuntu bash[47262]: time="2020-05-06 14:43:41" level=warning msg="Failed to handle p2p RPC" error="session shutdown" peer=16Uiu2HAm2y5ZwvHrMaXQajtMCC4NtEahubutoFWJq9UVcfznkrEN prefix=sync topic="/eth2/beacon_chain/req/status/1/ssz_snappy"
May 06 14:43:49 ubuntu bash[47262]: time="2020-05-06 14:43:49" level=info msg="Synced new block" block=0x69d499bc... epoch=4188 finalizedEpoch=4186 finalizedRoot=0x0cdd0b57... prefix=blockchain slot=134019
May 06 14:43:49 ubuntu bash[47262]: time="2020-05-06 14:43:49" level=info msg="Finished applying state transition" attestations=16 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0
May 06 14:44:02 ubuntu bash[47262]: time="2020-05-06 14:44:02" level=info msg="Synced new block" block=0x4d391bae... epoch=4188 finalizedEpoch=4186 finalizedRoot=0x0cdd0b57... prefix=blockchain slot=134020
May 06 14:44:02 ubuntu bash[47262]: time="2020-05-06 14:44:02" level=info msg="Finished applying state transition" attestations=36 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0
May 06 14:44:14 ubuntu bash[47262]: time="2020-05-06 14:44:14" level=info msg="Synced new block" block=0xc98feb12... epoch=4188 finalizedEpoch=4186 finalizedRoot=0x0cdd0b57... prefix=blockchain slot=134021
May 06 14:44:14 ubuntu bash[47262]: time="2020-05-06 14:44:14" level=info msg="Finished applying state transition" attestations=14 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0
May 06 14:44:26 ubuntu bash[47262]: time="2020-05-06 14:44:26" level=info msg="Synced new block" block=0x5e7fae36... epoch=4188 finalizedEpoch=4186 finalizedRoot=0x0cdd0b57... prefix=blockchain slot=134022
May 06 14:44:26 ubuntu bash[47262]: time="2020-05-06 14:44:26" level=info msg="Finished applying state transition" attestations=36 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0
May 06 14:44:30 ubuntu bash[47262]: time="2020-05-06 14:44:30" level=error msg="Failed to write to stream" error="session shutdown" handler=status prefix=sync
May 06 14:44:30 ubuntu bash[47262]: time="2020-05-06 14:44:30" level=warning msg="Failed to handle p2p RPC" error="stream reset" peer=16Uiu2HAm2y5ZwvHrMaXQajtMCC4NtEahubutoFWJq9UVcfznkrEN prefix=sync topic="/eth2/beacon_chain/req/status/1/ssz_snappy"
May 06 14:44:30 ubuntu bash[47262]: time="2020-05-06 14:44:30" level=error msg="Failed to write to stream" error="session shutdown" handler=status prefix=sync
May 06 14:44:30 ubuntu bash[47262]: time="2020-05-06 14:44:30" level=warning msg="Failed to handle p2p RPC" error="stream reset" peer=16Uiu2HAmEXZLXQscbwXQf4WNkccgdsBMkf6hd8Yx9xVgkWnMm5qR prefix=sync topic="/eth2/beacon_chain/req/status/1/ssz_snappy"
May 06 14:44:30 ubuntu bash[47262]: time="2020-05-06 14:44:30" level=error msg="Failed to close stream" error="session shutdown" prefix=sync
May 06 14:44:30 ubuntu bash[47262]: time="2020-05-06 14:44:30" level=warning msg="Failed to handle p2p RPC" error="session shutdown" peer=16Uiu2HAmEXZLXQscbwXQf4WNkccgdsBMkf6hd8Yx9xVgkWnMm5qR prefix=sync topic="/eth2/beacon_chain/req/status/1/ssz_snappy"
May 06 14:44:50 ubuntu bash[47262]: time="2020-05-06 14:44:50" level=info msg="Synced new block" block=0x04848b4a... epoch=4188 finalizedEpoch=4186 finalizedRoot=0x0cdd0b57... prefix=blockchain slot=134024
May 06 14:44:50 ubuntu bash[47262]: time="2020-05-06 14:44:50" level=info msg="Finished applying state transition" attestations=41 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0
May 06 14:45:06 ubuntu bash[47262]: time="2020-05-06 14:45:06" level=info msg="Synced new block" block=0xb13fb5ad... epoch=4188 finalizedEpoch=4186 finalizedRoot=0x0cdd0b57... prefix=blockchain slot=134025
May 06 14:45:06 ubuntu bash[47262]: time="2020-05-06 14:45:06" level=info msg="Finished applying state transition" attestations=54 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0
May 06 14:45:13 ubuntu bash[47262]: time="2020-05-06 14:45:13" level=info msg="Synced new block" block=0x22e8742e... epoch=4188 finalizedEpoch=4186 finalizedRoot=0x0cdd0b57... prefix=blockchain slot=134026
May 06 14:45:13 ubuntu bash[47262]: time="2020-05-06 14:45:13" level=info msg="Finished applying state transition" attestations=20 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0
May 06 14:45:15 ubuntu bash[47262]: time="2020-05-06 14:45:15" level=error msg="Failed to write to stream" error="session shutdown" handler=status prefix=sync
May 06 14:45:15 ubuntu bash[47262]: time="2020-05-06 14:45:15" level=warning msg="Failed to handle p2p RPC" error="stream reset" peer=16Uiu2HAm2y5ZwvHrMaXQajtMCC4NtEahubutoFWJq9UVcfznkrEN prefix=sync topic="/eth2/beacon_chain/req/status/1/ssz_snappy"
May 06 14:45:38 ubuntu bash[47262]: time="2020-05-06 14:45:38" level=info msg="Synced new block" block=0xdf6e4c81... epoch=4188 finalizedEpoch=4186 finalizedRoot=0x0cdd0b57... prefix=blockchain slot=134028
May 06 14:45:38 ubuntu bash[47262]: time="2020-05-06 14:45:38" level=info msg="Finished applying state transition" attestations=22 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0
May 06 14:45:54 ubuntu bash[47262]: time="2020-05-06 14:45:54" level=info msg="Synced new block" block=0xf6059355... epoch=4188 finalizedEpoch=4186 finalizedRoot=0x0cdd0b57... prefix=blockchain slot=134029
May 06 14:45:54 ubuntu bash[47262]: time="2020-05-06 14:45:54" level=info msg="Finished applying state transition" attestations=26 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0
May 06 14:46:02 ubuntu bash[47262]: time="2020-05-06 14:46:02" level=info msg="Synced new block" block=0x4b2ba7eb... epoch=4188 finalizedEpoch=4186 finalizedRoot=0x0cdd0b57... prefix=blockchain slot=134030
May 06 14:46:02 ubuntu bash[47262]: time="2020-05-06 14:46:02" level=info msg="Finished applying state transition" attestations=14 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0
May 06 14:46:10 ubuntu bash[47262]: time="2020-05-06 14:46:10" level=warning msg="Failed to handle p2p RPC" error="stream reset" peer=16Uiu2HAmEXZLXQscbwXQf4WNkccgdsBMkf6hd8Yx9xVgkWnMm5qR prefix=sync topic="/eth2/beacon_chain/req/status/1/ssz_snappy"
May 06 14:46:11 ubuntu bash[47262]: time="2020-05-06 14:46:11" level=error msg="Failed to close stream" error="session shutdown" prefix=sync
May 06 14:46:11 ubuntu bash[47262]: time="2020-05-06 14:46:11" level=warning msg="Failed to handle p2p RPC" error="session shutdown" peer=16Uiu2HAm2y5ZwvHrMaXQajtMCC4NtEahubutoFWJq9UVcfznkrEN prefix=sync topic="/eth2/beacon_chain/req/status/1/ssz_snappy"
May 06 14:46:14 ubuntu bash[47262]: time="2020-05-06 14:46:14" level=info msg="Synced new block" block=0xc4fe44ee... epoch=4188 finalizedEpoch=4186 finalizedRoot=0x0cdd0b57... prefix=blockchain slot=134031
May 06 14:46:14 ubuntu bash[47262]: time="2020-05-06 14:46:14" level=info msg="Finished applying state transition" attestations=15 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0
May 06 14:46:38 ubuntu bash[47262]: time="2020-05-06 14:46:38" level=info msg="Synced new block" block=0xf769b9fc... epoch=4188 finalizedEpoch=4186 finalizedRoot=0x0cdd0b57... prefix=blockchain slot=134033
May 06 14:46:38 ubuntu bash[47262]: time="2020-05-06 14:46:38" level=info msg="Finished applying state transition" attestations=16 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0

Bug Help Wanted

Source

mshean

👍1

Most helpful comment

@prestonvanloon Got it. One thing I thought of, alpha.6 does not have the following fix:
https://github.com/prysmaticlabs/prysm/pull/5737

This was the main memory problem for me post initial sync, it reduces one state copy for every incoming beacon_aggregated_and_proof object

Let's release alpha.7 soon

terencechain on 7 May 2020

👍2

All 4 comments

@mshean to confirm this is not during initial syncing right?

terencechain on 7 May 2020

@terencechain From user reports in the discord, this is after initial sync. Usually in the time range of 12 to 24 hours before it reaches a critical threshold.