Prysm: Failed to update assignments error=no signing key found in keys cache

[timjp87]

🐞 Bug Report

Description

When I start the validator it crashes during the attestation schedule check.

Has this worked before in a previous version?

This problem happened after the genesis event. I took down my beacon node to add the 3 additional prysm bootstrap-nodes because the node was not connecting to anyone. I attached to my validator tmux session and saw the crash. I restarted the validator and it crashes again.

I imported the keystore from eth2.0-deposit-cli with 20 or so validators from launchpad with the import function. When I ran the validator after this for the first time it crashed during the unlocking, because my node ran out of ram. Maybe this lead to some corruption?

🔬 Minimal Reproduction

Not sure this can be easily reporduced to be honest.

🔥 Error

alarm@atlas01 ~ % ./validator-v1.0.0-alpha.17-linux-arm64 --graffiti
[2020-08-02 18:31:28] INFO accounts-v2: (wallet path) /home/alarm/.eth2validators/prysm-wallet-v2
[2020-08-02 18:31:28] INFO accounts-v2: (wallet directory) /home/alarm/.eth2validators/prysm-wallet-v2
[2020-08-02 18:31:28] INFO accounts-v2: (account passwords path) /home/alarm/.eth2validators/prysm-wallet-v2-passwords
[2020-08-02 18:31:28] INFO accounts-v2: Successfully opened wallet
[2020-08-02 18:31:28] INFO direct-keymanager-v2: Decrypting validator accounts...
100% [===============================================================================================================================================================================] [33s:0s][2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0x8937728af6ff
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0xb1af9228e3cc
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0xa3e7745dc8bd
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0xa789318c55b2
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0x8eb18058e6fe
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0xaaf7cedc6a6f
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0xad3088473e2f
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0xaff0ec811845
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0x97b4348add54
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0x8dfb754cad07
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0x8532fe648507
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0xad92da8348c9
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0x95d369943c31
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0x924cf2d0118d
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0xa3bcd4c454d2
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0xb6ee16f1d095
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0x8bb22657204b
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0xa5fd5088ae1c
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0x8fe116bc1569
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0x881c35caae21
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0x91f07df0246a
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0xb699491bb0d2
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0xad8f4662b6c6
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0x9495d2dac88c
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0xb4964e32fd8c
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0x986e26d36d5e
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0x8c40c2ba9c91
[2020-08-02 18:32:02] INFO node: Validating for public key pubKey=0xb0a65ada32ae
[2020-08-02 18:32:02] INFO node: Checking DB databasePath=/home/alarm/.eth2
[2020-08-02 18:32:02] INFO node: Starting validator node version=Prysm/v1.0.0-alpha.17/f414cf36b99dea2d997d0141acb3d1a319fb5578. Built at: 2020-07-31 03:04:02+00:00
[2020-08-02 18:32:02] WARN validator: You are using an insecure gRPC connection. If you are running your beacon node and validator on the same machines, you can ignore this message. If you want to know how to enable secure connections, see: https://docs.prylabs.network/docs/prysm-usage/secure-grpc
[2020-08-02 18:32:02] INFO validator: Waiting for beacon chain start log from the ETH 1.0 deposit contract
[2020-08-02 18:32:02] INFO validator: Beacon chain started genesisTime=2020-08-04 13:00:08 +0000 UTC
[2020-08-02 18:32:02] INFO validator: Validator activated index=15170 publicKey=0x8937728af6ff
[2020-08-02 18:32:02] INFO validator: Validator activated index=15178 publicKey=0xb1af9228e3cc
[2020-08-02 18:32:02] INFO validator: Validator activated index=15184 publicKey=0xa3e7745dc8bd
[2020-08-02 18:32:02] INFO validator: Validator activated index=15191 publicKey=0xa789318c55b2
[2020-08-02 18:32:02] INFO validator: Validator activated index=15201 publicKey=0x8eb18058e6fe
[2020-08-02 18:32:02] INFO validator: Validator activated index=15185 publicKey=0xaaf7cedc6a6f
[2020-08-02 18:32:02] INFO validator: Validator activated index=15188 publicKey=0xad3088473e2f
[2020-08-02 18:32:02] INFO validator: Validator activated index=15196 publicKey=0xaff0ec811845
[2020-08-02 18:32:02] INFO validator: Validator activated index=15181 publicKey=0x97b4348add54
[2020-08-02 18:32:02] INFO validator: Validator activated index=15187 publicKey=0x8dfb754cad07
[2020-08-02 18:32:02] INFO validator: Validator activated index=15168 publicKey=0x8532fe648507
[2020-08-02 18:32:02] INFO validator: Validator activated index=15190 publicKey=0xad92da8348c9
[2020-08-02 18:32:02] INFO validator: Validator activated index=15205 publicKey=0x95d369943c31
[2020-08-02 18:32:02] INFO validator: Validator activated index=15176 publicKey=0x924cf2d0118d
[2020-08-02 18:32:02] INFO validator: Validator activated index=15195 publicKey=0xa3bcd4c454d2
[2020-08-02 18:32:02] INFO validator: Validator activated index=15206 publicKey=0xb6ee16f1d095
[2020-08-02 18:32:02] INFO validator: Validator activated index=15169 publicKey=0x8bb22657204b
[2020-08-02 18:32:02] INFO validator: Validator activated index=15180 publicKey=0xa5fd5088ae1c
[2020-08-02 18:32:02] INFO validator: Validator activated index=15189 publicKey=0x8fe116bc1569
[2020-08-02 18:32:02] INFO validator: Validator activated index=15194 publicKey=0x881c35caae21
[2020-08-02 18:32:02] INFO validator: Validator activated index=15167 publicKey=0x91f07df0246a
[2020-08-02 18:32:02] INFO validator: Validator activated index=15198 publicKey=0xb699491bb0d2
[2020-08-02 18:32:02] INFO validator: Validator activated index=15199 publicKey=0xad8f4662b6c6
[2020-08-02 18:32:02] INFO validator: Validator activated index=15182 publicKey=0x9495d2dac88c
[2020-08-02 18:32:02] INFO validator: Validator activated index=15197 publicKey=0xb4964e32fd8c
[2020-08-02 18:32:02] INFO validator: Validator activated index=15192 publicKey=0x986e26d36d5e
[2020-08-02 18:32:02] INFO validator: Validator activated index=15179 publicKey=0x8c40c2ba9c91
[2020-08-02 18:32:02] INFO validator: Validator activated index=15183 publicKey=0xb0a65ada32ae
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=1 pubKeys=[0xaff0ec811845] slot=0
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=1 pubKeys=[0xa5fd5088ae1c] slot=1
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=2 pubKeys=[0x8532fe648507 0xb699491bb0d2] slot=2
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=1 pubKeys=[0x986e26d36d5e] slot=4
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=2 pubKeys=[0x8fe116bc1569 0x8c40c2ba9c91] slot=5
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=1 pubKeys=[0x8dfb754cad07] slot=7
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=1 pubKeys=[0xb1af9228e3cc] slot=8
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=3 pubKeys=[0x8937728af6ff 0xad3088473e2f 0x924cf2d0118d] slot=11
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=1 pubKeys=[0x881c35caae21] slot=14
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=1 pubKeys=[0x9495d2dac88c] slot=15
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=1 pubKeys=[0xa3bcd4c454d2] slot=16
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=1 pubKeys=[0xa789318c55b2] slot=17
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=1 pubKeys=[0x8eb18058e6fe] slot=18
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=1 pubKeys=[0xad8f4662b6c6] slot=20
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=1 pubKeys=[0xaaf7cedc6a6f] slot=21
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=1 pubKeys=[0xa3e7745dc8bd] slot=22
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=1 pubKeys=[0x91f07df0246a] slot=23
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=1 pubKeys=[0xb4964e32fd8c] slot=24
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=1 pubKeys=[0x8bb22657204b] slot=26
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=1 pubKeys=[0xb0a65ada32ae] slot=27
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=1 pubKeys=[0xad92da8348c9] slot=28
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=2 pubKeys=[0x97b4348add54 0x95d369943c31] slot=29
[2020-08-02 18:32:02] INFO validator: Attestation schedule attesters=1 pubKeys=[0xb6ee16f1d095] slot=30
[2020-08-02 18:32:02] ERROR validator: Failed to update assignments error=no signing key found in keys cache
github.com/prysmaticlabs/prysm/validator/keymanager/v2/direct.(Keymanager).Sign
validator/keymanager/v2/direct/direct.go:268
github.com/prysmaticlabs/prysm/validator/client.(validator).signObject
validator/client/service.go:205
github.com/prysmaticlabs/prysm/validator/client.(validator).signSlot
validator/client/aggregate.go:114
github.com/prysmaticlabs/prysm/validator/client.(validator).isAggregator
validator/client/validator.go:520
github.com/prysmaticlabs/prysm/validator/client.(validator).UpdateDuties
validator/client/validator.go:382
github.com/prysmaticlabs/prysm/validator/client.run
validator/client/runner.go:77
runtime.goexit
src/runtime/asm_arm64.s:1148
Failed to sign slot
github.com/prysmaticlabs/prysm/validator/client.(validator).signSlot
validator/client/aggregate.go:116
github.com/prysmaticlabs/prysm/validator/client.(validator).isAggregator
validator/client/validator.go:520
github.com/prysmaticlabs/prysm/validator/client.(validator).UpdateDuties
validator/client/validator.go:382
github.com/prysmaticlabs/prysm/validator/client.run
validator/client/runner.go:77
runtime.goexit
src/runtime/asm_arm64.s:1148
could not check if a validator is an aggregator
github.com/prysmaticlabs/prysm/validator/client.(*validator).UpdateDuties
validator/client/validator.go:384
github.com/prysmaticlabs/prysm/validator/client.run
validator/client/runner.go:77
runtime.goexit
src/runtime/asm_arm64.s:1148

🌍 Your Environment

Operating System:

Arch Linux ARM (aarch64)

What version of Prysm are you running? (Which release)

validator-v1.0.0-alpha.17-linux-arm64

Anything else relevant (validator index / public key)?

[prestonvanloon]

This will be fixed in #6823