Lighthouse: Optimise validator keypair decryption

[paulhauner]

Problem

With our current default params for EIP-2335 keystores, it can take a few seconds to obtain a BLS keypair from it's encrypted form:

https://github.com/sigp/lighthouse/blob/e5fc6bab485fa54d7e518b325f4eb9201ba5c6a1/crypto/eth2_keystore/src/keystore.rs#L285-L296

When running 10s, 100s or more validators, this represents a significant (minutes or more) delay when starting a validator client.

Potential Solution

Create an encrypted cache-file to store BLS keypairs that have been decrypted from an EIP-2335 wallet. I believe Prysm have already done something like this.

The cache can be a simple flat-file which is effectively a Map<PublicKey, SecretKey>.

Considerations

Concurrency

The cache should never be accessed be separate processes, since this could lead to key duplication and slashing.

I think it is sensible to add a lock file to prevent any concurrent reads/writes to the cache-file.

Consistency

It will be important to ensure that for each entry in the cache there exists an EIP-2335 keystore somewhere with a corresponding public key. I think it would be safest to check this condition before any read or write.

Encryption

I haven't put any thought into a suitable encryption scheme for cache-file, yet. Perhaps @kirk-baird or @carlbeek have some suggestions? :)

[realbigsean]

I can pick this up

[kirk-baird]

If we take the encryption option:

Encrypting a single file can easily be done to provide 128 bits of security. Using AES is the simplest and probably best choice. I'd say AES128-CBC would be the best mode for our situation. Though we could use AES128-ECB if we want to also have concurrency at the risk malicious users may be able to delete part of the cache file.

We would still need a key for AES, a password based kdf would be the easiest option. However, this is an additional password for the users to remember / lighthouse to manage.
It'd make sense to use scrypt as the kdf here since we can leverage the existing code.

The benefit here would that we just do a single scrypt execution, AES decrypt a file and then deserialisation of public key and secret key which should be pretty quick. It would also be good to store the public keys in uncompressed form which will save performing a square root for each key (which is slow).

So to summarise we could use AES128-CBC to encrypt the flattened Map<publicKey, secretKey> and use Scrypt to generate an AES key from a password.

[Karmastic]

To clarify what Prysm does to address this: they have a single keystore file with all keys; importing a key adds it to this file. They encrypt this single file on disk, so there's only one decryption operation done, to decrypt all keys. It sounds equivalent to what you're discussing above.

[blacktemplar]

I am currently working on this issue and had the following idea:

What about using the concatenation of the passwords of all the stored secrets as password for the cache file? This would have the benefit of no new password for the user and if the user removes a password from the password dir, then one would need this password before one could encrypt the cache, so removing a password really ensures that no one without the password can access the secret (also not via the cache).

[paulhauner]

Resolved in #1695