Prysm: Add validator keys while validator client is running

[imnisen]

As we can deploy multi validators in one validator client node, currently a restart is the only way to pick up new keys. However, maybe the existed validators need to do their job while restarting, which seems not appropriate. So could we add new keys while node running?

[alonmuroch]

With a remote wallet you could simply add new keys.
Problem arises as the validator is currently also responsible of preventing slashing and if you add dynamically new keys I din't believe it has any db entry for them.

[mcdee]

Some thoughts on this:

• each keymanager is responsible for holding and caching its own keys. We probably want to retain this functionality, as keymanager.FetchValidatingKeys() is called frequently throughout the codebase
  
  
  
  • the alternative would be for the validator to have a key cache separate to the keymanager; although this is possible it would lead to duplication of data between keymanager and validator, so would rather avoid if possible
  
  
• we have three options for update functionality:
  
  
  
  • periodic refresh
    
    
    
    
    
    • this is the simplest option, but it introduces unnecessary work (when there are no changed keys) and requires a balance between responsiveness (lower refresh period) and efficiency (higher refresh period) that could be difficult to provide in a way that meets every case
    
    
    
  
  
  • keymanager push
    
    
    
    
    
    • this is a good option if all keymanagers can provide it. Although I believe we could do this with all current keymanagers, if we consider the case of HSMs that aren't directly aware of which of their keys are being used for validating this is not guaranteed for all implementations
    
    
    
  
  
  • ad-hoc refresh
    
    
    
    
    
    • this provides a good user experience in terms of functionality (user asks for a refresh, keymanager refreshes immediately) but requires some sort of interface to the validator to carry out this operation, so probably involves adding a gRPC server to the validator
    
    
    
  
  

I think that keymanager push is the best technical option, _if_ we can convince ourselves that future keymanagers will be able to provide us with the information. If not, we could go for the ad-hoc refresh. I'll investigate these two options over the next week and report back with my findings.

[mcdee]

I'm inclined to believe that we can move ahead with a transparent internal push system. The way this will work is:

• each keymanager maintains its own cache of validating keys (as it does today)
• the keymanager creates its own method to either find or be informed of changes to the keys it is managing
• the keymanager uses the information obtained from the change notifications to update its cache

This allows each keymanager to decide the best way to provide updates for its keys, whilst retaining the performance of having a local cache.

I'll take a look at implementing this for the wallet and remote keymanagers to see how the theory goes.

The only slight concern with this is that we'll have to introduce a mutex in to each keymanager, to avoid concurrent read/write of the cache. I don't think this should cause contention as we don't ask for keys that frequently, but something to keep an eye on.

[alonmuroch]

@mcdee what about a mechanism that pulls new keys from a remote wallet say once an epoch?
1) New epoch starts
2) key manager asks remote wallet for new keys (that he doesn't have)
3) remote wallet returns an array of those
4) key manager adds them to cache

This way we have a fixed time span from one update to another. Predictable and pretty cheap in terms of communication overhead (and complexity).

[mcdee]

A polling mechanism would be a fallback if I can't convince the remote wallet to have an update channel.

Currently playing with an notification implementation for wallet, that will give me an idea if it's going to be possible to use something similar for remote.

[prestonvanloon]

Marking this as "help wanted". If anyone is interested in implementing this, let me know and I will assign to you.

[mcdee]

I have an initial implementation of this for wallet; if anyone wants to take on something for remote that would be a great help.

[alonmuroch]

@mcdee I don't mind helping. Can you share what you work on?

[mcdee]

@alonmuroch I've just pushed https://github.com/mcdee/prysm/tree/dynamic-wallet-keys which uses fsnotify to track when keys are added to watched wallets for the wallet keymanager. It should be transparent as far as the end user is concerned, so no changes required to their configuration.

Note this is against v0.11 as there have been other changes to wallet in v0.11 and if this went against master it would cause conflicts (plus as it changes wallet functionality it makes sense to keep the changes off the current live testnet).

Definitely worth reviewing the code, and more testing is always good (I ran some initial tests that seem to behave but I'm sure there will be bugs in there somewhere). Please let me know how you get on.

[alonmuroch]

@mcdee I've added some test + fixed an issue I found
https://github.com/alonmuroch/prysm/tree/mcdee/dynamic-wallet-keys

One point I wasn't sure what u aimed for. You've set it to check changes in files so when adding new accounts only the index file changed which causes It to not get the account ID.
I've changed it to trigger on file created.
That limits it to manage only new accounts added to the wallet, did you have something more in mind?