Venus: Differentiate initial/catch-up chain sync from maintaining sync

[ZenGround0]

Description

Change the syncer interface to expose an explicit initial (or catch-up) sync method, which is given a head CID to sync to. Distinguish this from the maintenance of sync, which happens via the HandleNewTipset method.

At boot, some node activities (connecting to pubsubs, mining) should be gated behind the initial sync completing.

The head CID is to be obtained by a low-trust bootstrapping mechanism (outside scope here).

Acceptance criteria

When a node restarts or goes to sleep for a long time, it should traverse an arbitrarily long head. This "catch up" should happen automatically on restart. (In future, probably also in case a node detects it has been partitioned from the majority network for some time).

During normal operation a node should respect the maxNewChainLen limit and not swap its head for a fork that exceeds this.

Risks + pitfalls

Chain sync is security-sensitive work. It's important for us to have a thorough understanding of behaviour and good tets.

Where to begin

Modify the Syncer interface to differentiate between the two cases with a new method or an argument flag. Now call the syncer during the node startup process, probably after Load completes on the head of the nodes being bootstrapped off of. This requires messing with the startup flow so that bootstrapping occurs after chain loading.

[phritz]

See #1160, which depends on this one

[anorth]

This issue was filed a long time ago - it's still basically something we want, I'll update the description to match our intentions.

[frrist]

With the merge of #3111 we are getting closer to pushing this issue to completion. However, one largish piece of incomplete works that still sticks out to me is the ability for a node to determine if it has "CaughtUp" yet.

In #3111 we have cases for a "Syncing" mode and a "CaughtUp" mode but are still lacking the ability to determine when to switch from one to the other. Here is a way I could imagine that working:

Whenever a Filecoin node starts, it creates a Syncer and places it in “Syncing” mode with block/message pubsub turned off. The filecoin node will then “Say Hello” to the bootstrap peers and receive their current heads. The node then begins fetching and processing the chain head it gets from the bootstrappers. During this fetching/processing time the network will continue to grow the chain, and If the node takes longer than finality blocks to fetch and process it’s chain it will need to remain in “Syncing” mode. I believe this means at the end of every “Syncing” run we will need to “Say Hello..again” to bootstrap nodes in order to get the CID of the latest head. If the height of the new chain head is [some-number-probably-600-ish] ahead of our own we will want to reamine in “Syncing” mode, else we can switch to “CaughtUp”.

I propose at the end of every “Syncing” run (where “run” is a call to HandleNewTipset) we fetch the head from a bootstrap node, inspect its height and either continue “Syncing” or enter “CaughtUp” mode. To facilitate this, there are 2 changes I propose we make:

1. The method HandleNewTipset will fetch/accept a TipSet instead of a CID. This will do two things:
   
   
   
   • Allow the syncer to inspect the height of the TipSet and determine if it should switch to “CaughtUp” mode. (This could also be handled a level up and a flag could be passed to HandleNewTipset that indicates if the new head is trusted, if it is trusted ignore finality rules)
   
   
   • Allow the GraphSyncer fetcher to just do GraphSync -- Currently the GraphSync fetcher must first make a request for the TipSet to obtain the CID(s) for its parent(s). Doing this a level above obviates this requirement of the GraphSync fetcher.
   
   
2. The “Hello” protocol is extended such that nodes can "Hello Again" in order to get their most recent head. A call to this method will be triggered at the end of every “Syncing” run.

This “hello again thingy” could also be the job of something like a Peer Manager thingy, but for now the ~simple~ easy thing seems to be extending the hello protocol with an exported method that can be triggered from the syncer.

---

If this sounds reasonable to @anorth and @ZenGround0 I can write up issues for the steps described above.

---

Edit: I should probably have replaced "bootstrap node" with "trusted node" in the above.

[anorth]

We should design something here that will still make sense when the bootstrap protocol does not rely on "trusted" nodes. And also something that we can conceive of as still working after we face up to the reality of outages, temporary network partitions etc. To that end, I would like to avoid the need for a stateful "mode" of the syncer. However, we do need to solve the problem you point out that initial sync will take longer than the threshold we're likely to choose.

An alternative proposal:

• Add a flag to HandleNewTipset indicating whether the tip is trusted for fetching of a loooong chain
• The bootstrap/peer thingy does its thing periodically, not just once. Say once every few hours (less than whatever exclusion window). Each time, it pushes the consensus head tipset it determines to HandleNewTipset with trusted=true. For a caught up node, this will usually be no-op. For a node that has suffered but recovered from some partition, this will recover it as if it had been rebooted. If an initial sync takes longer than this refresh period, then it will be immediately re-executed.
• The pubsub hookup is deferred until at least the first bootstrap tip has been synced. Tips from there have trusted=false.

The bootstrap/peer thing is currently just to ask any node (which happen to be the bootstrap nodes initially). I concur we might desire the ability to re-trigger Hello. Later this will be replaced with #2674 as an ongoing process.

With this process, we don't need an explicit mode, though the syncer can still provide status about whatever current "run" it is executing.

I don't understand your point (1) above. Note that the Hello protocol does not provide a tipset, only the CIDs, so we need a protocol spec change to achieve it. But if there's no mode to switch, then the first sub-bullet goes away. The second bullet doesn't make sense to me (but the graphsync fetcher isn't done yet). Given a tipset key, fetching that tipset to get its parents gives you ... another tipset key. How is the situation now different?

Giving the syncer access to peers and hello directly is a shortcut to be avoided - it's easy not simple. The peer manager / bootstrapper should be the sole source of trusted heads, on a schedule that it determines.

[anorth]

I think we have enough here and in other issues that we don't need to file more to break the work down. Some decisions we can't make until we see how the code falls out, and that's ok.

[sternhenri]

Jumping in here to explain some of what went down in specsland about this (feel free to ignore if not at all relevant to your ongoing work):

• we had a proposal for some version of secure bootstrapping using trusted nodes to make sure a user wasn't eclipsed.
• we ended up blowing up that section of the spec, based on legitimate feedback that was mostly up to implementation.

That said, I'll add that the thread model I had in mind was what if you get eclipsed, ie first node you connect to connects you to a cluster they control (or collude with) and you never get to see the real chain, so you commit your power to some bogus one (potentially trying to catch up with main one).

Imo this is a bit of an "academic" problem in that I'd argue that all you need is a few websites publishing the nodes they're currently mining on: if PL, EFF, Archive.org (random examples) all ran a page that said current head is [x], then you would need an attacker to control your entire access to the internet in order to sync to the wrong head. If that's the case, all bets are off imo. At least that's my read for MVP.

[frrist]

Problems I think we have identified that need solving:

1. How do we hear about new heads when we are not subscribed to the block/message topic?
   
   
   
   • This sounds like it will be the job of the periodic bootstrap thingy
   
   
2. When do we subscribe to the block/message topic?
   
   
   
   • After we perform one sync with a head from a bootstrapper
   
   
3. How do know when to apply the finality limit check to a potential head? @sternhenri I heard we have plans to remove/modify this check, is that something being discussed?
   
   
   
   • If the head is not received from a trusted/bootstrapper peer.
   
   

@anorth does the above look right to you? If so I can start sketching some of it.

---

I don't understand your point (1) above.

I was proposing that we somehow (e.g. bitswap) obtain a TipSet (instead of just a TipSetKey) before calling into the fetcher. Reasons being:

• It will hopefully clean up this code, waiting on clification from @hannahhoward. (this is the reason for the second bullet)
  
  
  
  • It will clean up this code:
    
    https://github.com/filecoin-project/go-filecoin/blob/2ae7aa72d714379e56f5b9e73a47209a998fbae3/chain/syncer.go#L357-L366
  
  

Giving the syncer access to peers and hello directly is a shortcut to be avoided - it's easy not simple. The peer manager / bootstrapper should be the sole source of trusted heads, on a schedule that it determines.

We need a way to continue to hear about new heads while we are unsubscribed from the block topic, this was a means to accomplish that -- I don't really like it either. The Peer Manager / Bootstrapper thingy seems better suited for these needs. Looks like #2674 is setup to give us this.

[sternhenri]

To your question, I'm a bit out of the weeds of impl, so I'll focus on the one I'm explicitly tagged in:

1. I have not heard about not removing/modifying the finality check on receivedHeads, that said, our specific estimate for finality will indeed likely change. Thereafter, lmk what you think falls within my purview @frrist, and happy to help/spec. Otherwise, I'll keep butting out of your world not to confuse things by bringing in orthogonal considerations :b

[frrist]

Thanks @sternhenri, this clears up my confusion.

For clarity (and anyone who comes across this thread later), we only perform a finality check on a receivedHead when it comes from an untrustworthy peer. This helps us avoid being DDOSed by malicious peers -- more details on this will be covered in #2674.

[anorth]

With the trusted flag we have now, and the re-organisation to distinguish the initial chain fetch before subscribing to pubsub, we might be nearly done here.

There's still #2674, a distinct issue. We need to do something to ensure the current non-secure bootstrap gets a node close enough to the consensus head before handing over to the untrusted pubsub.

[sternhenri]

re something, my sense is there will be out of band ways to verify whether you're good (ie chain explorers) but heuristically, all you can do is just sample enough "heads" to have good confidence you've got the right one. (pretty subjective :b since it's hard to ascertain network size as is)

[frrist]

To whoever picks this up next there is prior art in https://github.com/filecoin-project/go-filecoin/pull/3366.

[anorth]

Made redundant by new spec.