Zebra: Tracking: sync correctness

[hdevalence]

Tracking issue for issues around sync correctness, getting confused downloading blocks, etc.

First Alpha

Hangs and Panics:

• [x] Fix network panic (#1183)
• [x] Fix state panic (#1182)
• [x] Diagnose and fix mainnet sync hangs (#1181)
• [x] Stop deadlocks between the sync service and checkpoint verifier by adding a timeout on block tasks
• [x] Stop hangs in ObtainTips and ExtendTips by adding a timeout to tip requests

Logging:

• [x] Fix or hide Failed to write to hedge histogram: ValueOutOfRangeResizeDisabled logs (issue: tower-rs/tower#475, fix: tower-rs/tower#484)

Performance:

• Reduce Zebra's memory usage
  
  
  
  • [ ] tuning RocksDB (#1486)
  
  
  • [ ] limiting queue lengths (#1389)
  
  

Duplicate Downloads:

• [x] Fix Duplicate/AlreadyVerified errors in the checkpoint verifier (#1259)
  
  
  
  • [x] Reduce the AlreadyVerified log level to trace
  
  
• [x] Cancel download and verify tasks if the sync service has dropped the channel (#1037)
• [x] Deduplicate download and verify tasks, by using a separate service to handle those tasks to completion - #1041, see also https://github.com/ZcashFoundation/zebra/pull/980#issuecomment-689279394
• [x] Make tip deduplication order-independent

Cleanup:

• [x] Remove redundant ObtainTips retry #1033
• [x] Remove the out of order block debug logs
  
  
  
  • Depends on out of order block handling in the state (#1011) and inbound network service (#889, inv messages)
  
  
  • For details, see https://github.com/ZcashFoundation/zebra/pull/1012#discussion_r485169112
  
  

Reduce Sync Restarts:

• [x] Hedge slow downloads by starting a duplicate request after a delay #1089
• [x] Skip the final hash in each set of BlockHashes, due to a zebrad bug
• [x] Increase default EWMA RTT time, so peers have to prove they are fast (this makes a huge difference) #993
• [x] Ignore errors for blocks which are already in the state
• [x] Tweak and document sync timeouts #993

Work out if we need to remove duplicate downloads using:

• [x] the list of hashes from earlier queries in the current ObtainTips or ExtendTips (download_set) [Yes, as done in the current code]
• [x] the list of hashes that are being downloaded and verified in spawned futures [No, because we deduplicate in each extension and we check that each extension is an extension]
• [x] the list of hashes that have failed download or verify recently [No, because failures cause us to restart with clean sync state]
• [x] the hashes of verified blocks in the state [Only in obtain_tips, as done in the current code].

Design Questions:

• [x] Work out how to get each new block as it is mined, when we have synced to the tip
  
  
  
  • In a79ce97 and #887, we skip responses that only extend by 1 block:
  
  
  • ObtainTips: https://github.com/ZcashFoundation/zebra/blob/main/zebrad/src/commands/start/sync.rs#L219
  
  
  • ExtendTips: https://github.com/ZcashFoundation/zebra/blob/main/zebrad/src/commands/start/sync.rs#L296
  
  
  • See #889, where we get new blocks through the gossip protocol.
  
  
• [x] Decide if the BLOCK_TIMEOUT is too low for slow or high-latency networks
  
  
  
  • [x] Work out if we can run Zebra over Tor [No, it's bad for Tor, and slow for Zebra]
  
  

Database:

• [x] Implement the sled tree changes in the design RFC #902, section Notes on Sled Trees

Future Releases

Documentation:

• [ ] Document minimum network requirements for Zebra
  
  
  
  • [ ] a minimum bandwidth of 10 Mbps
  
  
  • [ ] a maximum latency of 1 second
  
  
  • [ ] we don't support running Zebra over Tor
  
  
• [ ] Document the peerset target size config, as a useful option for bandwidth-constrained nodes
  
  
  
  • [ ] test the minimum supported peerset target size, and add it to the docs (it's probably 4-12)
  
  
• [ ] Check Zebra's memory usage during sync, and document the minimum requirement
• [ ] Update the sync RFC to document the new sync algorithm #899
  
  
  
  • See previous changes in a79ce97 and #887
  
  

Performance improvements:

• [ ] Work out how to improve genesis and post-restart sync latency, particularly on Testnet
• [ ] Deploy extra zcashd instances on Testnet (#1222)

Possible Future Work:

• [ ] Consider disconnecting from peers that return bad responses
• [ ] Consider disconnecting from peers that are sending blocks that we no longer want
• [ ] Churn peers, using a regular timer to disconnect a random peer
• [ ] Analyse the bandwidth and latency of current Zcash mainnet and testnet peers
• [ ] Create a peer reputation service

• [ ] Refactor out the core sync algorithms, and write unit tests for them #730

[hdevalence]

@teor2345 I think we don't need the sync to be able to go forward a block at a time, since as new blocks are generated we'll get them through gossip (to be implemented in #889).

[hdevalence]

On the BLOCK_TIMEOUT question, there are actually two timeouts -- the one applied in the timeout layer and the one used internally in the peer set's state machine (set to 10 seconds). So if we want to have timeouts greater than 10 seconds we'll need to update that timeout as well. With a 1-second latency allowance that would be 1.8 Mbps. Maybe that's too high, although I don't think that it's unreasonable for the server use case that Zebra aims for.

I think it would be fine to increase the timeout to a somewhat larger value, and increase the lookahead limit to compensate. What's a reasonable target speed?

[teor2345]

@teor2345 I think we don't need the sync to be able to go forward a block at a time, since as new blocks are generated we'll get them through gossip (to be implemented in #889).

Sounds good, and I guess it's ok to lag slightly in our first release. I'll update that TODO.

[teor2345]

On the BLOCK_TIMEOUT question, there are actually two timeouts -- the one applied in the timeout layer and the one used internally in the peer set's state machine (set to 10 seconds). So if we want to have timeouts greater than 10 seconds we'll need to update that timeout as well. With a 1-second latency allowance that would be 1.8 Mbps. Maybe that's too high, although I don't think that it's unreasonable for the server use case that Zebra aims for.

I think it would be fine to increase the timeout to a somewhat larger value, and increase the lookahead limit to compensate. What's a reasonable target speed?

I'm not sure if we need to make any changes here.

Requiring less than 10 Mbps is reasonable for servers. And a latency of 1 second should cover most servers, wifi, and even some mobile internet. But I don't know the exact answer, because download speed also depends on peer speeds and peer latency. We should do some testing 🤔

Do we expect people to be able to run Zebra over Tor?
Because that will decrease our bandwidth, and significantly increase our latency:

For Tor via exits, it seems like 6 seconds should be enough for most blocks:
https://metrics.torproject.org/torperf.html?start=2020-05-15&end=2020-08-13&server=public&filesize=1mb

For Tor via onion services, we'd need to allow 20 seconds per block:
https://metrics.torproject.org/torperf.html?start=2020-05-15&end=2020-08-13&server=onion&filesize=1mb

(Note that these are the 1 MB graphs, there are also 5 MB graphs, but we only need 2 MB.)

[hdevalence]

Sounds good, and I guess it's ok to lag slightly in our first release. I'll update that TODO.

To clarify, I don't think we'll be lagging here -- in initial block sync, we'll sync up to the current chain tip, and we'll stay in sync through block gossip.

[hdevalence]

I'm not sure if we need to make any changes here.

Okay, let's do nothing for now. I don't think that we can run Zebra over Tor anyways, since the Bitcoin protocol isn't really compatible with Tor.

[hdevalence]

Updated the list of checks with rationale about sync behavior.

[teor2345]

I updated the list of outstanding issues, I'll check them off as I submit PRs.

Even after these changes, Zebra still doesn't limit the number of requests which are waiting for peer responses, or handle peer reputation. So I've added them to the "future work" section.

[hdevalence]

I don't think it makes sense for us to limit the number of requests waiting for peer responses, because I don't think that there's a way for us to work out in a principled way what an appropriate limit for the number of in-flight requests should be. Instead I think it would be better to use backpressure from the peer set to block new requests when the peer set is at capacity, which is what we already do. If this isn't working well, I would prefer to try to figure out why, and how we could improve it. Perhaps this could be addressed by setting the peer set's target size to a smaller parameter.

[teor2345]

Yeah, that was basically the conclusion I came to after thinking about it a bit more.

It might also be something we want to document for users - if they are on a slower network, try setting the target peerset size lower.

[teor2345]

I have split this tracking issue into "First Alpha" and "Future Releases".

[mpguerra]

What is left here for the Alpha release now?

I have removed #1183 from the Alpha milestone and #1182 is done.

Do we also want to implement the logging tickets?
How much of the testing (if any) has been done?

[teor2345]

These bugs make testing really difficult:

• network panic #1183 - this panic still happens on the latest main branch
• tower-rs/tower#475 - the large number of redundant error logs make it difficult to see the Zebra log messages that are actually important

I still can't get Zebra to reliably sync to the tip. So we haven't done enough testing yet - we need to do a full sync after we've fixed all the common errors.

[teor2345]

@hdevalence said on discord that tower-rs/tower#475 is fixed by tower-rs/tower#484.

For some reason the ticket is still open.

[mpguerra]

Can we move this out of Alpha Release milestone now that #1183 is closed?

[teor2345]

Can we move this out of Alpha Release milestone now that #1183 is closed?

What should we do about these first alpha testing tasks?

• Manually run Zebra on Mainnet to verify that the sync works after recent fixes
  
  
  
  • Optional: also run Zebra on Testnet
  
  

We're still making changes that might affect sync behaviour.

[mpguerra]

These sound to me more like ongoing checks that should be done by developers to validate their changes whenever they make changes to the sync code (or changes that may affect it) rather than finite one-time only tasks.

If these checks can't be automated my suggestion would be to add these to the PR template as a reminder for developers. Otherwise, if we can automate, let's create a ticket to implement these checks in our CI after the alpha.

[oxarbitrage]

I think it will be good to document somewhere(maybe in the main README) machine specifications to sync the mainnet up to current tip. This is recommended RAM, disk space, CPU, etc. With those specs we can document an estimated time to be in sync, something like "at the moment of writing, with the recommended specs, sync up to block XXXXX will take around XX time" This way developers and users can rent appropriate services or try locally and know what to expect.

[mpguerra]

I think it will be good to document somewhere(maybe in the main README) machine specifications to sync the mainnet up to current tip. This is recommended RAM, disk space, CPU, etc. With those specs we can document an estimated time to be in sync, something like "at the moment of writing, with the recommended specs, sync up to block XXXXX will take around XX time" This way developers and users can rent appropriate services or try locally and know what to expect.

We have some similar details in #1374

[oxarbitrage]

I see, sorry i didn't saw that. Thanks.

[teor2345]

There isn't anything specific to be done for "testing on mainnet", so I removed those items, and moved this ticket out of the first alpha release.

[mpguerra]

adding #862 to this epic

[teor2345]

adding #862 to this epic

That ticket modifies the state service interface, which impacts sync and verification. So it doesn't really belong in sync correctness.

The state service cleanup in #1302 might be a better place for it.