Zebra: Flesh out 'start' command to actually start syncing.

[dconnolly]

• [x] Start a sync component / service that is in charge of managing peer requests for blocks and firing validation / management. - #140
• [x] When a storage interface actually exists, send internal requests to save down blocks to the block cache and derived data to the state storage. - #136, #414

[hdevalence]

We might want to flesh out the design details first -- should we add the design label and figure out a) what logic will be required and b) where it should live in the DESIGN.md organization?

[dconnolly]

Some of this exists in the tests stub, also the seed command is already doing a lot of stuff. For now, blocked on having chain state management (aka 'storage').

[hdevalence]

After #414 is merged, we can start with a rough pass of this. The start command already exists, so we can start fleshing out its contents.

As for what it should do, currently the connect stub tries to download the block chain by setting the genesis block as the chain tip, uses a Request::FindBlocks to ask the network for following block hashes, attempts to download those blocks, and optimistically updates the chain tip without validation.

To do this with the state layer, I guess we should:

• [x] - have a way to query blocks by height rather than just by hash
• [x] - have a way to query the state for the current chain tip
• [x] - download blocks and feed them into the state system

We should also consider how we want to have feedback between the component that's fetching block hashes and the component that's fetching the blocks themselves. In the stub code, the hashes are fetched mostly without blocking on the block downloads, which is great for throughput. The resolution to this will eventually depend on the details of the verification architecture.

[yaahc]

have a way to query the state for the current chain tip

@hdevalence can I get more elaboration on what this should entail and how this should be structured? I've started working on this in #422 and I have it so we can query blocks by height and it should be relatively easy to query the block with the largest height but I'm not sure what other tip state might be useful

download blocks and feed them into the state system

Do we want to set this up similar to zebra_network where the init fn spawns some background tasks to populate the state service?

[hdevalence]

have a way to query the state for the current chain tip

@hdevalence can I get more elaboration on what this should entail and how this should be structured? I've started working on this in #422 and I have it so we can query blocks by height and it should be relatively easy to query the block with the largest height but I'm not sure what other tip state might be useful

I think for a start it's enough just to be able to query for the current chain tip's block hash.

download blocks and feed them into the state system

Do we want to set this up similar to zebra_network where the init fn spawns some background tasks to populate the state service?

No, I think we'll want to do that outside of the zebra_state crate, since otherwise we'd need to have zebra_state depend on zebra_network... more generally, I think we want to keep the components driving the state changes separate from the state layer itself.

[yaahc]

I think for a start it's enough just to be able to query for the current chain tip's block hash.

Okay, I added a query for the tip to #422, take a look and lmk what you think about it. I haven't added any tests but I'm assuming that's the next step for that PR, so that's what I'm going to start working on next.

No, I think we'll want to do that outside of the zebra_state crate, since otherwise we'd need to have zebra_state depend on zebra_network... more generally, I think we want to keep the components driving the state changes separate from the state layer itself.

Makes sense, I'll make that a follow up PR then once #422 is done.

[yaahc]

So it looks like the next step on this will be to create the sync component as mentioned here https://github.com/ZcashFoundation/zebra/issues/140. I'm going to start with taking the block downloading logic in the connect stub and break that out into an abscissa component.

I don't really know what to do about the validation bits, I'm guessing that those aren't implemented currently so my plan will be to just continue to shove downloaded blocks straight into the state service the same way connect does. Also, should I be working primarily on the start command or should I continue to just prototype everything in the connect command? I'm a little unclear on what the difference between them is in practice.

[hdevalence]

I don't really know what to do about the validation bits, I'm guessing that those aren't implemented currently so my plan will be to just continue to shove downloaded blocks straight into the state service the same way connect does.

One possibility would be to make a stub service using service_fn or something that does no-op validation, so that there's already a validation Service in play and later it can be replaced with the checkpointer @teor2345 is working on.

Also, should I be working primarily on the start command or should I continue to just prototype everything in the connect command? I'm a little unclear on what the difference between them is in practice.

The connect command is going to be deleted, it's just a scratchpad basically that you can use to do stuff to zcashd and see what happens. The start command should be the thing that will have the real code in it.

Another thing to consider is whether we should hold on to different pieces of the node using Abscissa components, e.g., a component for the network service, for the state service, etc.

[teor2345]

I don't really know what to do about the validation bits, I'm guessing that those aren't implemented currently so my plan will be to just continue to shove downloaded blocks straight into the state service the same way connect does.

One possibility would be to make a stub service using service_fn or something that does no-op validation, so that there's already a validation Service in play and later it can be replaced with the checkpointer @teor2345 is working on.

I've started working on a validation stub in #448. At the moment, I just have the message types. But I could implement the service so it just adds the blocks to the state?

I'll also need a separate state for the transaction mempool, but that's out of scope right now.

[yaahc]

But I could implement the service so it just adds the blocks to the state?

This seems like a good idea to me, I think we're going to need this stub service right away.

[teor2345]

I'll try to get a stub written today.

[yaahc]

Okay so my next step on this is going to be to take the code in connect as a starting point for the new start subcommand, and then modify it to use the zebra-consensus stub instead of pushing directly into the state service.

Once I've got that setup I need to pick a next step. I'm thinking I could either work on modifying the in_memory state backend to handle multiple blocks of the same height or I could look into pulling down sled or that other DB backend that eliza linked recently.

@ZcashFoundation/zebra-team lmk which you think I should do first

[hdevalence]

Either seems fine to me. My sense is that getting some kind of persistence is probably better to do first, since it "unlocks" more future work that we can do in parallel.

[yaahc]

Okay, I'll focus on specing out a path to persistence, either via serde and in_memory or via an embedded DB library.

[yaahc]

@ZcashFoundation/zebra-team okay so for my next step I think I need to work on either filling out the start subcommand or Adding support for multiple blocks of the same height. LMK which you think I should prioritize.

[dconnolly]

@ZcashFoundation/zebra-team okay so for my next step I think I need to work on either filling out the start subcommand or Adding support for multiple blocks of the same height. LMK which you think I should prioritize.

Since I'm fiddling with zebrad node commands to make our fleet deployments ✨ shine✨, filling out start would be immediately useful there. ❤️ 🦓

[yaahc]

Okay, that shall be my goal then.

[teor2345]

@ZcashFoundation/zebra-team okay so for my next step I think I need to work on either filling out the start subcommand or Adding support for multiple blocks of the same height. LMK which you think I should prioritize.

Since I'm fiddling with zebrad node commands to make our fleet deployments ✨ shine✨, filling out start would be immediately useful there. ❤️ 🦓

Sounds good!

I don't think we can make more state changes yet. We'll need to track recent side-chains, so I'll have to think a bit more about the chain reorganisation design (#452). Once I've done enough validation to make a chain (the first part of #477), I'll move on to chain reorganisation and checkpointing (#429).

Hopefully we don't run into too many issues with the state before then :-)

[yaahc]

Hopefully we don't run into too many issues with the state before then :-)

:crossed_fingers: lmk if you do of course

[teor2345]

Hopefully we don't run into too many issues with the state before then :-)

🤞 lmk if you do of course

Chain splits happen for about 0.3% of blocks, so the start command might also run into some state failures. Depending on how many blocks you load, and how you're loading them.

[yaahc]

This doesn't surprise me, the connect subcommand which is basically an incomplete version of start already consistently panics if its run long enough because it always encounters a fork in the chain. Representing the forks is definitely my next task.

My main question is how? I feel like this is probably block chain 101, like theres gotta be some basic block chain data structure that I just need to look up to figure it out. My best bet so far is one of the following approaches:

• Have two trees, one which maps from hash -> block, and one which maps from height -> [hash]
• Have One tree and n chains, the tree maps from hash to block and each chain represents a linked list of hash -> hash

[teor2345]

This doesn't surprise me, the connect subcommand which is basically an incomplete version of start already consistently panics if its run long enough because it always encounters a fork in the chain.

If we need a quick fix, we can just ignore duplicate blocks at the same height. There's no chain validation yet, so we won't notice any breaks in the chain.

[teor2345]

My main question is how? I feel like this is probably block chain 101, like theres gotta be some basic block chain data structure that I just need to look up to figure it out. My best bet so far is one of the following approaches:

* Have two trees, one which maps from hash -> block, and one which maps from height -> [hash]

* Have One tree and n chains, the tree maps from hash to block and each chain represents a linked list of hash -> hash

Conceptually, the chain is a tree data structure, and each fork represents a branch. The best fork is the one with the largest proof of work.

We'll definitely need a hash -> block mapping, because that's part of the protocol.

I added a draft API to the top of the chain reorg ticket #452.

[dconnolly]

My main question is how? I feel like this is probably block chain 101, like theres gotta be some basic block chain data structure that I just need to look up to figure it out. My best bet so far is one of the following approaches:

* Have two trees, one which maps from hash -> block, and one which maps from height -> [hash]

* Have One tree and n chains, the tree maps from hash to block and each chain represents a linked list of hash -> hash

Conceptually, the chain is a tree data structure, and each fork represents a branch. The best fork is the one with the largest proof of work.

The best fork is literally the longest one, aka, the most trusted/most validated, and thus the largest amount of proof of work of all the branches. 👍

// @acityinohio @hdevalence

[hdevalence]

I wrote up notes from our discussion on pipelineable block discovery: https://github.com/ZcashFoundation/zebra/issues/504