Zebra: Tracking: block validation

[teor2345]

Create a block validation service, which validates blocks and transactions. This service should perform semantic validation, before submitting blocks to the state for contextual validation.

Used by the block fetching and validation service in #140.

• [x] Parallel verification design #763
• [x] Create a high-level wrapper service dispatching to either checkpoint or full validation #691
• [x] Create a checkpoint verifier #429
• [ ] Validate block headers #477
• [ ] Validate transactions #196
• [ ] Re-enable any tests that have been disabled
• [ ] Test validation on the full chain #745

[hdevalence]

Should we try to minimise parsing before validation?

So far we've been trying to do parsing as validation, by making the internal representation of the chain data structures enforce all of the structural constraints (for instance, a version 1 transaction cannot have snarks, a version 4 transaction has a binding signature iff it has spend or output descriptions, etc.). This means that any data that comes in over the wire is already immediately checked at the edge of the application for structural validity.

What types should we use for unvalidated data?

I'm not sure about this, but here are some rough ideas:

Looking at the validation levels we defined (raw bytes / structural / semantic / contextual), we want to figure out whether we should have different types corresponding to the different levels. We already check structural validity as noted above, so we don't have any representation of raw bytes. So the remaining things to consider are different types for different levels of validity.

I'm not sure if there's a clear reason to maintain information about the semantic validity of some data except as an intermediate state on the way to contextual validity. For instance, if we have some future that checks all the signatures and proofs and then waits on a future representing the contextual validity check, we might have encapsulated the intermediate state in such a way that we don't actually need to pass around information about semantic validity, and therefore might not need a separate type for it.

Strictly speaking, at the level of contextual validity, some transaction or block is never valid-in-itself but valid-in-context, the context being some particular chain state. So if we were to have a distinct type representing contextually valid data, the type would have to be parameterized somehow by a particular chain state, and you'd need to have some kind of subtyping relationship that expresses how relationship between chain states translates into a relationship between validities (e.g., if a tx is valid in the context of state A, it's also valid for A plus extra blocks). This all seems somewhat complicated to me, although I might be needlessly overcomplicating things here.

The alternative would be to not attach the context at the type level, and instead use the zebra-state layer to handle it: to get validated data, you query zebra-state, and because only validated data goes in there, you know the answer you get is OK.

[teor2345]

We spoke about two possible designs for the block contextual validation service:

1. a future that validates then commits the block to the state
2. a future that validates the block, and then returns a new type ContextuallyValid

Option 1 seems simpler, because the validation service already needs to know about the state for contextual validation.

We need to make these validity constraints part of the validation and state contracts:

• the chain state and block validation service return contextually valid blocks and transactions
• the mempool state and transaction validation service return semantically valid transactions

Option 2 might be useful for transaction semantic validation, so that we can distinguish between contextually valid transactions in blocks, and semantically valid transactions in the mempool. But we might not need a new type to enforce these contracts. The API design could be enough: module names, service names, and API documentation.

[teor2345]

(From #196.)

I think there are a few different levels of context:

• semantic verification - no context
• network verification - only needs to know the network
  
  
  
  • for example: checking the embedded network type in the address
  
  
• transaction set verification - needs to know the other transactions in the block or state
  
  
  
  • for example: double-spend checks
  
  
• full contextual verification - needs to know the whole chain state
  
  
  
  • for example, full block validation
  
  

[daira]

Is it necessary to distinguish between semantic and network validity? You always know the network when you're semantically validating, I think.

[teor2345]

It's useful to know which checks are conditional on the network. But I'm not sure if we'll put those services in different modules. Passing a Network argument may be enough.

[hdevalence]

It might be preferable to pass a Network argument to the service constructor and maintain it as part of the verification context, rather than passing it to Service::call, which would require changing the Request type to be a tuple.

[teor2345]

Option 2 might be useful for transaction semantic validation, so that we can distinguish between contextually valid transactions in blocks, and semantically valid transactions in the mempool. But we might not need a new type to enforce these contracts. The API design could be enough: module names, service names, and API documentation.

I think a good design here has two separate state objects:

• the chain state, as implemented in ZebraState
• the mempool state, let's call it ZebraMempool
  I've documented this design as part of #448.