Cosmos-sdk: Add hooks to allow app modules to add things to state-sync

[michaelfig]

Summary

Please provide a way for app modules (like Agoric's x/swingset) to add additional state to the snapshots generated by the state-sync mechanism for the new clients to download.

@erikgrinaker said on Discord:

Currently the Cosmos SDK automatically dumps IAVL data from the rootmulti store, there aren't any hooks to extend this - but feel free to open a feature request for it, although it'll likely need some design work.

Problem Definition

For the https://agoric.com chain, we maintain some crucial (and large) state outside of the IAVL tree. We'd like our x/swingset module to be able to save external state to snapshots so that when the snapshot is downloaded by clients doing state-sync, they will be able to use our external state to catch up to the external
state of the chain as well.

We use the Cosmos SDK as I/O from our deterministic Javascript world. So, if a new validator is going to skip past (many) blocks while using state-sync, that validator also needs to download enough information to allow it to skip past evaluating all the blocks to update the JS heap.

The feature will accommodate chains that have state outside of the IAVL tree.

There will be greater complexity to include this feature, but it will make state-sync more general.

Proposal

Somehow provide a callback so that app modules can choose to publish additional state to the snapshots. Also, allow the module to decide how to interpret that downloaded state during state-sync. I don't have suggestions for how to accomplish this and am leaning on the Cosmos SDK team to decide the best course of action.

On Discord, @zmanian said:

I suspect the Agoric heap state will be large enough that it should be sent as multiple chunks

and @erikgrinaker replied:

That shouldn't really be a problem, we currently just chunk a byte stream of serialized data items. It shouldn't be too hard to extend this with additional item types, but needs some sort of API to register schemas and (de)serializers and such.
I suppose in the most naïve case the API could just export a binary reader/writer, although that might prevent us from implementing any sort of incremental chunk verification scheme later.

---

For Admin Use

• [ ] Not duplicate issue
• [ ] Appropriate labels applied
• [ ] Appropriate contributors tagged
• [ ] Contributor assigned/self-assigned

[michaelfig]

Can somebody please triage this request? I'd like to discuss what we can do to implement it.

[zmanian]

Whats the status of heap snapshots for Agoric?

Because that would be a dependency.

[michaelfig]

We don't need heap snapshots to take advantage of state sync. Even with heap snapshots, we still need to supply a transcript that is the delta from the snapshot to the state as of the synced block. Without heap snapshots, we can supply transcripts that go back to genesis, which is not as optimised, but still correct.

But without this per-module state sync hook, we can't even send transcripts, which mean we can't do state sync at all.

[zmanian]

My impression is that a transcript if effectively the same as replaying every block?

[michaelfig]

My impression is that a transcript if effectively the same as replaying every block?

Basically, but at the smart contract level, not the Cosmos SDK level. Heap snapshots are essentially optimised transcripts.

My point is that we can test and debug the Cosmos module state sync hook even before we have full application heap snapshot support (which is in the pipeline).

[zmanian]

can you have a begin/ end blocker right now that captures the transcript as data structure for a certain height?

[michaelfig]

can you have a begin/ end blocker right now that captures the transcript as data structure for a certain height?

We've talked about doing this as an experiment. Last time we tried this we had performance problems, but that was a year ago, and we've learned more since then.

We'll definitely try it again to see if it works to put everything in the IAVL tree.

[zmanian]

The state sync systems doesn't require putting everything in the IAVL tree. I think that would be performance disaster.

It just requires storing and retrieving things in a chunkable verifiable form at the snapashot height.

There needs to be some work the SnapshotManager to support this but this is the path I'd recommend.

[michaelfig]

The state sync systems doesn't require putting everything in the IAVL tree. I think that would be performance disaster.

Yeah, that was our first experience. Not only the transcripts, but the entire kernel database need to saved for a given snapshot, which is a lot of stuff.

There needs to be some work the SnapshotManager to support this but this is the path I'd recommend.

Sounds good. Any way we can get this on people's radar for the folks who are actually going to implement it? I don't know anything about SnapshotManager yet, and our use case would be satisfied by something fairly generic, so I hope I'm not the one to do the work. :wink:

[michaelfig]

can you have a begin/ end blocker right now that captures the transcript as data structure for a certain height?

Oh, were you meaning a data structure to send to the SnapshotManager? Yes, we can do that.

[okwme]

Hi all,
The ICF and core SDK contributors are still working through how to prioritize community requests for features within the available developer resources dedicated to the Cosmos SDK.

For this one it would be great to understand if there were other SDK users who are storing chain state outside of IAVL and also want this feature to utilize state sync.

I'll come back to this thread with updates on that process as it should be eventually reflected in the CONTRIBUTING.md section of this repo.

In the mean time I'd encourage you to continue fleshing out the needs from a user perspective as well as seeking out other users who would benefit from this feature to factor into the conversation about prioritization.

[alexanderbez]

To be quite honest, I'm not totally understanding the problem and context here. Is this a pretty niche use-case? I'm having a hard time wrapping my head around how something like this would even work.

[robert-zaremba]

@alexanderbez - the use-case is to enable a non-SDK DB sync within the snapshot sync mechanism.

Apparently, Agoric is using a different module for one of their modules and they would like to benefit a snapshot-sync mechanism.

[robert-zaremba]

The state sync systems doesn't require putting everything in the IAVL tree.

Correct. But currently it only supports the Multistore.

@michaelfig What DB do you use?

This is how I see it:

1. We will need to define an interface for an external DB Reader and Writer.
2. Reader and Writer will need to be implemented on the module side.
3. I would strongly insist that Reader and Writer will use checkpointed state of the DB. Otherwise the whole process will be staging on the module replaying module messages from the genesis. And things can go nasty if some of this messages will interact with other modules and change the multistore state.
4. Once snapshot is completed, the node will use a standard mechanism to replay transactions from the last block in the snapshot till _now_.

[robert-zaremba]

@michaelfig for helping prioritizing this, could you asses your level of support, how _much_ do you need this and by when?

[alexanderbez]

Seems like a very niche use-case. I would like to see an ADR or RFC describing the exact approach taken care. I wary of bringing additional complexities into the SDK's store logic, but if this can be done cleanly it sounds reasonable to me.

[michaelfig]

The state sync systems doesn't require putting everything in the IAVL tree.

That's a relief.

Correct. But currently it only supports the Multistore.

@michaelfig What DB do you use?

We use a set of databases optimised for our particular access patterns, and we'd like the flexibility to be able to change their implementation. The representation is not exposed to Cosmos SDK, and is not necessarily portable to other environments.

1. We will need to define an interface for an external DB Reader and Writer.
2. Reader and Writer will need to be implemented on the module side.
3. I would strongly insist that Reader and Writer will use checkpointed state of the DB.

It may be as simple as adding a hook for the state-sync system to tell our module to checkpoint, then our module can put the portable checkpoint into the multistore.

1. Once snapshot is completed, the node will use a standard mechanism to replay transactions from the last block in the snapshot till _now_.

Also, once the snapshot is restored, we would need another hook to tell our module to reconstruct the optimised databases from the current multistore before the transaction replay begins.

Does that sound more palatable?

[robert-zaremba]

I wouldn't go through multistore. As @erikgrinaker mentioned, a much better design is to register additional readers / writers to the snapshot manager. This would function similarly to storing things into IAVL.

[michaelfig]

@michaelfig for helping prioritizing this, could you asses your level of support, how _much_ do you need this and by when?

As of next week, our testnet will be on the 0.40 branch, but we won't be able to implement state-sync until these hooks exist. So, new validators will be unhappy if they have to sync from scratch, but it won't be any worse than our current testnet. I will need to disable state-sync while that's the case.

The need for state-sync will increase as time goes on if we have any long-lived epochs.