Cosmos-sdk: Support IBC proofs in C

[ethanfrey]

If we want to support IBC to applications written outside of the basecoin frameworks, and especially written in languages other than golang, we need to provide a library for them, ideally in C, as most languages can call into C libraries without much problem.

I documented an issue about light-client verification of headers here: https://github.com/tendermint/light-client/issues/43 This issue focuses on verifying proofs once you have a header.

Note that all development for the next major release is on the unstable branch, and new features should base upon that.

Merkle Proofs

In essence an IBC packet is a key-value pair that is stored in the merkle tree, and which needs a proof that matches the AppHash in a trusted blockchain header. Take a look at the proof code, which basically wraps the KeyExistsProof logic from merkleeyes (also unstable branch).

This all depends on the iavltree data store from the merkleeyes repository. If we want to support other stores with merkle proofs, we need to extend this logic to be more flexible. If anyone has a concrete example that needs support, please add a link to this issue, until then, we just focus on tis data store.

IBC packet structure

All algorithms until this point is pretty solid and unlikely to change much. IBC packets work, but will continue to evolve to support things like returning errors back to the calling chain and being relied over multiple hops (rather than passed directly to the destination chain). So, code here is a good example to build on, but also most likely to evolve before the cosmos 1.0 release.

Take a look at the Tx structure of an IBC Post Packet which contains information to prove a packet.

You also need to support some packets to register a chain and update the headers. You can use the same Register and Update packets, but that can vary between abci apps. What is essential is that these packets are processed according to light-client proofs, and that the IBC Post packets are consistent over all chains and properly verified and processed

TODO

If you wish to implement this, please add any comments to this issue to get a dialogue on how best to proceed. One could also try to wrap a go library into a C ABI which is much less work than porting it all, but less performant.

[mdyring]

Copied from Slack:

mdyring [10:34 AM]
@ethanfrey this phrase got me worried "This all depends on the iavltree data store from the merkleeyes repository." from the basecoin repo. So to do IBC an app needs to use iavltree?

ethanfrey [10:37 AM]
Nope. We just haven't had another example to support yet. I believe Alexis wanted to work on refactoring this in the next few weeks to be a bit more general using interfaces.

We do rely on providing a Merkle proof. Do you have an implementation that you could share. It would be good to have some concrete examples to make sure we support them.

[10:38]
Your app should be able to parse Merkle proofs from any supported db. Each db type with Merkle proofs will need extra code to verify proofs from it. But that should not be so hard.

mdyring [10:43 AM]
@ethanfrey there is some code in the block-finance repo, but that app hash part is not done yet.. but i'd expect IBC to work using the same app hash returned on commit() - regardless of how this is calculated?

srm
[10:43 AM]
I still don't understand merkleproofs: when to use them, why to use them and most importantly how to use them. Is there already some documentation on this whole feature?

mdyring [10:44 AM]
@ethanfrey or are you talking about something entirely different when the "merkle proof" is used? IBC is still entirely new to me so I have a very basic understanding of it

ethanfrey
[10:47 AM]
Can you please add this discussion to the issue as it is important information to capture?

mdyring [10:50 AM]
@ethanfrey i think it would be very healthy to develop the IBC code more fully when there is a 3rd party zone ready to use it.. to avoid any unintended dependencies or assumptions based on the cosmos code base
(i'd be happy to volunteer to be test pilot on that - needs to be done anyway in a not too distant future)

[ethanfrey]

Short answer.

The IBC packet must be written in the data store of the sending abci app (thus making it Provably part of consensus).

Any data store used by an abci app should be able to:

1. Efficiently create an app hash each block that is unique to the current data store state.

2. Provide efficient proofs of any key value in the storage, tying it to the app hash stored in the header. This is done with Merkle proofs.

Two implementations of data stores that provide Merkle proofs are merkleeyes/iavl/Iavltree from tendermint and the patricia trie used in by ethereum. Please add links to any other open source data store that provides Merkle proofs.

https://en.m.wikipedia.org/wiki/Merkle_tree

[zmanian]

I'd advocate strongly for implementing IBC packets and proof verification in Rust as a static/dynamic library with a C api. This could then be consumed by C and C++ applications and via FFI by everything else.

The advantages of Rust's type system in the correctness of this process would be considerable in addition to memory safety in something that must parse untrusted data.

my understanding of step 1 is implementing tendermint wire format in rust.

[adrianbrink]

@zmanian I have implemented the wire format in rust. I'll extract it into its own crate.

[mdyring]

Requiring a new dependency (static or dynamic lib) to parse IBC is a step in the wrong direction IMHO.

Some people will have a preference to build from source - so having to setup a Rust build environment just to build this bit of the puzzle seems overly complicated for what should be a fairly simple job.

Though I am sure Rust has many merits for this use case, any Tendermint app must have a robust parser anyway, since the TX payloads themselves are untrusted. If it can't parse untrusted data in a safe manner, it is hosed anyway.. So why treat IBC differently?

For the non-Go crowd, there is already a requirement to build or link Protobuf and potentially gRPC if desired. I believe Protobuf is still a good fit for IBC, since people already need to pull it in if they are using ABCI over socket.

But if not Protobuf, then at least the wire format should be documented and simple enough to allow people to roll their own parsers, if they so desire.

I have no strong feels for/against Protobuf, but I think it is beneficial to Tendermint and Cosmos adoption to keep the barrier to entry as low as possible (i.e. number of dependencies to a minimum).

[ethanfrey]

I agree. We have go as reference. No need to add another canonical implementation, people should be able to roll their own in any language.

We do need to document the dependencies, so it is not hard to implement. Right now they are compatible crypto library (ed25519) and the binary encoding (go-wire). The second part is much more tied to go and while I find it great for use in go, I don't see it as super cross-language yet.

I think we need to put work into making go-wire truly portable (ported) to many languages and documented. Or we should use another format. I prefer the first - but there is a non-trivial amount of work to expose go-wire to the greater world.

[ethanfrey]

Oh, and the merkle data store. Can we collect a list of potential stores in various languages? Any need to export one to any other language?

[cwgoes]

Replaced by https://github.com/cosmos/cosmos-sdk/issues/843 by IBC specification meeting.