Bigchaindb: [HTTP API] Can queries that return partial transaction payloads be trusted?

[sbellem]

There are currently many open issues, that wish to provide queries via the HTTP API on certain aspects of transactions, such as assets, and owners (public keys). Such issues propose to return partial chunks of a transaction payload.

How can an API consumer trust such responses?

Transaction payloads that use the ed25519 signature scheme can be verified to be authentic, provided one has the entire payload. It is important to note that the message used to form the signature is the entire transaction payload, except for the fulfillment uri(s). Thus, in order to verify the authenticity of a transaction payload one needs the entire transaction.

Why does this matter?

It seems that the structure and content of the transaction payload has been and is being influenced by the idea of querying it. But of what value is a response coming from a BigchainDB federation that cannot be verified to be authentic?

If we think of applications that may depend on the data they read, and make critical decisions based on that data. If the data cannot be trusted, we put these applications at risk.

Moreover one could as well dump all of BigchainDB data in whatever data store one prefers, and build the querying layer on top of it. At least they could be controlling the machines on which the querying is happening against, and also have the freedom to choose the querying/searching engine they are most comfortable with.

Basically: should BigchainDB provide querying beyond transactions. blocks and votes at all? These three main components have the possibility of being verified, whereas partial constructs do not.

Also, should we sacrifice the simplicity of the transaction payload for the sake of being able to query chunks of it that cannot be trusted?

What is BigchainDB's main purpose?

Could querying be offloaded to add-on layers? Different applications may need different types of queries, for which different data stores may be better than others. So why limit users to a particular type of querying? More concretely, some applications may need to establish interconnections between different transactions, or characteristics, in such a way that a graph database will be best suited. Others may need to perform information retrieval tasks, such than a search engine like elasticsearch is best suited.

We also need to keep in mind that by offering a metadata field, the possibilities are more or less infinite.

Proposal

1. Keep querying capabilities of BigchainDB simple, such that responses preserve the level of authenticity that can be provided via cryptoconditions.
2. Identify the type of querying that may potentially be relevant: graph, text search, relational, etc.
3. Think of ways of how data could be piped into different data stores dedicated to specific types of querying.
4. Ask whether we really need to implement one or more of the ways, defined in 3, above.

If for whatever reason, the above 4 points are rejected. Then the question that remains, would be "How can we make querying trusted in the context of a decentralized architecture?"

We have to keep in mind that many organizations make critical decisions based on data. Hence reading data, is one thing, trusting it is another.

What is the value of data that cannot be trusted?

[TimDaub]

Good point, thanks for bringing this up.

I think the problem your outlining can be split into multiple sub problems. I'll try to answer some of them. Could be that I'm forgetting some though.

1. How can I validate that was I requested is what I get

For transactions and blocks, the answer is easy:

You take the id of the object, hash the message again and compare the two values. If they're equal you've retrieved the right thing.

OK, but what if we just return partial payloads as was proposed in #830?

Obviously there it becomes tricky. When making a request like:

/transactions?fields=id,conditions&fulfilled=false&owners_after={owners_after}

you cannot validate against the provided transaction id anymore.
What you can validate though is the validity of the condition, as it's uri is provided.
So how to resolve this?

One approach I could imagine, is to hash the transaction as a merkle tree and then follow the branch/chain that was returned upwards. I'm not an expert on this, but here is how this could potentially look like:

In fact, IPLD does exactly this kind of thing with JSON, which helps solve this use case quite well I believe.

2. How can I validate that something should be there?

Now, providing some hash/id and retrieving something that is known already seem to be rather straight forward. What about retrieving something that the user doesn't know already?

Lets compare the two use cases to highlight the differences once again:

1. Retrieve object by id/hash and validate if return value is according to id/hash
2. Retrieve or rather search for things that are unknown.

Examples of this could be something like:

• Give me all conditions that correspond to a certain public key
• Give me all assets that match a certain uuid (which is not a hash, meaning I cannot verify against it).

For this, I currently have no ideas. It's probably a whole different ticket/concern.

[sbellem]

Thanks @TimDaub !

Regarding the first part, the Hash Tree approach: good idea! But don't you need more than what you query in order to verify the authenticity of what you retrieved?

Considering the graph you gave above. If one retrieves conditions. In order to verify the authenticity one would still need other data, namely:

• hash(fulfillments), with which one can calculate the hash(hash(condition) + hash(hash(fulfillments)
• hash(hash(asset) + hash(metadata))
• txid

It's true that one needs less, for instance one does not need to retrieve the asset and metadata, nor their individual hashes, but one still needs the right hand side node leading up to the root of the tree.

follow the branch that was returned upwards

So yes, but one needs each sibling of each node on the path of that branch.

Hash trees make membership verification more efficient, than a list, but we still would need to return more than what was queried, it seems.

Some useful references:

• http://bitcoin.stackexchange.com/a/38399/36231
• https://www.quora.com/Cryptography-How-does-a-Merkle-proof-actually-work

[TimDaub]

It's true that one needs less, for instance one does not need to retrieve the asset and metadata, nor their individual hashes, but one still needs the right hand side node leading up to the root of the tree.

Exactly.
We would have to send this information about the merkle tree with the JSON.
Hence in #830, I decided to remove ?fields until we implement a merkle-tree inspired approach.

[TimDaub]

@sbellem Would you be OK with that we create a ticket outlining how a concept like?fields (partial payloads essentially) should be implemented on the HTTP API using merkle tree and close this ticket as its formulated more as a question and not as an actual task?

I can create the new ticket if you want.

[sbellem]

@TimDaub You can create the ticket for the fields approach, and I can create the other tickets because this issues raised more than one question I think, so I will create tickets accordingly.

[vrde]

Why adding complexity to the system if the client can just:

1. retrieve the transaction.
2. ignore everything and get just the payload.

What's the use case?

[sbellem]

Why adding complexity to the system if the client can just:

retrieve the transaction.
ignore everything and get just the payload.

What's the use case?

Yes, transactions are atomic objects in that sense, and the server returns only these atomic objects. Consumers interested in parts are responsible to pick the transaction apart for what they need.