Bigchaindb: API: expose bigchaindb functionality

[diminator]

This issue is to list endpoints that should be exposed by the API.

Useful documents for this discussion, to add context:

• https://github.com/bigchaindb/bigchaindb-examples/blob/master/server/lib/api/views.py
• https://github.com/diminator/rfcs/blob/36e30bd746281a539457eab8b1ba8f2e30e34ae4/0012-ledger-web-api/0012-ledger-web-api.md

Main categories:

bigchaindb metadata

• [ ] get_info or get_metadata: info on ledger instance, asset types, federation, etc (to be fleshed out)

  transactions

• [x] get_transaction: #554

• [x] write_transaction: #554
• [x] get_status: #555
• [ ] get_tx_by_metadata_id
• [x] get_owned_ids#624

• [x] escrow => move to driver/templates cc @TimDaub

  asset: #599

• [ ] get_txs_by_asset_id (all transactions related to the asset)

• [ ] get_asset (maybe. get the CREATE transaction for the asset)

  blocks

• [ ] get_block #624

  votes

• [ ] get_votes #624

• ...

[r-marques]

The most important for clients would be:

• transactions
• metadata
• asset #599

Less important to clients (but useful if they want to validate themselves)

• blocks
• votes

[r-marques]

Some methods that could go into the api:

• Transactions:
  
  
  
  • get_transaction
  
  
  • get_owned_ids
  
  
• Metadata:
  
  
  
  • get_tx_by_metadata_id
  
  
• Asset:
  
  
  
  • get_txs_by_asset_id (all transactions related to the asset)
  
  
  • get_asset (maybe. get the CREATE transaction for the asset)
  
  
  • We could have some more digital asset queries to better support divisible assets. The tradeshare example should give us some ideas
  
  

[diminator]

I would group get_tx_by_metadata_id as part of transactions as it is a query on the transaction and not on the state of the DB

[sohkai]

Looks like we'd also like a websocket API for interledger, which is probably worth its own project.

get_asset (maybe. get the CREATE transaction for the asset)

To me, it makes more sense if get_asset returned just the asset payload rather than the entire transaction. If you want the create transaction, you can use get_txs_by_asset_id (maybe /assets/<ID>/transactions?) or even a convenience get_creation_tx_of_asset (maybe /assets/<ID>/transactions/creation or /assets/<ID>/creation_transaction).

Less important to clients (but useful if they want to validate themselves)
- blocks
- votes

I wouldn't spend the effort to implement these until we have a clear use case.

[diminator]

get_transaction, get_owned_ids:

include backlog or not?

yes, have a single call to retrieve all transactions independent from their status.
status needs to be included in the response:
[{"tx": tx, "status":status}]

[diminator]

write_transaction:

tx could get dropped by the assignee, no way to figure out what was the cause of the problem unless it is in the changefeed or kept (=DDOS vulnerability)?

[diminator]

get_tx_by_metadata_id or get_tx_by_payload_id

defer (any querying) on the asset payload until digital assets are defined

[diminator]

get_owned_ids:

• what if we want to return not the id but input.condition or something else?

e.g
/transactions/unspents/<owner_id>/conditions

[diminator]

escrow and fulfill_escrow or abort_escrow

move to client/driver transaction templating

[diminator]

get_block or get_votes:

maybe combine them?

[r-marques]

get_owned_ids actually returns {"txid": .., "cid", ..}
Thats all you need right now to specify the inputs of a transaction, I guess you can always query the transaction if you want more information

Maybe get_block should also return the status of the block? The votes are more consensus related, not sure it is helpful to return them unless you want to do some validation client side but a endpoint for it would be nice /block/<block_id>/votes

[diminator]

Maybe get_block should also return the status of the block? The votes are more consensus related, not sure it is helpful to return them unless you want to do some validation client side but a endpoint for it would be nice /block//votes

:+1:

[diminator]

get_owned_ids actually returns {"txid": .., "cid", ..}
Thats all you need right now to specify the inputs of a transaction, I guess you can always query the transaction if you want more information

Would it make sense to include all transactions where you are partial owner or only the ones where you are a full owner?

[r-marques]

I think you should return all transactions where you are partial owner. Even if you are a partial owner in a threshold transaction if may require only one signature to go through

[diminator]

I think you should return all transactions where you are partial owner. Even if you are a partial owner in a threshold transaction if may require only one signature to go through

Ok, that will only work if we can cross-correlate the owners_after list with the conditions

[r-marques]

It was assumed that they always match and we do have a KeyMismatchException when they don't match

[diminator]

We'll have to discuss this, probably in another topic

[TimDaub]

Here is a summary of what I could gather from this thread:

First, lets cover the easy parts:

Transactions

• get_transaction: HTTP GET /transactions/<id>
• write_transaction: HTTP POST /transactions
• get_status: HTTP GET /transactions/<id>/status

(These endpoints are already in place, I think we can leave them as they are).

Blocks

• get_block: HTTP GET /blocks/<id>
• write_block: Not exposed

Questions:

• Do need a block listing endpoint? (My answer: Only when we have a user case)

Votes

TBD

Assets

get_asset: HTTP GET /assets/<id>

Returns the content of an asset from the respective CREATE transaction.

write_asset: Not exposed, use HTTP POST /transactions for creation of an asset.

New endpoints (proposals)

Query transactions by keys

get_owned_ids: HTTP GET /transactions?owner_after=1AAAbbb...ccc:

Returns a list of all TransactionLinks linking to unfulfilled conditions where 1AAAbbb...ccc is in owners_after.

Questions:

• Do we really want to return the whole transaction body (it's more RESTful, but also more resource intense)?
• Is filtering on /transactions with a query parameter the right way to apply REST here?
• Is owners_after the right keyword for filtering, or might there be a more appropriate name?

Query transaction by metadata id

get_tx_by_metadata_id: HTTP GET /transactions?metadata_id=1AAAbbb...ccc:

Returns a list of all transactions (full transaction body) where 1AAAbbb...ccc is the metadata_id.

Query a transaction by asset id

get_tx_by_asset_id: HTTP GET /transactions?asset_id=1AAAbbb...ccc:

Same as above, you get the idea.

Feedback welcome.

[sbellem]

Query transactions by keys

get_owned_ids: HTTP GET /transactions?owner_after=1AAAbbb...ccc:

Returns a list of all transactions (full transaction body) where 1AAAbbb...ccc is in owner_after.

Questions:

• Do we really want to return the whole transaction body (it's more RESTful, but also more resource intense)?
• Is filtering on /transactions with a query parameter the right way to apply REST here?
• Is owners_after the right keyword for filtering, or might there be a more appropriate name?

Query transaction by metadata id

get_tx_by_metadata_id: HTTP GET /transactions?metadata_id=1AAAbbb...ccc:

Returns a list of all transactions (full transaction body) where 1AAAbbb...ccc is the metadata_id.

Query a transaction by asset id

get_tx_by_asset_id: HTTP GET /transactions?asset_id=1AAAbbb...ccc:

Same as above, you get the idea.

Feedback welcome.

My opinion on the above:

• One single endpoint /transactions with support for query parameters
• Return entire payload
• The filtering keyword should be in line with the transaction schema definition

[TimDaub]

One single endpoint /transactions with support for query parameters
Return entire payload
The filtering keyword should be in line with the transaction schema definition

:+1: I agree.

[ttmc]

• Is owners_after the right keyword for filtering, or might there be a more appropriate name?

I think you meant owner_after, but anyway, as per @r-marques proposal on issue #743, owners_after will be renamed public_keys so this parameter would probably become public_key.

[TimDaub]

Cool, as a general rule I like what @sbellem proposed:

The filtering keyword should be in line with the transaction schema definition

[r-marques]

I got mixed feelings about get_owned_ids because looking at the this thread there seems to be some confusion about what it should return.

get_owned_ids as implemented in the server returns a list of unfulfilled conditions (similar to unspent outputs in bitcoin lingo) in the form of a list of TransactionLinks

Another api call which is common in blockchains is get all the transactions involving a particular public key, regardless of if its owners_before or owners_after or if it was already spent. Its basically the entire history involving a particular public key

Return entire payload

Don't necessarily agree with this. If you want the unspent conditions you only need TransactionLinks, if you want to know the status of a transaction you only need the actual status, if you want an asset you should get an asset object not the entire transaction that created that asset.

The main reason for this is that when querying the database we can already ask the database to only return part of the document instead of the entire document. Although we haven't been using this functionality a lot it would be more efficient and reduce intra cluster traffic

[TimDaub]

get_owned_ids as implemented in the server returns a list of unfulfilled conditions (similar to unspent outputs in bitcoin lingo) in the form of a list of TransactionLinks

Just updated my post in that respect. I'm not sure though if it makes sense to actually return just links or the actual object (the unfulfilled condition). Any thoughts on that?

[r-marques]

I am really not sure about this one:

• returning only the conditions is what you actually need (not entirely true because you also need the asset id and you need make sure you are not mixing conditions from different assets )
• you could only get the transaction links and then you could query for the txids you wanted, but you will probably do this all the time so it would make more sense to just return the entire transaction.

Our biggest problem right now is that we don't have a use case or feedback from users to make a better decision.

Assuming that the main api client will be the driver we should do what makes more sense for the driver and go with @sbellem proposal.

[sbellem]

Regarding whether the entire transaction payload should be returned or not, my answer was strictly within the context of the API endpoint /transactions for the GET HTTP call.

In any case we should be more formal with this, as this is the definition of the API contract, and what is required here is the definition of the request and response schemas.

I still think that we should return the entire payload of a transaction for GET /transactions?q=whatever because this is what this endpoint says: get transactions for this specific query parameter.

If a user of an API wishes to retrieve only assets, conditions, fulfilments, and so forth, then I think that we should have endpoints for this.

Another approach would be to have a boolean query parameter (e.g.: partial) such that when set to true only specific parts of the transactions are queried/returned. But this will probably require messy code, parsing the query parameters and using if/else clauses to determine what part of the transaction should be queried and returned.

It seems to me at first glance that grouping the parts (asset, etc) of a transaction by endpoints would make things clearer, easier to implement, maintain, and extend.

And how many parts de we have any ways? Three? Asset. Conditions. Fulfilments. Is that correct?

[ttmc]

And how many parts de we have any ways? Three? Asset. Conditions. Fulfilments. Is that correct?

Also metadata.

[sbellem]

Ah ok, thanks @ttmc

Just in case this is helpful, here's a brief clarification of the suggestion given above regarding whether we should return the entire payload or not:

Case 1: responses containing entire transaction payloads

When hitting the endpoint /transactions via GET with whatever query parameter we may eventually support, always return the full transaction payload.

Case 2: responses containing partial transaction payloads

For partial payloads, we could support the four main parts, via dedicated endpoints:

1. asset --> /asset
2. conditions --> /conditions
3. fulfillments --> /fulfillments
4. metadata --> /metadata

Performing GET operations on any of the above endpoints with whatever query parameter we support would then return the corresponding parts. Example:

GET /conditions?owners_before=abc

would return all conditions for which owners_before is abc, along with their corresponding transaction id.

[sbellem]

I would like to re-emphasize that I think that we should be more formal about the design of the HTTP API contract.

In the past we have used apiary for instance. We should seek a tool similar to that to clearly outline the schemas of both requests, and responses for each endpoint.

With apiary, for instance, we could actually define the contract in markdown, and make pull requests for proposed additions, changes, etc.

It is crucial that the HTTP API be clearly defined as it is the point of contact for the external world to interact with a BigchainDB federation.

[TimDaub]

I would like to re-emphasize that I think that we should be more formal about the design of the HTTP API contract.

Agree 100%.
Instead of implementing right away, I'll submit a proposal for each endpoint first.

[diminator]

closing this in favor of #830