Bigchaindb: Define a json template for digital assets

[r-marques]

At this point the digital asset looks like this:

{
    "hash": "<sha3 hash of the payload>",
    "payload": {
        ...
    }
}

The digital asset is defined by the payload which is a generic json object. The hash is added by bigchaindb to allow users to query bigchaindb for the digital asset hash.

This is a simple way to keep track of digital assets on bigchaindb but it is not enough.
We need a generic template for a digital asset so that we can provide more full feature queries on the digital assets.

[r-marques]

Proposal a JSON template for digital assets.

This proposal may change as needed to address comments or during implementation. Once the implementation is finished we will add the spec to the documentation.

Right now the payload field is being used for 2 different purposes: to define the digital asset; and to add extra information to transactions.

The proposal is to divide them into two separate fields. A digital asset field that is created in a CREATE transaction in which its properties are used by the consensus rules. And a payload field which as generic information that users may want to append to their transactions, and are not used by the consensus rules.

A digital asset may have several characteristics:

• Unique ID
• Whether the digital asset is divisible or un-divisible
• Whether the digital asset can or cannot be updated in the future
• The amount of the digital asset
• If divisible it can be possible to refill in the future or not.
• Data field provided by the user

The transaction would have an asset field with this structure:

{
    "id": "<uuid>",
    "divisible": "<true | false>",
    "updatable": "<true | false>",
    "refillable": "<true | false>",
    "data": "<json document>"
}

• id: Unique identifier
• divisible: Whether the asset is divisible or not. Defaults to false.
• updatable: Whether the data in the asset can be updated in the future or not. Defaults to false.
• refillable: Whether the amount of the asset can change after its creation. Defaults to false.
• data: A user supplied json document with custom information about the asset. Defaults to null.
• _amount_: The amount of "shares". Only relevant if the asset is marked as divisible. Defaults to 1. The amount is not specified in the asset, but in the outputs. (see https://github.com/bigchaindb/bigchaindb/issues/125#issuecomment-241381414 )

Changes to be the current transaction model.

data field is renamed to payload. Remove the uuid field and nested payload.

Previous CREATE transaction model:

{
    "id": "<sha3 hash>",
    "transaction": {
        "version": "transaction version number",
        "fulfillments": [],
        "conditions": [],
        "operation": "<string>",
        "timestamp": "<timestamp from client>",
        "data": {
            "uuid": "<uuid4>",
            "payload": {
                "title": "The Winds of Plast",
                "creator": "Johnathan Plunkett",
                "IPFS_key": "QmfQ5QAjvg4GtA3wg3adpnDJug8ktA1BxurVqBD8rtgVjP"
            }
        }
    },
}

New CREATE transaction model:

{
    "id": "<sha3 hash>",
    "transaction": {
        "version": "transaction version number",
        "fulfillments": [],
        "conditions": [{
          "cid": "<condition index>",
          "condition": { },
          "owners_after": ["<new owner public key>"],
          "amount": "<int>"
        }],
        "operation": "<string>",
        "timestamp": "<timestamp from client>",
        "asset": {
            "id": "<uuid>",
            "divisible": "<true | false>",
            "updatable": "<true | false>",
            "refillable": "<true | false>",
            "data": "<json document>"
        },
        "metadata": {
             "id": "<uuid>",
             "data": "<json document>"
        },
    },
}

New TRANSFER transaction model:

{
    "id": "<sha3 hash>",
    "transaction": {
        "version": "transaction version number",
        "fulfillments": [],
        "conditions": [{
          "cid": "<condition index>",
          "condition": { },
          "owners_after": ["<new owner public key>"],
          "amount": "<int>"
        }],
        "operation": "<string>",
        "timestamp": "<timestamp from client>",
        "asset": {
            "id": "<uuid>",
        },
         "metadata": {
             "id": "<uuid>",
             "data": "<json document>"
        },
    },
}

The amount is specified in the condition.

Implementation details

We need to add extra arguments to util.create_tx(..., data=None, divisible=False, updatable=False, amount=1, refillable=False)
Probably safer to add the uuid server side.
Not sure at this point how the update or refill of a digital asset would be done.
TRANSFER transactions probably don't need the asset field (it can be marked as null) since it would be mostly used to change ownership of the digital asset. We may need references to the asset, especially for divisible assets we need to make sure that different inputs relate to the same digital asset (so that a transaction does not try to join shares from different divisible assets).

Difference between asset.data and payload

asset.data is user defined data related to the digital asset being created.
payload is user defined data related to a specific transaction:

Example asset and payload:

"asset": {
    "id": "b6280227-44c4-4667-91dd-de9c79f0bc81",
    "divisible": true,
    "updatable": false,
    "amount": 10,
    "refillable": false,
    "data": {
        "creator": "Andy Warhol",
        "title": "32 Campbell's Soup Cans",
        "dateCreated": "01-01-1962",
        "exampleOfWork": "https://en.wikipedia.org/wiki/Campbell%27s_Soup_Cans#/media/File:Campbells_Soup_Cans_MOMA.jpg",
        "editions": 10
    },
    "payload": {
        "spool_operation": "REGISTER"
    }
}

[diminator]

updatable: Whether the data in the asset can be updated in the future or not. Defaults to false.

How does the new TRANSFER looks like?

• will it also have an asset field?
• is it copied from the input.asset field?
• If not, then how to update?

[r-marques]

Not sure at this point how the update or refill of a digital asset would be done.

Just think this is a desirable feature that should be supported. But don't know how it would be implemented yet.

[diminator]

ok, maybe start from what you need to get the divisible assets working (and maybe the backlink from any TRANSFER to it's genesis CREATE tx)

[ttmc]

Maybe change payload to something more clear, like tx_metadata. I can foresee many people getting confused about what informatoin to put where, hence the need for clear names.

[TimDaub]

divisible: Whether the asset is divisible or not. Defaults to false.
amount: The amount of "shares". Only relevant if the asset is marked as divisible. Defaults to 1.

I'd suggest to remove divisible as an attribute and only have amount.
A digital asset with:

• amount < 1: Is invalidated by consensus
• amount == 1: Is a "non-divisible" asset
• amount > 1: Is a divisible asset

• If divisible it can be possible to refill in the future or not.
• Whether the digital asset can or cannot be updated in the future

For now, I'd leave both of these goals/requirements out until we have a use case for them (which we haven't so far, if I'm not mistaken).

How does the new TRANSFER looks like?

I'd like to know more about this as well. I could see that we drag asset.data along the way (validated by consensus), whereas payload can individually set respective to any transaction.

Maybe change payload to something more clear, like tx_metadata. I can foresee many people getting confused about what informatoin to put where, hence the need for clear names.

Second that. It should be clear that asset.data is something that was being set in a CREATE transaction, whereas payload is specific to an individual transaction (both CREATE and TRANSFER).

[sohkai]

@TimDaub

• Whether the digital asset can or cannot be updated in the future

For now, I'd leave both of these goals/requirements out until we have a use case for them (which we haven't so far, if I'm not mistaken).

Actually this has already come up multiple times with COALA modelling; depending on the queryability of transactions, we may need to come up with ways to update a transaction's state (to support forward linking, as opposed to our current backward links). However, I agree in that we should spin updates and refills out into entire projects, since they'll take some investigation and will likely be involved to implement.

---

Just to put on digital paper some thoughts about updatability:

• Git does diffs; might be difficult to reconcile across transactions if no indices are used (also would have to figure out merge algorithms)
• CouchDB does full document revisions; might be a waste of space but at least it's easy
  
  
  
  • Also implements MVCC for collision handling; first-to-save wins, others get error
  
  
• We might have to consider how an update affects other assets that have links back to other assets (e.g. If I hold a Key asset has a link to the creation transaction of a House asset, what happens if I update the House asset?)

[r-marques]

@TimDaub

I'd suggest to remove divisible as an attribute and only have amount.

My thought was that nothing prevents you from creating a divisible digital asset with amount=1 that can be refilled.
This is similar to the spool protocol in which we would create a piece and only specify the number of editions later. Or even having unlimited editions and refill as needed.

For now, I'd leave both of these goals/requirements out until we have a use case for them (which we haven't so far, if I'm not mistaken).

I agree. In an initial implementation we will not be able to refill or update a digital asset. Just put it there because is a use case we will want to support.

I am not sure yet how the TRANSFER will look like. I will now more while implementing divisible assets and figuring out what is needed. I don't think there is a need to drag asset.data around. We just need a way to easily point to it from any subsequent transaction.

Maybe change payload to something more clear, like tx_metadata. I can foresee many people getting confused about what informatoin to put where, hence the need for clear names.

I think I will renamed it to OP_RETURN, this way everyone knows how it works.

[r-marques]

@sohkai

We might have to consider how an update affects other assets that have links back to other assets (e.g. If I hold a Key asset has a link to the creation transaction of a House asset, what happens if I update the House asset?)

I don't think this is a problem for us since we don't know and don't care about what the digital asset is (its just a json document). Its the user responsibility to know what they are writing, "updating" in the database

[sohkai]

@r-marques

we don't know and don't care about what the digital asset is (its just a json document). Its the user responsibility to know what they are writing, "updating" in the database

True; I meant the comment as more a concern for queryability, since we're "updating" the database but the existing links are still to the old objects.

[r-marques]

In terms of queryability you would need to be able to return the history of the digital asset or return the most recent version.

If Key has a link to an old version of House (not sure how this would work) then its ok. That Key asset was valid for the old version of House, now you should decide if that data is still valid after the House update not the database.

This is mostly expeculation right now because I don't know how this would be implemented or what are the use cases.

[diminator]

I think I will renamed it to OP_RETURN, this way everyone knows how it works.

hahahaha

[r-marques]

Some more thoughts related to the digital asset definition.

Since the asset field is defined at the transaction level this means that we cannot have multiple assets per transaction. Each transaction deals with only one digital asset. We can still have multiple inputs and multiple outputs in which the multiple inputs are currently owned shares of a divisible asset and multiple outputs can be sending shares do different public keys.

I am fine with this for now. It simplifies the initial implementation but if the need arises for the future we can consider moving asset to the outputs (conditions) so that we can have a single transaction dealing with multiple digital assets.

Regarding linking TRANSFER transactions to the digital asset we can use the asset field to hold the unique id of the digital asset. This way we can with a single query retrieve the digital asset or filter to retrieve the entire history of a digital asset.

To implement this we could look at the asset field of the input(s) and use that asset id. At the same time we can make sure that all the inputs are for the same digital asset.
A problem here is that we need to retrieve the input transactions at creation time. Right now we only do this when signing to retrieve the conditions.

[r-marques]

There needs to be an additional change to initial model.
In order to allow for multiple input/output the amount needs to go into the condition.
This way a transaction can transfer shares to multiple public keys, as well as combining multiple shares in the input

[r-marques]

I finished implementing the digital asset template as described in this issue.
There is only one detail missing.

Since we now actually have a place to store digital assets does it make sense to still index the payload, and query by payload?
The way I see it the payload could just be a generic json document provided so that users can attach extra information to a transaction. So we could just remove the uuid in the payload and the querying by the payload.

What would be the equivalent of get_tx_by_payload_uuid now that we have digital assets (this is not to discuss querying, this is only to replace the functionality of get_tx_by_payload_uuid)?
We could have get_transactions_by_asset_id. This would return all transactions related to that asset id and/or get_asset_by_id and this would return the created transaction that contains the actual data about the digital asset.

[sohkai]

We could have get_transactions_by_asset_id. This would return all transactions related to that asset id and/or get_asset_by_id and this would return the created transaction that contains the actual data about the digital asset.

This sounds pretty good. Is it a problem if there's a large number of transactions related to an asset (e.g. do we need to think about pagination / etc)?

So we could just remove the uuid in the payload and the querying by the payload.

It would be nice to be able to filter transactions based on their payload, for example if I wanted to only get a certain type of transaction, but I guess right now we don't really do this (we only allow searches by the uuid?)?

[r-marques]

This sounds pretty good. Is it a problem if there's a large number of transactions related to an asset (e.g. do we need to think about pagination / etc)?

This would be mostly an api issue I guess

[TimDaub]

We could have get_transactions_by_asset_id. This would return all transactions related to that asset id and/or get_asset_by_id and this would return the created transaction that contains the actual data about the digital asset.

OK, this can be an option. I'd like to make sure though that we have a similar API for querying a payload by hash (and eventually by IPLD-hash).

[r-marques]

OK, this can be an option. I'd like to make sure though that we have a similar API for querying a payload by hash (and eventually by IPLD-hash).

So should I keep the payload uuid and querying by payload?

[r-marques]

I am still not sure if TRANSFER transactions should carry the entire digital asset payload or if the asset uuid is enough.

The advantage of carrying the entire asset payload is that it may make validations simpler.
The disadvantages is that it is not efficient (we end up with a lot of redundant data) and we would still need to compare it to the CREATE transaction to make sure the new transaction is not trying to change the digitial asset

[diminator]

I am pro linking to the asset field of the CREATE. If indexed, it should be a O(1) lookup, no?

[r-marques]

Its a secondary index should be O(log(n))

[r-marques]

done in #680