Neo: Create Storage.Add Interop (Neo 3)

[igormcoelho]

[EDIT:] The old proposal required another Storage flag, but now it doesn't need anything, except an interop method Storage.Add. Using it, and mempool value caches, verification transfers can be performed safely.

Example:

// .... during verification
Storage.Add("myaddress", -100);
Storage.Add("youraddress", +100);

This is supposed to work on verification, because values can be cached on mempool, and when Block is finally proposed, the "correct" value will be calculated following to tx deterministic order. So, if Tx A transfered something as network fee, and Tx B tries to do the same, this "double spending" won't be allowed on mempool time (not need to wait for block processing).

[EDIT:] old explanation using flags.
~There's a nice solution for Neo native and user tokens, if we create the concept of Storage for balances. Usually, a Storage can hold any kind of information, and it could be read-only or read-write, depending on an attribute. We propose another attribute isBalance, that can also be used together with the const attribute.~
~The behavior of isBalance key is the following: it only accepts non-negative bignumbers; every change creates a an special notification that can be aggregated after contract execution.~

~Example:~

// creating a storage balance
BigInteger funds = 0;
Storage.PutEx("myscripthash", funds, StorageAttribute.IsBalance);
// This automatically generates a notification `contract_hash + special balance flag + key + value_changed_up +funds`
// ...... NEP-5 transfer example 
BigInteger myfunds = Storage.Get("myaddress");
BigInteger yourfunds = Storage.Get("youraddress");
Storage.Put("myaddress", myfunds - value_transfer);
Storage.Put("youraddress", yourfunds + value_transfer);

This helps implementing network fees in verification time (will explain better on a longer proposal): https://github.com/neo-project/specification/pull/3

[igormcoelho]

This is a draft proposal, but its evolving. More deeply, perhaps this "balance" could be the combination of 3 attributes:

• integer / cache
• non-negative (integer property)
• notify changes

If we build this way, perhaps it could be easier to understand.

The current behavior of storage is like 'volatile' in languages: all the time we need to read the only trusted state. This is needed for things like:

x += "ab"
x += "cd"
both done in parallel, whats the result? x="abcd" or X="cdab" ? We cannot know unless we order them (in block only).

But if we deal with integers only:
x += 3
x += 4

Result is 7, in any order.

So if we constrain a storage to int only, we already gain some things. This allows us to try to emulate non-volatile storage on mempool. So before reading the real value on storage, it may be reduced by other things on mempool (like netfee consuming resources from.a storage key).

Non-negative is a property that allows this storage to never be negative, so mempool caches may never spend more than existing value.

Notify changes is a storage property that just notify every change on that storage.

@vncoelho this int caching could also be used for mempool voting of consensus nodes.

[vncoelho]

Exactly, @igormcoelho.

The int used for storages pre-balance will be a similar thing to the votes computation!
But we can have a rule to enable votes computation only if network has been stuck for X time in order to avoid this unnecessary calculation.

[igormcoelho]

So, one thing we could do,is during Verification just allow storage read and writing on int/cache values. If I spend some network fee on verification, I'm actually spending it already on mempool cache for that key, what affects parallel tx using the same input address (thus preventing double spending on verification). The definitive value is only calculated after block is relayed and persisted, and after that this cache is dropped, and definitive storage values are guaranteed to be deterministic.

Network fees should be the first to be consumed on a block header,before processing the tx applications, as this guarantee that tx netfee will always be paid.

[igormcoelho]

In the case of payable contracts, it is also simple this way. You transfer extra money for it (which is deduced already on verification cache), and it consumes the necessary part and gives you back the change on Application trigger (if necessary). Automatic refunds can be easily done too.
This is useful for token minting,for example, where the contract may be perhaps invoked twice, but funds will be spent once only (the next invocation would fail).

@shargon

[erikzhang]

What is its usage scenario?

[igormcoelho]

What is its usage scenario?

I think we can use it to provide storage-write during Verification @erikzhang , which is useful to guarantee that network fees will be paid, and consumed even if multiple tx are submitted in parallel. Regular storage cannot provide write access over verification, due to non-determinism, but we can solve non-determinism here by allowing caches to be updated on mempool directly.

The application for us is easy: we update Native NEP-5 with this IntCache storage flag, and only allow writes during verification if it has this flag. From verification perspective, what we do is creating a Map, that computes all changes made on mempool to storage keys of type IntCache, and when we read data, we read IntCache before reading the actual storage value. When block is persisted, we will drop these IntCaches (when removing tx from mempool), and persist the correct value on storage, given the tx order from the blocks. I'll try to draft a PR, so we can see this better on code...

[lock9]

But what will happen in this scenario:
x *= 4
x += 2
?
The result may change if the order is not correct and we are only using integers

[igormcoelho]

I get your point, this was just an example I gave to show how it would work for NEP-5 transfers... in fact, this would work for additive markovian functions, so multiplication would not be allowed. Good thing we don't need them on NEP-5 ;)

[igormcoelho]

But in fact, you gave me an idea now... I was prototyping something like Storage.SetCache, but now I think it's better to write Storage.IncreaseBy and Storage.DecreaseBy. This way, only simple additive stuff is allowed. More general is: Storage.Add.

[lock9]

Good idea! Now it looks safer :)

[igormcoelho]

@erikzhang perhaps the proposal now is more clear... just a Storage.Add interop is enough to do many nice things.

[erikzhang]

I did not see how useful this is. We can implement network fees in verification perfectly with #791.

[igormcoelho]

This allows (theoretically)removing all fee reverifications on.mempool, except by the last one when speaker proposes de block.
This is a complement to the change on #791.

The example I have in mind now is this Erik:

// NeoContract (perhaps NEP-5 Native GAS?)
object Main(string op, object[] args)
{
   if (TriggerType == Verification) // note that this can run on Verification
   if (op == "transfer") {
        byte[] from = args[0];
        byte[] to = args[1];
        BigInteger value = args[2];
        assert( Runtime.CheckWitness(from) );
        if(Storage.Get(from).AsBigInteger()>=value) {
              Storage.Add(from, -value);
              Storage.Add(to, value);
              return true;
        } 
   }
    return false;
};

This code above is supposed to run fine during _Verification Trigger_... The trick here is that we are not using Put, so theoretically it's possible to cache intermediate Add values on mempool, so that new transactions that arrive _already know_ that Storage.Get will return a lower value than the one actually persisted. When it comes to the Block Persist itself, when tx are finally ordered, this verification takes place one last time and will actually cause a storage change after executed, thus finally consuming resources.

[erikzhang]

We can optimize verifications without this. Just disallow all state read APIs in verification.

[igormcoelho]

We can optimize verifications without this. Just disallow all state read APIs in verification.

I agree, but NEP-5 native gas will be an exception to this rule, right? I was trying to do it in a way that we wouldn't require any exception... If we allow only Storage.Get and Storage.Add we can do this. Mempool is non-deterministic by nature, so it can control itself until it gets ordered and executed deterministically on a block.

I don't know if this is necessary now, we can leave such thing for the future, but I'm just saying it's possible to do that.

[shargon]

I don't like this :S why we need to limit the storage access? We need to be able to read the storage in verification

[vncoelho]

We need to solve this puzzle before NEO 3.

Storage of balances I agree, Shargon.
The main discussion is the scalability of the mempool, we need Auxiliary Data Structures (ADS) for reverifying almost instantaneously. The ADS is what we designed with prebalance and limmitted states.

[igormcoelho]

This proposal allows both Storage.Get and Storage.Add on verification @shargon. Only Put will be forbidden. The reason is that Add is order-independent, and Put is not.

[shargon]

But currently is not allowed put in verification.

[igormcoelho]

But currently is not allowed put in verification.

Yes. And I hope it stays that way.. such nondeterministic creature :joy:

[erikzhang]

We don't need this. We can just disable all state dependent interop functions on verification. That's all we need to do. In this way, we don't need to reverify the transactions in mempool, we just reverify the fees, which doesn't need to execute the witness scripts.

[igormcoelho]

Not feasible anymore... however, lead to nice improvements, such as garbage collection and dettached TState objects ;)