Neo: Contract redirection record

[erikzhang]

Currently, when a smart contract is upgraded, all stored data is migrated to the new contract. If there is a large amount of data in the contract, the workload of the migration will be very large and may cause a DoS attack.

So I proposed a redirection mechanism. Instead of migrating data directly during each contract upgrade, a redirection record is created. When the contract reads the data, it can find the original contract hash based on the redirection record, thus accessing the correct data.

The downside to this is that the old contract cannot be redeployed in the future. But I don't think this will have any effect. Unless a hash collision occurs, this probability is negligible.

[shargon]

Maybe we can iterate all the keys and move these keys to the new one, is faster than Get, Remove and Put.

[igormcoelho]

I agree Erik, for the current data format.. however, on MPT, the scripthash is just a header, so migration would happen immediately, independent of how many keys are there. So, I think its better to adopt MPT asap.

[erikzhang]

however, on MPT, the scripthash is just a header, so migration would happen immediately, independent of how many keys are there. So, I think its better to adopt MPT asap.

How does MPT help migration happen immediately?

[igormcoelho]

On MPT, storage keys are spread through all the tree (or trie to be precise), so when a change is made on the leaf (value change for example), the root node changes. However, when a change is made on root, no change needs to be propagated down the tree. Our current proposal creates a MPT for every contract, so a scripthash change will just affect the MPT contract root node, a constant-time change O(1). To make things more solid, we can also propose a global MPT node that stores the scripthashes of the existing contracts (with storage capabilities), but even this change would only require a maximum of 20 steps in a near impossible scenario, so it will also be O(1).
On general terms, problem is fully solved with MPT. However, MPT is not good for parallel write access, and this is one thing we may want to think carefully.
Currently, there's no good/perfect way to allow parallelism on storage, but this will become even harder with MPT. One natural parallelism that will keep working without locks, as explained on whitepaper, is by executing different contracts in parallel (without dynamic invoke). But when executing operations on th same contract, I don't see many options instead of a lock mechanism. I think the trace proposal (https://github.com/neo-project/neo/issues/602) can help in this sense, but it is only a performance feature that we can think about for the future, and does not impact in current design.

To finish, even if storage is not affected by migration, perhaps it's nice to leave a proxy from the old contract to the new one, in order to make things easier for users when migrating contracts. In this sense, I fully agree with this proposal.

[igormcoelho]

@rodoufu we need to prepare a tutorial on MPT for the whole community understand our proposal. We also need to make sure that issue with "other implementations" is effectively resolved.

[rodoufu]

@igormcoelho, when you say tutorial, what are you thinking about?

[igormcoelho]

@igormcoelho, when you say tutorial, what are you thinking about?

Some slides, or a medium post... explaining step by step how the proposal will be. The simpler the better :)

[igormcoelho]

@erikzhang This contract redirection is an amazing idea, just to enforce it here... I always want this feature!!

In the case of NEP-5 "frameworks" and contract inheritance this would be very useful, as people may migrate the base, and it will still be accessible: https://github.com/neo-project/neo/issues/772

[shargon]

Is possible to use leveldb in parallel? is thread safe?

[erikzhang]

Is possible to use leveldb in parallel? is thread safe?

Yes.

[vncoelho]

@erikzhang and @igormcoelho, it is another topic, however, a little bit related, what about a native class that contracts inherits if they are willing to use migrate?

[igormcoelho]

In fact, it's a connected topic brother... very well remembered, it's fundamental to the concept of redirection here. Perhaps, users should be able to disable redirection when invoking the appcall (Call and CallWithRedirection). This way, you can have "raw" call (doesnt trusting migration), and "fluid call", that trusts contract and its migration. These models may interoperate for a "safe" inheritance model (native contract is in fact a contract with continuous migration... every release is a new migration, transparent to the user).

[igormcoelho]

In the case of massive migration, it's better to solve this with MPT storages. A single storage key changes the whole storage location @rodoufu (whole storage is moved in O(1) constant time)

The proposal by Erik is in fact much deeper (in my opinion), it means that you could invoke some contract, and it "automatically" invoke another one (migrated from the first one).
The point there is:

• Do we want/need this feature?
• How to implement it?

To implement it, my opinion is to have two different types of calls, one accepting direction, and another that won't accept. There are situations where you must be sure that scripthash matches exactly.

Some examples of needed discussion:

• Withdraw funds from contractA on verification. This will invoke contractA... but wait, contractA now is contractB (after migration), but which funds will be taken back, contractA, or contractB? What happens if I put funds on contractA, after migration to contractB? Should it migrate from contractA to B? Will it allow another future deploy on contractA after migration to contractB? Will a contract actually invoke the next one (by redirection), or only a transparent redirection (I favor this option)?

[igormcoelho]

My opinion in what should happen after redirection (we can discuss over the bullet points):
1) Funds (native) should be redirected after migration, and no future deploy allowed on the same key (this allows native assets to flow).
2) Redirection should happen in a transparent manner (such as on verification), so that it's resolved directly after resolving script itself (in a single transparent step). Example: entryscript calling Call contractA will actually execute contractB, and contractB will see callingscript as the entryscript (not contractA redirected to it). Native NEP5 can implement this by calling some sort of System.Contract.MigratedTo, and re-moving the assets, on the fly (native can be done automatically), or even letting them be there, as future access will also provide valid access (unless verification rules changed after migration).
3) Verification should work properly, using redirection, without noticing that redirection happened (this is natural due to the workings of Verification Trigger... a new contract will be called over the same trigger). This is important to be able to move any NEP5 asset (non-native), from past accounts to newer ones.
4) Calls should have the ability to CallWithoutRedirection, so that it will fail if contract has migrated. This should be the default on final/sealed contracts (no migration allowed)

[shargon]

In my opinion we need to know how many information could produce a big delay, because maybe is easier to solve it with multi threads.
We need some benchmarks.

[erikzhang]

The bottleneck of storage migration lies in the hard disk IO, not the CPU. So multithreading does not solve the problem here.