Neo: Move consensus to a plugin

[vncoelho]

@igormcoelho, @shargon, @erikzhang and @jsolman, what do you think about that?

Maybe it is great for portability, because in the future Consensus could be isolated easily and even run in a module in a different language.

[shargon]

Mmmm, i don't like the idea very much ... Could be isolated but not in a plugin, i think

[vncoelho]

If you prefer isolated let's create it in a different project.
@shargon, do you see an easy path to load a plugin generated in other environment/language?

[shargon]

Is not easy job, but why we should use other language for consensus?

[vncoelho]

It is not a requirement, it is an idea for ensuring NEO as a template.
If it was designed for accepting any consensus in an isolated environment we could perform, easily, different tests for Consensus ideas that researchers investigate.

In terms of other languages it is just an idea that this layer could run as an isolated module. There is not a currently need for mixing the consensus layer with a code written in a different language.

[vncoelho]

But the same applies for the NEO-VM, right @shargon?
It would be great to easily just set a C++ module to execute all VM executions, @igormcoelho.

[shargon]

A modular VM (in other language) is easier, but when you need to share the storage, types, p2p, etc... it's more complicated

[igormcoelho]

In fact, if neo-project/neo becomes a middleware, it's easy to code Consensus on other languages, and attach them via interprocess communication. P2P network protocol should also be attached as another cooperative process (that could also be written on other language). So Neo project becomes blockchain specification itself (persistance could also be ported to other databases), and some common implementations. On time, it's innevitable and necessary to have multiple implementations of consensus, to guarantee that they won't share common failures. However, it's for the future... perhaps a near future.

[erikzhang]

Moving the consensus to the plugin does not help create consensus on other languages. You can load .net standard dll as plugins only.

[igormcoelho]

@vncoelho I agree with Erik that consensus neo-plugin is not a feasible way.. separating this part will require much deeper efforts.

[vncoelho]

I got it, thanks.

In this sense, what is the best direction?

[igormcoelho]

Vitor, in my point of view, this is what happens today, we have Akka for communication, but we have lots of direct accesses between Neo main components:

If we change Akka to IPC (Inter-process Communication), like mkfifo or any other techniques, we can do it quickly on shared memory (RAM), but with an overhead of serializing/deserializing between components (another component was proposed here to handle NeoVM service, which is not good for performance):

Finally, a third option would be to embed NeoVM on each component, keeping IPC between them:

This is a major redesign that is only worth when we have all these small components working well in many different languages. Anyway, the best options would be to keep solid single-language projects (like C# neo) in their current format, which is better for performance, and to propose something like that (hybrid projects) in another separated branch. Configurability will come with the lost of some performance.

P.S.: this is a huge simplification of current design, so it could have much more dependencies not presented here, for example, on NeoContract layer the NeoVM works together with Storage, and storage is not presented on this model. Consensus recovery also stores data on the persistance part... so, much more things to design carefully.

[igormcoelho]

Ok Vitor, I'll close this for now, it's unfeasible for us to follow this path now.