Tendermint: cleveldb socket

[ebuchman]

We need to use cleveldb instead of goleveldb because its drastically better.

But there are issues around cross compiling Go+C.

Solution is to talk to CLevelDB running in a separate processes - we only need to build the binaries for that once (or else rarely) for each platform.

Alternatively, they might be some magic we can do with Go plugins ...

[greg-szabo]

How about BadgerDB? - Frey
Let's try that too. - Bucky
There's a PR in tmlibs that is related to this: https://github.com/tendermint/tmlibs/pull/58

[adrianbrink]

The issue with including C is that it screws with deterministic builds right?

Because, I think in the long run it makes sense to be able to include C in our builds.

[melekes]

@odeke-em just curious, have you looked at go plugins or other means of registering components (with C code)?

[odeke-em]

@odeke-em just curious, have you looked at go plugins or other means of registering components (with C code)?

@melekes great question. I see very many problems with plugins that I am not sure it is a maintenance road that I would want to take also that would require passing around the *.so files which are another dependency that at that point I'd rather just ask the customer to install gcc on their computer. Using a socket or rather RPC IMHO is a better option, decouples dependencies since now we'll just need a URL, but also we can vendor a Docker container with the CLevelDB service that anyone can run locally or on the cloud.

[srmo]

Solution is to talk to CLevelDB running in a separate processes

Hm, we as core users would violate one of a main restriction/selling points in our services with that: namely that there are no additional service dependencies that require additional maintenance, etc.
Do you consider keeping a embedded/in process hook?

[srmo]

PS:
otherwise, we woul very much appreciate if this RPC/Socket interface to an external DB is abstracted away such that we can use whatever DB we like.
Are the following discussions related to this?
https://github.com/tendermint/tendermint/issues/674
https://github.com/tendermint/tendermint/issues/803
https://github.com/tendermint/tendermint/issues/1161

IIRC, there was a PR not long ago that replaced the leveldb with MongoDB (not great performance wise) but it was a start. I think that PR / proposal was rejected.

[odeke-em]

PS:
otherwise, we woul very much appreciate if this RPC/Socket interface to an external DB is abstracted away such that we can use whatever DB we like.

@srmo in deed, already done in https://github.com/tendermint/tmlibs/pull/162 and for brevity there is a sample test/usage https://github.com/tendermint/tmlibs/blob/d5d68050977ca4d44308b078b977aa7de65fd4d2/remotedb/remotedb_test.go#L24-L40

[srmo]

Thanks for the heads up. Will you keep the embedded option?

[odeke-em]

Will you keep the embedded option?

@srmo what do you mean? What's going to happen is that the server will be deployed remotely e.g. via a Docker container or by a database broker i.e. a separate process from the client. Am not sure "embedding" will apply here.

[srmo]

„Embedded“ as it is know. In our use cases with tendermint core, deploying additional components is not option.

[srmo]

*Edit (on mobile) : as it is now, i.e. in process DB

[odeke-em]

„Embedded“ as it is know. In our use cases with tendermint core, deploying additional components is not option.

Yeah, within the same process doesn't fly and that's the point of the isolation, some machines might not have gcc installed. Your requirements don't permit IPC either? In such a case as on the same machine, perhaps it'd be a cross compiled binary for your system(having been built on a system with gcc) then deployed on as a server on the same machine. Depends though on the urgency and use case though. We certainly could explore a case of plugins/shared object libraries that then get linked at runtime time to run in the same process. However, the usecase and urgency are what warrant the complexity and prioritization of solutions. So, which brings me to the question: what's your usecase? What are your constraints? Hope this makes sense.

[srmo]

Our stack revolves around the following:

• A) a node, consisting of tendermint + abci app
• B) a special functional node, connecting to A

This provides us with a deployment scenario where customers only have to install
A) the node
B) the functional node

In total 3 services. That's also our main selling point, you only have to install and maintain min 2, max 3 services, Tendermint + ABCI app (+ functional node).

This works because tendermint core uses an in-process/embedded DB, namely leveldb.
If this possibility is removed by requiring installation of an additional service, we loose one of the main selling points. Why? Because this additional service requires additional maintenance/backup strategies/ all the administrative overhead, which is not necessary in the current setup.

So as long as this solution works as a backup, we would be fine. Otherwise the whole selling point has to be reworked/redesigned.

Sure, we also wondered how an external DB with powerful query options would help in certain use cases but this brings us back to the points raised above.

Does this help?

[srmo]

PS: I'm not sure if IPC helps, it might but I'm not very familiar with deployment processes for such a setup.

[ebuchman]

We will continue to support in-process Go based level db for folks that just want the simple build/ops pipeline. For folks with stricter requirements around the robustness and performance of the underlying database, they will be able to use the SocketDB.

Closing this in favor of new issue to track this work https://github.com/tendermint/tmlibs/issues/203

[ackratos]

We need to use cleveldb instead of goleveldb because its drastically better.

will we use cleveldb at cosmos mainnet launch? @ebuchman

[melekes]

will we use cleveldb at cosmos mainnet launch?

Choice of the database is up to validators & other people running full-nodes. Tendermint does not enforce it.

[ackratos]

Choice of the database is up to validators & other people running full-nodes. Tendermint does not enforce it.

I see, thank you:)
Does tendermint has any recommendations/preference considered performance and stability (if you want run some nodes by yourselves)?
As golevel db is default implementation, how confident are you about the stability of cleveldb?

[melekes]

I'd definitely use cleveldb 🦇 Although, we have some reports of possible memory leaks (make sure to use latest version).

[odeke-em]

I'd definitely use cleveldb 🦇 Although, we have some reports of possible memory leaks (make sure to use latest version)

@melekes long time man! Please share with me reports of memory leaks if these reports are public. Thanks.

[ackratos]

make sure to use latest version

Thank you very much!
you mean latest tendermint right? I presume leveldb (compiled lib) itself doesn't leak

[melekes]

you mean latest tendermint right? I presume leveldb (compiled lib) itself doesn't leak

I meant latest leveldb.

[ackratos]

I meant latest leveldb.

okay, thank you. we are using latest release https://github.com/google/leveldb/releases/tag/v1.20
would you suggest latest master branch?

[melekes]

No, v.1.20 is fine.

[melekes]

⚠️ We've fixed a memory leak in Tendermint connected to CLevelDB https://github.com/tendermint/tendermint/blob/master/CHANGELOG.md#v0302