Lnd: Feature Request: channel.db auto-sync with cloud services
Use Case
- A large amount of server based computing is through cloud services such as AWS and Azure.
- channel.db backup is essential to maintaining a healthy LND node.
- Having a method to sync the channel.db with outside services would be a good fallback measure.
Questions
- Would channel.db be more suited for an SQL platform or noSQL key value base platform?
- Is channel.db latency looking to be a large bottleneck? Could we allow a parameter to move the channel.db completely to cloud?
junderw
All 11 comments
Related to #175.
Roasbeef
on 26 Jul 2018
Do you mean backups, or like full on replication?
Roasbeef
on 26 Jul 2018
Full on replication. Live.
In fact, even wondering if possible to replace channel.db and have it access the RDS/Dynamo for each read/write.
junderw
on 26 Jul 2018
Yes that would be possible but requires a bit of re-working to introduce an interface in front of boltdb so we can add additional backends. It would require re-implementing the KV-store interface for each new storage backend.
Roasbeef
on 26 Jul 2018
Yeah... it looks like channeldb is calling OS level file system directly... so it would need to create a mock filesystem that could translate into other interfaces without increasing latency too much...
In our discussions we think this is probably a must-have for production level deployments for exchanges. So it might not be super high priority right now. But eventually.
junderw
on 26 Jul 2018
so it would need to create a mock filesystem that could translate into other interfaces without increasing latency too much
Yeah so the goal here would be that channeldb gets some generic KV-store, and uses that directly. For the major SQL based databases, we'd need to create an abstraction layer to intelligently map the buckets/keys etc to tables. For other KV stores, then it's simpler as we can map to the same calls more or less 1:1.
So it might not be super high priority right now. But eventually.
Agreed that it's definitely something on our radar. Doing so would also make unit testing certain areas of lnd easier as we wouldn't need to create a tear down a test DB each time, instead it would just be reading/writing from like an in-memory map.
Roasbeef
on 26 Jul 2018
For other KV stores, then it's simpler as we can map to the same calls more or less 1:1.
Definitely, AWS dynamoDB or Azure Tables are KV databases. No need to migrate to SQL if not necessary... though maybe some people would like SQL. it's a preference thing imo
junderw
on 26 Jul 2018
@junderw Are you working on this? I would like to experiment on it and see if it is feasible. The only issue I can see is making sure that all writes are atomic and maintain the _consecutio_ of backups.
AWS speaking, maybe we can use SQS on top of DynamoDB (or even S3 馃 )
Are there other any considerations to take into account?
tiero
on 1 Oct 2018
I think supporting S3 would be a great idea as it's API it's widely used (minio, etc).
xsb
on 1 Oct 2018
The first step here is actually to abstract out the database usage for boltdb into a _distinct_ interface. One good target would be to emulate the walletdb interface. With that alone we can then easily map to any existing database (that meets our consistency requirements), and emulate the bucket structure and transactional semantics.
Alternatively, since the only _precious_ data is the channel information, and also the up to date instances of the switch itself (so we can continue forwarding in a fail over case), we can instead create _new_ implementations of the existing htlcswitch related interface, and erect a new set of interfaces for the channel storage. With this, the graph information is stored locally, while the crucial information required to perform a handoff are stored in the distributed database.
The latter option is nice, as it then lets the new database possibly implement optimizations that wouldn't be possible with the KV model due to the restrictions it adds. For example, it doesn't need to worry about buckets, and could instead implement certain operations as a columnar store or the like.
Roasbeef
on 3 Oct 2018
@tiero
Are you working on this?
No I am not. I am currently working on internal issues for my company.
I would appreciate any help on this issue. Thanks.
junderw
on 3 Oct 2018
Related issues
Richard87
路
3Comments
ealymbaev
路
3Comments
Roasbeef
路
3Comments
alexbosworth
路
3Comments
MrManPew
路
3Comments
Most helpful comment
The first step here is actually to abstract out the database usage for boltdb into a _distinct_ interface. One good target would be to emulate the
walletdbinterface. With that alone we can then easily map to any existing database (that meets our consistency requirements), and emulate the bucket structure and transactional semantics.Alternatively, since the only _precious_ data is the channel information, and also the up to date instances of the switch itself (so we can continue forwarding in a fail over case), we can instead create _new_ implementations of the existing
htlcswitchrelated interface, and erect a new set of interfaces for the channel storage. With this, the graph information is stored locally, while the crucial information required to perform a handoff are stored in the distributed database.The latter option is nice, as it then lets the new database possibly implement optimizations that wouldn't be possible with the KV model due to the restrictions it adds. For example, it doesn't need to worry about buckets, and could instead implement certain operations as a columnar store or the like.