K3s: Consider replacing embedded sqlite with *embedded* etcd in the build

[sandys]

Part of the simplicity and attraction of sqlite is that it is embedded inside k3s. That makes it very simple to use and manage.

etcd3 can also be embedded in a similar way

1. https://godoc.org/github.com/etcd-io/etcd/embed
2. http://docs.activestate.com/activego/1.8/pkg/github.com/coreos/etcd/embed/
3. https://github.com/sensu/sensu-go/blob/master/backend/etcd/etcd.go

Since the HA direction needs etcd anyway.. I'm proposing dropping support for sqlite as the default embedded non-HA option and switch to embedded etcd as the default. This will reduce overall effort of maintainability of two entirely different datastores.

EDIT:

1. this issue also talks about how to do it and use it as a jumping off point into an external etcd cluster - https://github.com/etcd-io/etcd/issues/10554

[semtexzv]

Another alternative for HA is something like this, https://github.com/rqlite/rqlite , which is still very lighweight compared to etcd

[liyimeng]

it seems pretty simple to embed etcd, rsqlite is more lightweight in which term? less memory footprint?

[sandys]

@liyimeng thanks for your question.
Well im not claiming sqlite is more lightweight - i think the current default data store of k3s is sqlite because it is embedded inside the k3s daemon (and doesnt need an external etcd service running).

However, etcd can also be embedded - that basically makes it so much simpler for k3s devs since the HA solution is based on etcd only. so they can completely remove the sqlite code.

[cjellick]

This isn't really the direction we are taking k3s. We are going to continue to focus on and invest in non-etcd solutions for k3s's database.

For single node installs, the primary motivation for using sqlite is that etcd is much more resource intensive that sqlite. We want this to be able to run well on pretty slim machines.

For HA, check out https://rancher.com/docs/k3s/latest/en/installation/ha/. The recommended HA deployment will be to back k3s with an external SQL database (postgres to start, but mysql soon after). Though, external etcd will also be supported if that is your preference. The primary motivation here is that we believe sql dbs are less challenging and complex to operatre, especially when you consider managed solutions like RDS.

Longer term for HA, we are working on a distributed embedded DB solution. Currently, we are using dqlite for that. See: https://github.com/rancher/k3s/issues/300. The motivation for this solution is to make an HA k3s install feel like a simple, single appliance for edge scenarios. Think of managing thousands of small geographically dispersed kubernetes clusters. In this scenario, we don't want the DB to be a separate concern for the operator. We want the operator to have that "all-in-one" experience.

I'm going to close this issue, but we can continue the discussion in comments here if you have more input.

[sandys]

Hi Craig
If your HA strategy is SQL based, then it makes entire sense to stay SQL.

However i might not agree with the stand that a HA solution needs RDS -
from a management or a cost perspective. Managed databases as very
expensive. Three EC2 nodes are much cheaper than the equivalent multi-AZ
RDS node.

We would prefer to adopt something that is built inside k3s to give us that
capability without the need or cost of external db.

In which case dqlite is indeed what we are banking on..

On Fri, 1 Nov, 2019, 01:56 Craig Jellick, [e-mail] wrote:

This isn't really the direction we are taking k3s. We are going to
continue to focus on and invest in non-etcd solutions for k3s's database.

For single node installs, the primary motivation for using sqlite is that
etcd is much more resource intensive that sqlite. We want this to be able
to run well on pretty slim machines.

For HA, check out https://rancher.com/docs/k3s/latest/en/installation/ha/.
The recommended HA deployment will be to back k3s with an external SQL
database (postgres to start, but mysql soon after). Though, external etcd
will also be supported if that is your preference. The primary motivation
here is that we believe sql dbs are less challenging and complex to
operatre, especially when you consider managed solutions like RDS.

Longer term for HA, we are working on a distributed embedded DB solution.
Currently, we are using dqlite for that. See: #300
https://github.com/rancher/k3s/issues/300. The motivation for this
solution is to make an HA k3s install feel like a simple, single appliance
for edge scenarios. Think of managing thousands of small geographically
dispersed kubernetes clusters. In this scenario, we don't want the DB to be
a separate concern for the operator. We want the operator to have that
"all-in-one" experience.

I'm going to close this issue, but we can continue the discussion in
comments here if you have more input.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/rancher/k3s/issues/845?email_source=notifications&email_token=AAASYU73RTGGQWUV3J3KBQTQRM5PZA5CNFSM4I3O6TE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECZEM7I#issuecomment-548554365,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAASYUZS6OZM3ZRR2FJO6QDQRM5PZANCNFSM4I3O6TEQ
.

[cjellick]

Makes sense. That's while we'll support both options. Some operators really like the RDS option (or getting a sql db from their in-house db ops team). Others will like the dqlite option (which will have its own set of trade-offs, I think).

[ibuildthecloud]

Embedded etcd is coming.... stay tuned.