Etcd: Incremental snapshot support

[swapnilgm]

I was wondering, is there any incremental snapshot support for etcd? I don't see any command on etcdctl for same as of now. For use cases where one requires regular snapshots of etcd for backup purpose, having same data as part of multiple full snapshots is really costly from storage point of view.
It would be nice to have this feature. Do you already have any plan for this?

As far as my investigation lead,

• There is some incremental snapshot implementation used internally by etcd. But it dosen't expose this to client at all. There is no API or command to access this. Why is it so?
• make-mirror command over etcdctl uses the snapshot/restore and watch based mechanism to sync the clusters. Why can't we use similar approach to have incremental snapshot? Here I propose, instead of keeping PUT and DELETE operation from watch events on destination client, we can dump it in json or some defined format in file say inc_snapshot_fromRev_toRev. So that at the time of restore, user can support backup directory with full snapshot and these incremental snapshot. And our restore command, with some additional flag to know that we have to restore from incremental snapshot list, will restore the etcd cluster. While restoring, it will restore cluster from base full snapshot. And then apply the operation from incremental snapshots sequentially using probably embedded etcd.

[swapnilgm]

@xiang90, Maintainers: Any opinion/updates about this issue?

[xiang90]

etcd/raft does snapshot internally to reclaim disk from WAL files. We do not support incremental snapshot from external. But that is something doable and wont be super hard. Do you have any interest working on it?

[swapnilgm]

Yeah sure. I'll take it forward.

[hexfusion]

I think this is important for sane backup/restore strategy +1

[gaddamas]

@xiang90, Could you please point me to documentation on snapshots or explain in detail about how snapshot functionality is used in etcd. Some questions: 1) is there snapshot at raft log (wal) level and another snapshot at bbolt db level. after adding 900k keys to a 3 node cluster, i have restarted all three nodes. two of the nodes after restart were busy with exchanging snapshot as shown in the log below. Is one node trying to send entire db to another node or is it trying to send snapshot of log from last committed index 900168? Thank you
sender node:
2018-08-23 15:34:24.489264 I | etcdserver: wrote database snapshot out [total bytes: 9223372036854775807]
2018-08-23 15:34:24.489504 I | rafthttp: start to send database snapshot [index: 900167, to f078dbaf5ed2490b]...
2018-08-23 15:34:24.491349 W | rafthttp: database snapshot [index: 900167, to: f078dbaf5ed2490b] failed to be sent out (ioutil: short read)
...
2018-08-23 15:34:25.090355 I | etcdserver: wrote database snapshot out [total bytes: 9223372036854775807]
2018-08-23 15:34:25.090482 I | rafthttp: start to send database snapshot [index: 900168, to f078dbaf5ed2490b]...
2018-08-23 15:34:25.092223 W | rafthttp: database snapshot [index: 900168, to: f078dbaf5ed2490b] failed to be sent out (ioutil: short read)
receiver node:
2018-08-23 15:34:24.392607 I | rafthttp: receiving database snapshot [index:900167, from ec5c70c3e0fa567c] ...
2018-08-23 15:34:24.392890 E | rafthttp: failed to save KV snapshot (unexpected EOF)
2018-08-23 15:34:24.489257 E | etcdserver: publish error: etcdserver: request timed out
2018-08-23 15:34:24.690707 I | rafthttp: receiving database snapshot [index:900168, from ec5c70c3e0fa567c] ...
2018-08-23 15:34:24.691197 E | rafthttp: failed to save KV snapshot (unexpected EOF)
.....

[wenjiaswe]

cc @jingyih backup/restore
cc @jpbetz nonvoting members

[jpbetz]

Part of the non-voting member design describes non-disruptive snapshot transmission (https://github.com/etcd-io/etcd/issues/9161). Once that's in place, we could just enable it for etcdctl snapshot save?

[xiang90]

etcdctl snapshot save

etcdctl snapshot save --forever could potentially starts a non-voting member and keep on receiving data and making snapshots.

[moooofly]

etcdctl snapshot save

etcdctl snapshot save --forever could potentially starts a non-voting member and keep on receiving data and making snapshots.

find no --forever flag in etcdctl snapshot save command

[root@e5d68e04-f6f6-11e9-b1f2-00163e069dac ~]# etcdctl version
etcdctl version: 3.3.11
API version: 3.3
[root@e5d68e04-f6f6-11e9-b1f2-00163e069dac ~]# 
[root@e5d68e04-f6f6-11e9-b1f2-00163e069dac ~]# etcdctl snapshot save -h
NAME:
        snapshot save - Stores an etcd node backend snapshot to a given file

USAGE:
        etcdctl snapshot save <filename>

GLOBAL OPTIONS:
      --cacert=""                               verify certificates of TLS-enabled secure servers using this CA bundle
      --cert=""                                 identify secure client using this TLS certificate file
      --command-timeout=5s                      timeout for short running command (excluding dial timeout)
      --debug[=false]                           enable client-side debug logging
      --dial-timeout=2s                         dial timeout for client connections
  -d, --discovery-srv=""                        domain name to query for SRV records describing cluster endpoints
      --endpoints=[127.0.0.1:2379]              gRPC endpoints
      --hex[=false]                             print byte strings as hex encoded strings
      --insecure-discovery[=true]               accept insecure SRV records describing cluster endpoints
      --insecure-skip-tls-verify[=false]        skip server certificate verification
      --insecure-transport[=true]               disable transport security for client connections
      --keepalive-time=2s                       keepalive time for client connections
      --keepalive-timeout=6s                    keepalive timeout for client connections
      --key=""                                  identify secure client using this TLS key file
      --user=""                                 username[:password] for authentication (prompt if password is not supplied)
  -w, --write-out="simple"                      set the output format (fields, json, protobuf, simple, table)

[root@e5d68e04-f6f6-11e9-b1f2-00163e069dac ~]# 

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.