Etcd: Cannot restore etcd V2 data after backup when data-dir exists snapshot files.

[zhousoft]

etcd version:

etcd Version: 3.2.4
Git SHA: c31bec0
Go Version: go1.8.3
Go OS/Arch: linux/amd64

step 1.
I start a new etcd node and set several data(only V2 data)，make sure there is no snapshot file，then make a backup.

etcdctl backup --data-dir /home/etcd_data --backup-dir /home/etcd_data_backup

kill the old etcd process, start a new etcd:

/etcd -name etcd1  --data-dir=/home/etcd_data_backup/ -force-new-cluster

it wokers well, the data set before has be recovered.

step 2.
Removing all data dir created in setp 1 and start a new etcd with "--snapshot-count '10'" and set enough data to generate snapshot file,then:

etcdctl backup --data-dir /home/etcd_data --backup-dir /home/etcd_data_backup

kill the old etcd process, try to restore from backup,

/etcd -name etcd1  --data-dir=/home/etcd_data_backup/ -force-new-cluster

restoring failed, get the message as:

panic: recovering backend from snapshot error: database snapshot file path error: snap: snapshot file doesn't exist
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xb5129c]

goroutine 1 [running]:
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.NewServer.func1(0xc4201e6678, 0xc4201e6470)
        /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:279 +0x3c
panic(0xd628e0, 0xc42016b410)
        /usr/local/go/src/runtime/panic.go:489 +0x2cf
github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog.(*PackageLogger).Panicf(0xc42018bfc0, 0xf31eff, 0x2a, 0xc4201e64e0, 0x1, 0x1)
        /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog/pkg_logger.go:75 +0x15c
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.NewServer(0xc42025e000, 0x0, 0x1402540, 0xc42016b350)
        /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:374 +0x2e39
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed.StartEtcd(0xc420093180, 0x0, 0x0, 0x0)
        /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/embed/etcd.go:147 +0x7c0
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.startEtcd(0xc420093180, 0x6, 0xf0e97d, 0x6, 0x1)
        /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/etcd.go:186 +0x58
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.startEtcdOrProxyV2()
        /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/etcd.go:103 +0x15ba
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.Main()
        /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/main.go:39 +0x61
main.main()
        /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/etcd/main.go:28 +0x2

a workaround works for me is to copy the db file form /home/etcd_data/member/snap/ to /home/etcd_data_backup/member/snap/, but I am not sure is it correct way?

[hexfusion]

a workaround works for me is to copy the db file form /home/etcd_data/member/snap/ to /home/etcd_data_backup/member/snap/, but I am not sure is it correct way?

@zhousoft I believe that you are running into a known issue that to restore etcd v3 you require v3 data. Please review the note in the upgrade doc below. I understand you are not upgrading from v2 but as you only have v2 data the problem is the same.

https://github.com/coreos/etcd/blob/master/Documentation/upgrades/upgrade_3_0.md#upgrade-checklists

It is recommended to write a dummy v3 key so that some v3 data exists.

ref: https://github.com/coreos/etcd/issues/9480

[hexfusion]

So to answer your original question I believe you could also do a v3 snapshot when the node is still running as an alternative to copying the directory. This would give you latest v3 snapshot then restore v2 as you are doing.

$ ETCDCTL_API=3 etcdctl --endpoints $ENDPOINT snapshot save /home/etcd_data_backup/member/snap/db

[zhousoft]

@hexfusion Thanks for the reply, it is helpful.