Etcd: after migrate virtual machine, etcd ERROR~~~

[yangqinjiang]

Mar 20 10:12:12 iZ94ue30g7wZ systemd[1]: Starting Etcd Server...
Mar 20 10:12:12 iZ94ue30g7wZ etcd[3977]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://localhost:2379
Mar 20 10:12:12 iZ94ue30g7wZ etcd[3977]: recognized environment variable ETCD_NAME, but unused: shadowed by corresponding flag
Mar 20 10:12:12 iZ94ue30g7wZ etcd[3977]: recognized environment variable ETCD_DATA_DIR, but unused: shadowed by corresponding flag
Mar 20 10:12:12 iZ94ue30g7wZ etcd[3977]: recognized environment variable ETCD_LISTEN_CLIENT_URLS, but unused: shadowed by corresponding flag
Mar 20 10:12:12 iZ94ue30g7wZ etcd[3977]: etcd Version: 3.2.5
Mar 20 10:12:12 iZ94ue30g7wZ etcd[3977]: Git SHA: d0d1a87
Mar 20 10:12:12 iZ94ue30g7wZ etcd[3977]: Go Version: go1.8.3
Mar 20 10:12:12 iZ94ue30g7wZ etcd[3977]: Go OS/Arch: linux/amd64
Mar 20 10:12:12 iZ94ue30g7wZ etcd[3977]: setting maximum number of CPUs to 1, total number of available CPUs is 1
Mar 20 10:12:12 iZ94ue30g7wZ etcd[3977]: the server is already initialized as member before, starting as etcd member...
Mar 20 10:12:12 iZ94ue30g7wZ etcd[3977]: listening for peers on http://localhost:2380
Mar 20 10:12:12 iZ94ue30g7wZ etcd[3977]: listening for client requests on localhost:2379
Mar 20 10:12:12 iZ94ue30g7wZ etcd[3977]: recovered store from snapshot at index 82148214
Mar 20 10:12:12 iZ94ue30g7wZ etcd[3977]: recovering backend from snapshot error: database snapshot file path error: snap: snapshot file doesn't exist
Mar 20 10:12:12 iZ94ue30g7wZ bash[3977]: panic: recovering backend from snapshot error: database snapshot file path error: snap: snapshot file doesn't exist
Mar 20 10:12:12 iZ94ue30g7wZ bash[3977]: panic: runtime error: invalid memory address or nil pointer dereference
Mar 20 10:12:12 iZ94ue30g7wZ bash[3977]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xb5828c]
Mar 20 10:12:12 iZ94ue30g7wZ bash[3977]: goroutine 1 [running]:
Mar 20 10:12:12 iZ94ue30g7wZ bash[3977]: github.com/coreos/etcd/etcdserver.NewServer.func1(0xc42015e678, 0xc42015e470)
Mar 20 10:12:12 iZ94ue30g7wZ bash[3977]: /builddir/build/BUILD/etcd-d0d1a87aa96ae14914751d42264262cb69eda170/src/github.com/coreos/etcd/etcdserver/server.go:279 +0x3c
Mar 20 10:12:12 iZ94ue30g7wZ bash[3977]: panic(0xd6bde0, 0xc4212ba420)
Mar 20 10:12:12 iZ94ue30g7wZ bash[3977]: /usr/lib/golang/src/runtime/panic.go:489 +0x2cf
Mar 20 10:12:12 iZ94ue30g7wZ bash[3977]: github.com/coreos/pkg/capnslog.(*PackageLogger).Panicf(0xc420152660, 0xf3964c, 0x2a, 0xc42015e4e0, 0x1, 0x1)
Mar 20 10:12:12 iZ94ue30g7wZ bash[3977]: /builddir/build/BUILD/etcd-d0d1a87aa96ae14914751d42264262cb69eda170/Godeps/_workspace/src/github.com/coreos/pkg/capnslog/pkg_logger.go:75 +0x15c
Mar 20 10:12:12 iZ94ue30g7wZ bash[3977]: github.com/coreos/etcd/etcdserver.NewServer(0xc4201c6000, 0x0, 0x15b9160, 0xc4212ba1c0)
Mar 20 10:12:12 iZ94ue30g7wZ bash[3977]: /builddir/build/BUILD/etcd-d0d1a87aa96ae14914751d42264262cb69eda170/src/github.com/coreos/etcd/etcdserver/server.go:374 +0x2e39
Mar 20 10:12:12 iZ94ue30g7wZ systemd[1]: etcd.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Mar 20 10:12:12 iZ94ue30g7wZ systemd[1]: Failed to start Etcd Server.
Mar 20 10:12:12 iZ94ue30g7wZ systemd[1]: Unit etcd.service entered failed state.
Mar 20 10:12:12 iZ94ue30g7wZ systemd[1]: etcd.service failed.
Mar 20 10:12:13 iZ94ue30g7wZ systemd[1]: etcd.service holdoff time over, scheduling restart.
Mar 20 10:12:13 iZ94ue30g7wZ systemd[1]: start request repeated too quickly for etcd.service
Mar 20 10:12:13 iZ94ue30g7wZ systemd[1]: Failed to start Etcd Server.
Mar 20 10:12:13 iZ94ue30g7wZ systemd[1]: Unit etcd.service entered failed state.
Mar 20 10:12:13 iZ94ue30g7wZ systemd[1]: etcd.service failed.

[yangqinjiang]

[yangqinjiang]

Who can help me??

[gyuho]

Can you provide a way to reproduce?

[yudai]

Most likely it's from https://github.com/coreos/etcd/blob/5b9741e804a44990f67b5eeab686b7141a801346/raftsnap/db.go#L65

The snapshot file expected to exist did not exist for some reason. Maybe it's lost in the VM migration?

[hexfusion]

recognized environment variable ETCD_DATA_DIR, but unused: shadowed by corresponding flag

Is ENV path same as flag?

[jiaxuanzhou]

@yangqinjiang is it the same version of the etcd before and after the migration ?

[gyuho]

@yangqinjiang This happens when the snapshot file does not exist, probably deleted during migration. Please check and reopen if it still happens. We will improve the error message.