etcdctl check: add load test

[xiang90]

we have performance test to ensure the cluster's performance is good enough.

on the other dimension, we probably want to ensure the cluster can hold enough data without OOM.

I would suggest to start with:

1. small with 10,000 1kb kv pairs
2. medium with 100,000 1kb kv pairs
3. large with 1,000,000 1kb kv pairs
4. xlarge with 3,000,000 1kb kv pairs

/cc @spzala maybe you are interested in this one as you worked on the doc for perf check.

[xiang90]

this can help us to have a unified load test, and handle issues like https://github.com/coreos/etcd/issues/9112 much easier.

[CaoShuFeng]

It will be very useful for us to know much performance degradation will occur when kv paris become larger and larger.
So I'd suggest to test the read/write speed here.

[xiang90]

It will be very useful for us to know much performance degradation will occur when kv paris become larger and larger.

For that, you should try out benchmark tool in this repo. If you have a specific use case, you should do it yourself.

[CaoShuFeng]

For that, you should try out benchmark tool in this repo. If you have a specific use case, you should do it yourself.

Oh, there is already a benchmark tool!
Thanks

[spzala]

@xiang90 sure, sorry missed it earlier. I will be working on it. Thanks!

[spzala]

@xiang90 hi, to play with benchmark I was trying to install/run it as go get github.com/coreos/etcd/tools/benchmark and go run tools/benchmark/main.go but both gives me following errros

go get github.com/coreos/etcd/tools/benchmark
# gopkg.in/cheggaaa/pb.v1
../go/src/gopkg.in/cheggaaa/pb.v1/pb_x.go:43:13: undefined: unix.IoctlGetWinsize
../go/src/gopkg.in/cheggaaa/pb.v1/pb_x.go:60:26: undefined: unix.IoctlGetTermios
../go/src/gopkg.in/cheggaaa/pb.v1/pb_x.go:70:12: undefined: unix.IoctlSetTermios
../go/src/gopkg.in/cheggaaa/pb.v1/pb_x.go:88:12: undefined: unix.IoctlSetTermios

Am I doing something wrong? Is this something am missing in my environment? Googling on the error gives me very little information related to golang. Thanks for any help!

[spzala]

I am running go version go1.9.2 linux/amd64

[gyuho]

@spzala Try go install -v ./cmd/tools/benchmark

[spzala]

Aha, that's it!! Worked like a champ. Thanks @gyuho

[spzala]

@xiang90 hello, if I understood the issue correctly, we need to test the cases you have suggested manually and make sure how much physical memory is used. And then create a test similar to one in benchmark put.go but with memory usage? or are you referring a test related to check perf? Thanks! /cc @gyuho

[xiang90]

@spzala

Right now, in etcdctl there is a perf check command: https://github.com/coreos/etcd/tree/master/etcdctl#check-perf-options.

etcdctl usually gets installed by users, and the perf check command is very simple and intuitive

etcdctl check perf --load=small

the benchmark tool can also measure throughput/latency like perf check. but it is designed for flexibility not for ease of use. users have to provide all sorts of parameters instead of just small or large.

similarly, we want to add a easy to use command etcdctl check datascale to ensure the deployment can hold the given set of data. the command should measure the memory usage, and report success if the memory usage is <= 60% (or some other reasonable value) of the memory on the machine.

[spzala]

@xiang90 perfect, thank you so much!!