Etcd: low get performance when use auth

[qmhu]

When enable auth for etcd, request latency increase a lot. the test result is as below:

test env: Ubuntu14.04 Server virtual machine+16G
test case: php script to put a value in etcd and send 100 sequence get request and record the time cost for 100 get request then calculate average latency for every get request.

http: 0.0006s
http+auth: 0.14s
https: 0.08s
https+auth:0.17s

besides, enable https in etcd also make latency increase a lot.

[yichengq]

The cause is that we use bcrypt to hash the password, and it needs ~100ms to check the password could match the hash value.

Any idea to improve this? @barakmich

[qmhu]

any ideas?
my project is a web project which build by php + wordpress and I need get some config from etcd, i want to perceive config change from etcd real time. so I have to get config very frequently.

but consider 0.17s every request's latency, I have to reduce request to etcd as much as possible.

[citywander]

Maybe you can cache, plaintext and hash.

[xiang90]

@qmhu @citywander We are going to investigate more and try to solve it soon. Stay tuned.

[xiang90]

@philips @barakmich Any opinion on this one? We can either cache the result or just use plain test as password (does etcd really need to protect the passwords stored in it?)

[barakmich]

NO. We should not save passwords in plaintext. Come on, guys. Bcrypt is there for a reason. Now, that said...

Basic Auth is sending plaintext on the wire. This is why we have TLS. If you generate a bearer token (a la OAuth) after the first auth and store it etcd-side with a TTL, you can get a plaintext token, the comparison speedup that will eventually timeout gracefully. So your first request will still be ~100ms, but the subsequent ones can be considerably faster.

[xiang90]

@barakmich If we do want the security, the bcrypt is probably the only option and the slowness is one of its feature.

To speedup, we need to go the plaintext token way. It changes both the server side and client side. If we go that way, we probably need to move this to a lower priority for 2.2 release.

[xiang90]

Lower priority of this issue since this is not a block, but a performance improvement. I am still not sure which is the best way to solve it.

/cc @philips @yichengq @barakmich @citywander @qmhu Any suggestion would be appreciated.

[qmhu]

From my side, we can't stand for 0.1s+'s latency for every request. I support using plaintext but enable cache in server side is fine to me as long as the latency can regain back to a normal level.

by the way, if etcd can support multiple get, we can reduce the get request times,and this will help to improve my site's performance.

@xiang90 @barakmich

[thepwagner]

In the HTTPS case, why not pull the "username" from the client's X509 DN?

That would sidestep the bcrypt() bottleneck, and smarter clients could KeepAlive to perform multiple requests with a fixed authentication cost.

[roboll]

3916 implements authentication using common name, that should solve this.

[xiang90]

closed by https://github.com/coreos/etcd/pull/5991

[ZeaLoVe]

@xiang90 Could you please tell me how this problem solved? any docs?
I tried version 3.1, performance still bad. In my case, I try to use etcd with http+auth.

[andybrown668]

I have the same issue using the latest (?) docker image 3.3.9
I guess I'm using authentication in the wrong way because the performance degradation using basic auth makes it a complete non-starter.
My implementation (in go) makes use of an etcd client with this config:

    cfg := client.Config{
        Endpoints: endpoints,
        Username: os.Getenv("ETCD_USERNAME"),
        Password: os.Getenv("ETCD_PASSWORD"),
        Transport: client.DefaultTransport,
        // set timeout per request to fail fast when the target endpoint is unavailable
        HeaderTimeoutPerRequest: time.Second,
    }
    c, err := client.New(cfg)

I then retain the connection and also retain the keys api I get from calling NewKeysAPI
It looks like I get the same order of slowdown as the orginal reporter of this issue but the issue was closed a long time ago.

What should I be doing to get authenticated access and performance?