Etcd: Etcd Server Rate Limiting

[vivekpatani]

Hello,

Are there any efforts to have rate limiting made available as a feature? This could also potentially include something along the lines of a rate limit trigger based on some predefined conditions. For example, start rate limiting when the DB size reaches 80% of quota or something similar.

Thanks.

[vivekpatani]

Any response on this?^

[tangcong]

@vivekpatani Do you take a look whether the Qos feature #8483 meets your demands? I submitted an initial proposal draft before.

[vivekpatani]

In addition to your proposal, would it make sense to add:

1. The ability to limit a request type (like PUT - they are expensive) and optionally allow some request type (COMPACT) to be not be subject to the QoS/Rate Limiter.
2. In addition to this, a server would trigger rate limiter just on a custom user defined rule set. For example: The server reaches 80% storage capacity, only then should the QoS/rate limiter act according to the guidelines in the first 1.

What is your use case/objective for this QoS/RateLimiter component?

@tangcong

[tangcong]

1. its aims to throttle expensive read requests based on indicators such as request path, packet size, grpc method and latency and try to ensure that cluster performance does not degrade too much.
2. It can provide simple rate limiting capabilities in multi-tenant scenarios.(For example, different tenants have different paths.)

[tangcong]

I agree with your idea. I think we can add a new indicator of "storage capacity" to meet your demand. @vadimdemedes

[vivekpatani]

Generic RL flow. @tangcong

Will be discussing this issue in the next ETCD meeting on August 27 - https://docs.google.com/document/d/16XEGyPBisZvmmoIHSZzv__LoyOeluC5a4x353CX0SIM/edit#

[vivekpatani]

@tangcong https://docs.google.com/document/d/1wQQ_L3cLyI1t14zp-PvarbjJYtlRsz_P9LwDR3uelc8/edit

[vivekpatani]

@gyuho @wenjiaswe how do I channel this to get community feedback?

[wenjiaswe]

@vivekpatani thank you very much for the doc. I will take a look.

@gyuho @tangcong @jingyih @ptabor this topic was discussed on August 27th community meeting https://youtu.be/CuBQ21Du0hg. @vivekpatani now has added a design doc as requested in the meeting, would you please help to take a look?

[tangcong]

thanks @vivekpatani. Your proposal looks similar to my early proposal. It is a further supplement. Maybe we can work together. I think it is necessary to add API to support add/update/delete qos rules. I have simply implemented the QoS API and etcdctl command before. I haven’t finished the POC because I’m busy with other things.

[vivekpatani]

@wenjiaswe @tangcong @gyuho @jingyih @ptabor have submitted the PR for review, thanks.^

[tangcong]

@wenjiaswe @tangcong @gyuho @jingyih @ptabor have submitted the PR for review, thanks.^

thanks. I also have finished the QoS POC. it has following advantages:

● Implement QoS API,rules can be adjusted dynamically.
● Support multiple rate limiter and rule type.
● Users can customize rules simply through conditional expressions.

more detail info and implementation, please see pr 12290.
@wenjiaswe @tangcong @gyuho @jingyih @ptabor

[gyuho]

For reference

https://github.com/etcd-io/etcd/pull/12290#issuecomment-697060195

Thanks all for the POC and design docs.

We believe such full-featured QoS/QPS layer will be very hard to maintain compatibility with other projects (or code), or even confuse users. We'd like to keep etcd as minimal as possible, rather than adding another complexity to maintain. grpc-proxy is another example. Not everybody is using it but its lack of maturity has been slowing down or blocking the core development.

Unless this can be simply implemented as a gRPC server middleware, I believe this is not the problem etcd is designed to solve. It's already pretty easy to embed rate limiter on client-side however you want, whereas built-in rate limiter will never satisfy all the requirements for all use cases.