Tikv: trigger/list/delete fail points through HTTP

[siddontang]

Feature Request

Is your feature request related to a problem? Please describe:

Now we have already installed lots of fail points and used them in tests, but only in uint or local integrated tests, we can't use them in Schrodinger or at runtime.

Describe the feature you'd like:

We can build a release version with fail support, add HTTP handlers to help us trigger/list/delete fail points.

Refer https://github.com/etcd-io/gofail#http-endpoint

[kennytm]

Seems this issue is more suitable for https://github.com/pingcap/fail-rs?

TODO

Triggering a fail point via the HTTP API is planned but not implemented yet.

[siddontang]

@kennytm

I don't like the way of using an HTTP environment to start a fail service, especially in Schrodinger.

Another reason is that I don't like using another port for fail separately, we should use the same HTTP port of TiKV, but unlike Go, there is no standard HTTP lib so we have to write our HTTP logic directly in TiKV. 😭

[liufuyang]

Is this issue easy to solve? Can I try give some help? :)

[BusyJay]

Yes, it's quite easy. What needs to do is to setup the failpoint registry when TiKV starts, and then provide HTTP interface at status server.

https://github.com/tikv/tikv/blob/3f10327e648b4c1c6935154f8bca4e2abd894419/src/server/status_server.rs#L65-L73

[liufuyang]

Okay I seem to have some time to look at this issue now.
The question I have is, how would a user uses the http failpoint when finished? So one can use it like this?:

# Use PUT to add
curl SERVER_IP:PORT/failpoints -XPUT -d'{"name": "some_name", "actions": "some actions"}' 

# Use GET to list
curl SERVER_IP:PORT/failpoints  

# Use Delete to delete
curl SERVER_IP:PORT/failpoints -XDELETE -d'{"name": "some_name"}'

[BusyJay]

How about put the name in the URL part?

[liufuyang]

If that is easier when using, I can try that as well :)
I have now made some demo code to try out fail-rs, and it is using json payload. https://github.com/liufuyang/fail-rs-http-demo Will try to see whether I can change it to URL part

[liufuyang]

So I made some demo http code here https://github.com/liufuyang/fail-rs-http-demo
Do you mind go take a look at it to see whether that is the way we would like to implement here for tikv?