Hyper: Make sleep_on_error timeout configurable, enable it by default + document it

[klausi]

Follow-up to #1358 .

Now that we have internal IO error handling when a Hyper server is under heavy load (or DoS attack) we should make this configurable and enabled by default for a better developer experience.

• [ ] Make sleep_on_error timeout configurable (currently hard-coded to 10ms)
• [ ] Enable sleep_on_error per default
• [ ] Document it. https://docs.rs/hyper/0.11.21/hyper/server/struct.Http.html#method.sleep_on_errors is a pretty good start, should we say more about it? Maybe a footnote somewhere on https://hyper.rs/ ?

[BatmanAoD]

Disclaimer: I am not a web developer, either frontend or backend.

Swallowing errors by default seems like a terrible idea to me. This is the best way to make unreliable and difficult to debug software. In fact, JavaScript's "swallow errors by default" design philosophy is perhaps the primary reason why I'm not a web developer; I was hoping that a full-stack Rust framework (with WebAssembly on the frontend) would make the idea more appealing to me, because Rust seems to have precisely the opposite design philosophy.

However, it does seem that for a DoS attack or heavy load, the server needs to be able to ignore requests for some short period of time. But the user should still be able to do something with the specific errors that are generated.

So my suggestion would be to permit users to specify callbacks to invoke when specific errors occur, or possibly a generic callback invoked when any ignored error occurs.

A reasonable default behavior might be to automatically set up these callbacks to simply log the error. That would accomplish the goal of making robust server setup easier, while also ensuring that there's still an easy way to discover and investigate possible bugs--or, in a production environment, actual attempted DoS attacks.

[avkonst]

Completely agree with the above comment.

What is the behavior, when timeout is 0 ms?

I consider it totally appropriate to reject new connections and free related resources immediately if they can not be accepted due to lack of handlers allowed. It is unlikely that there will be anyone allowed in 10ms (or whatever small interval), but context for timeouts and inbound connections will be held in RAM.

Upper layer should be notified about such rejections.

[seanmonstar]

1488 was just merged, and included some changes that touch this. Here's how it works now:

• sleep_on_errors is set to be default on
• When enabled, it will log at the error! level about the failures, so users can still see bad things are happening.
• The timeout duration itself isn't configurable, it's just 1 second.
• It can be turned off, and a completely custom strategy can be created by a user just by providing their own "incoming" stream of accepted connections.