Frameworkbenchmarks: Responses including cached Date header

[an-tao]

I did some testing in the benchmark project and I found that in the tests 'Json' and 'Plaintext', the response 'Date' header of some frameworks is always the same as the first response. It hints that the response entire text string including headers and the body is cached in memory for every request. According to the benchmark document, this seems to be illegal. But the framework that uses caching occupies a high position in the rankings. Am I wrong with the test rules? Does the test client check the Date header to ensure that each response is generated separately?

[nbrady-techempower]

It would be against the test rules to cache the response. Would you mind pointing out the frameworks you believe to be doing that so we can investigate further? Thanks!

[an-tao]

I think the ulib framework is caching the response, maybe I was wrong, but when I followed it's dockerfile to run the test, I got responses by the curl command as follows:

As you can see, all the Date headers are the same. Do you think that what I suspect is reasonable? If I am wrong, please correct me. Thank you!

[dividedbynil]

@stefanocasazza

[stefanocasazza]

I simple follow as other frameworks to not update the Date header for performance reason. So far it is not a requirement to have a Date header timed aligned

[fafhrd91]

Rust top performing frameworks do update date header.

[nbrady-techempower]

Per the test rules, the response must be composed on the spot. This should include the proper date, though we currently do not check that the date header is different from request to request, just that it's present. I'll look to add this as part of the verification step before the next round, marking tests that don't fix this in the future as failed.

@bhauer @msmith-techempower @michaelhixson Anything to add here?

[zloster]

@nbrady-techempower Some test implementation are using time-expiration caching for the date header. They refresh the date header once a second or so. Out my head I can recall:

• https://github.com/TechEmpower/FrameworkBenchmarks/blob/d157dfc58f2cae4c0f57ba6eddfca0b26fd73525/frameworks/Java/netty/src/main/java/hello/HelloServerHandler.java#L78-L87
• https://github.com/TechEmpower/FrameworkBenchmarks/blob/d157dfc58f2cae4c0f57ba6eddfca0b26fd73525/frameworks/Java/spring-webflux/src/main/java/benchmark/DateHandler.java#L11-L19

The HTTP date header is with an 1 second granularity - https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.3.1 So caching for a second should not have any big practical implications. Maybe the rules should be amended if this behaviour is allowed.

[nbrady-techempower]

@zloster I think we're probably ok with that but I will double check. Whatever verification we put into the toolset likely wouldn't flag that anyway.

[zloster]

@nbrady-techempower I agree.
Just wanted to make sure this fine detail don't get lost.

[boazsegev]

F.Y.I,

The facil.io framework also caches the date string using a single second granularity (not the whole header, but still).

I was assuming that's allowed, since the result doesn't effect the response (it should always show the correct date/time)...

[benaadams]

I assume the General requirements have been updated for this:

1. ... For Date, we expect that the rendered date be accurate. However, it does not need to be rendered from the system clock to a byte buffer for each request. Re-rendering once per second is an acceptable tactical optimization (and is an optimization baked into many frameworks).

[nbrady-techempower]

Yeah just to confirm, we're ok with re-rendering/caching just the date header every one second. Date headers that remain unchanged will cause tests to fail in upcoming rounds.

[nbrady-techempower]

@helyho Java/voovan needs to be updated, as it will fail before the next round because of this issue.

@smthing Same with Java/smart-socket

[nbrady-techempower]

This should be resolved with #4452