Frameworkbenchmarks: Is it correct to hard-code the Content-Length ?

[joanhey]

Looking code from others frameworks, I found that:
https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/C%2B%2B/ulib/src/fortune.h#L151-L182
https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/C%2B%2B/ulib/src/json.usp#L25-L36

Perhaps others frameworks do it too.

In the Requirements I don't see anything, only that:
The response content length should be approximately 28 bytes. (for json)

And in all tests:
The response headers must include either Content-Length or Transfer-Encoding.

Also it is not the same, check the body length to add it in each request, to use hard-coded or Transfer-Encoding: chunked.

In this test is easy to hard-code the Content-Length, but not in a realistic app.

[volyrique]

The JSON test requirements state quite clearly that the object must be serialized every time, so to me it doesn't sound right either to assume the body length (How can someone know the length of the JSON data if it hasn't been serialized yet?!). In addition to that, it looks like even the step of preparing the headers is done only once (refer to the check here) in a buffer that is subsequently shared with other requests, which is forbidden even by the plaintext test requirements (the most relaxed ones). @stefanocasazza Any opinion?

[nbrady-techempower]

I don't think we're clear enough on the rules for headers. For instance, in the plaintext requirements we do state:

This test is not intended to exercise the allocation of memory or instantiation of objects. Therefore it is acceptable but not required to re-use a single buffer for the response text (Hello, World). However, the rest of the response must be fully composed on the spot. It is not acceptable to store the entire payload of the response, headers inclusive, as a pre-rendered buffer.

@bhauer @msmith-techempower @michaelhixson I'm thinking maybe we should be more specific about headers in the general requirements. We currently have:

All test types require Server and Date HTTP response headers. We expect the Server header to be whatever is normal for the platform or framework. If the framework does not normally provide a Server response header, we nevertheless require that you provide one as this roughly normalizes network load across all implementations. For Date, we expect that the rendered date be accurate. However, it does not need to be rendered from the system clock to a byte buffer for each request. Re-rendering once per second is an acceptable tactical optimization (and is an optimization baked into many frameworks).

Thoughts?

[msmith-techempower]

We are very specific about date, but I suppose there is room to argue on the server header.

I believe many implementations probably are doing the work to produce the correct content length for the current tests, but obviously there will be cheaters who think counting bytes every time is going to cost them some meaningful performance numbers.

Personally, I do not believe that top contenders will eke much from such optimizations, and have not rigorously enforced implementations must calculate their content length on every request. Sure, a real-world framework would do this for the developer, and no developer is likely going to manually enter the content length of a payload in practice. I just do not suspect it will matter much at the top.

So, I can go either way, but I do not think it is worth much thought.

[bhauer]

Without having spent much time considering all options here, my gut reaction is:

1. I would expect that a majority of frameworks would calculate the length based on the length of the response. As an application developer, I would not expect my framework to require that I determine the length of my content myself.
2. If a framework does not calculate the length of the response or provide content chunking, it is _acceptable_ for the test implementation to hard-code the length as long as that is considered standard practice for the framework (which incidentally makes a strong argument for classifying the framework as a platform or micro-framework). If, on the other hand, the test implementation hard-codes the length and this is considered exceptional for the framework, the implementation should be marked as "stripped."

[xfix]

My personal opinion is that frameworks must not hardcode specific values in JSON and Fortune benchmarks - it makes no sense, the output values don't have known length. In JSON benchmark, there is a requirement of value being serialized. This implies dynamic content, where you cannot just go send "Content-Length: 27\r\n" without computing length of JSON.

In fortune benchmark, the requirements are clear: "the code cannot have foreknowledge of the table's size". Hardcoding Content-Length is clearly foreknowledge of the table's size.

Explicitly setting Content-Length header in JSON and Fortune benchmarks sounds fine to me, but you need to base Content-Length out of output, you cannot simply hard-code the value.

I don't have any particular opinion about Content-Length in plaintext benchmark. The benchmark is so simple that it could be feasibly considered to be constant value, where I think assumptions like could be considered acceptable.

[bhauer]

I feel mostly in agreement with you @xfix. Frankly, I find an optimization such as hard-coding the content-length to be an extreme measure that makes me instinctively suspicious.

I feel in agreement with respect to Fortunes. It has always been intended to approximate a dynamic scenario where the application code has no foreknowledge of the data being processed. I am 99% confident in suggesting that we mark any tests that hard-code response length for Fortunes as "stripped."

I am in softer agreement, but still agreement, for the JSON test type. It too is intended to approximate a dynamic scenario.