Frameworkbenchmarks: Rules clarification on retrieving World object

[bgrainger]

I'm requesting a rules clarification for Single Database Query https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Framework-Tests-Overview#single-database-query and Multiple Database Queries.

The wiki states:

For every request, a single row from a World table must be retrieved from a database table.
...
The row should be converted to an object using an object-relational mapping (ORM) tool.
...
Use of an in-memory cache of World objects or rows by the application is not permitted.

I have noticed two different approaches to achieving this. The first approach selects the whole row, e.g., with SELECT id, randomnumber from WORLD where id=$1 then reads both returned fields and converts them to an object. Some examples:

https://github.com/TechEmpower/FrameworkBenchmarks/blob/a94705893dd6a0e7e8312c79d0e4dff7ba8eb990/frameworks/Java/officefloor/src/woof_benchmark_tpr/src/main/java/net/officefloor/benchmark/Logic.java#L47-L52

https://github.com/TechEmpower/FrameworkBenchmarks/blob/a94705893dd6a0e7e8312c79d0e4dff7ba8eb990/frameworks/Dart/start/server.dart#L345-L347

https://github.com/TechEmpower/FrameworkBenchmarks/blob/a94705893dd6a0e7e8312c79d0e4dff7ba8eb990/frameworks/PHP/fat-free/index.php#L38-L42

The second approach selects just the randomnumber field and relies on id being the same as the query parameter. Some examples:

https://github.com/TechEmpower/FrameworkBenchmarks/blob/a94705893dd6a0e7e8312c79d0e4dff7ba8eb990/frameworks/C%2B%2B/drogon/drogon_benchmark/controllers/DbCtrlRaw.cc#L22-L29

https://github.com/TechEmpower/FrameworkBenchmarks/blob/a94705893dd6a0e7e8312c79d0e4dff7ba8eb990/frameworks/PHP/swoole/swoole-server.php#L33-L41

https://github.com/TechEmpower/FrameworkBenchmarks/blob/a94705893dd6a0e7e8312c79d0e4dff7ba8eb990/frameworks/D/hunt/mmap/DemoProcessor.d#L169-L173

The second approach seems like it would substantially reduce the required bandwidth from the database to the application (returning just one field, not two) and require less deserialisation; I haven't tested both approaches side by side in the same application to prove there's a performance difference, but I assume it could have a material impact on the results.

Additionally, the second approach feels like it's violating the spirit of the rules: it is not retrieving "a single row", the (whole) row is not being converted, and (arguably) in-memory caching (of the ID) is being used.

Can there be an official clarification of whether this approach to the problem is permitted or not? (If it's disallowed, several dozen frameworks may need to be updated to comply.)

[nbrady-techempower]

Hey @bgrainger,

You're right and I've updated the rules for test #2 to clarify that reading just the random number is not acceptable. We should identify the implementations that are doing that here and try and get them addressed.

There may have been some confusion, since the updates test does allow it:

6. At least the `randomNumber` field must be read from the database result set.

Thanks for pointing that out!

[bhauer]

@bgrainger Thanks for doing the legwork necessary to identify this and for bringing it to our attention.

I agree with your assessment that selecting only the randomNumber column does not seem to align well with the spirit of the test. However, as @nbrady-techempower points out, we had previously conceded in the Updates test that reading just the randomNumber field was sufficient.

For the Multi-queries test, Nate has decided (and I agree) that we should stick with the original vision of requiring that the full row (including the id column) is read.

Some background: When we originally conceived of our database tests, we had assumed an ORM would be used. When we broadened the database tests to allow non-ORM implementations (which was a very long time ago), we should have been more clear about expectations. It's no surprise that a lack of specificity led to (unintended) flexibility.

[joanhey]

I don't see it in the requirements now.
https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Framework-Tests-Overview

[nbrady-techempower]

I don't see it in the requirements now.

Which requirement is missing?

For every request, a single row from a World table must be retrieved from a database table. It is not acceptable to read just the randomNumber.

This is in the single requirements test and also in the multi requirements test is:

Each row must be selected randomly in the same fashion as the single database query test (Test #2 above)

I might be missing what you're talking about. It is Monday morning here. 🥱

[nbrady-techempower]

I think we can close this, now. Thanks all!