Frameworkbenchmarks: Drogon core benchmarks

[fafhrd91]

i noticed drogon-core does not use load results into memory and does not use json serializer.
as i remember this is against rules.

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/C%2B%2B/drogon/drogon_benchmark/controllers/QueriesCtrlRaw.cc#L40

[fafhrd91]

for queries as well,

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/C%2B%2B/drogon/drogon_benchmark/controllers/DbCtrlRaw.cc#L29

[fafhrd91]

updates bench

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/C%2B%2B/drogon/drogon_benchmark/controllers/UpdatesCtrlRaw.cc#L44

it also does not create World objects which violates rules
https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/C%2B%2B/drogon/drogon_benchmark/controllers/UpdatesCtrlRaw.cc#L100

[fafhrd91]

I expect some respect to other developers. This cheats are very unpleasant.

[nbrady-techempower]

@an-tao

[an-tao]

@nbrady-techempower Is there such a rule in db tests (must be serialized with a json serializer)?
I can only find the rule in the JSON Serialization test:

A JSON serializer must be used to convert the object to JSON.

But there is no such an explicit rule in the four database tests.

@fafhrd91 , Can you find out the rules of database tests as clear as the serialization rule in the JSON Serialization test? If not, this can only mean that we have different understandings of the rules, and the term “cheats” is disrespectful to me. If you are right, I will modify my tests, and I hope TFB will make this rule clear.

BTW In fact, I can modify it now, and I don't think that this change will have any big impact on the result (comparing QPS of json test and DB test will get this conclusion), but I will only modify it after the TFB clear this rule because I didn't cheat, this must be clearly stated.

[an-tao]

updates bench

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/C%2B%2B/drogon/drogon_benchmark/controllers/UpdatesCtrlRaw.cc#L44

it also does not create World objects which violates rules
https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/C%2B%2B/drogon/drogon_benchmark/controllers/UpdatesCtrlRaw.cc#L100

This test is in raw mode(that is, tests without an ORM), there is no need to create a world object.
https://github.com/TechEmpower/FrameworkBenchmarks/blob/968971a96045f0d10dbf3fb61694d2733d06759b/frameworks/C%2B%2B/drogon/benchmark_config.json#L40

[fafhrd91]

5. Database updates

1. The request handler must retrieve a set of World objects, equal in count to the queries parameter, from the World database table.
2. Each World object must have its randomNumber field updated to a new random integer between 1 and 10000.
3. Each World object must be persisted to the database with its new randomNumber value.
4. Each World object must be added to a list or array.
5. The list or array must be serialized to JSON and sent as a response. (I guess you invented new json serializer, and its name sprintf, fancy)

p.s. for sure, I disrespect anyone who cheats.

[an-tao]

A JSON serializer must be used to convert the object to JSON. (In JSON Serialization test)
The list or array must be serialized to JSON and sent as a response. (In updates test)

Can you tell the difference between verbs and nouns? Don't you see that the second constraint is weaker than the first one? (It allows serialization using any method).
If they mean the same thing, why not use the same expression? If I'm cheating, why not use sprintf in the JSON Serialization test?
Do you know what the Serialize virtual function does in MFC? You don't have to use a serializer to serialize an object, understand?

1. Database updates

xii. For raw tests (that is, tests without an ORM), each updated row must receive a unique new randomNumber value. It is not acceptable to change the randomNumber value of all rows to the same random number using an UPDATE ... WHERE id IN (...) clause.

Raw mode is allowed in this test, If ORM is not used, where can I get a World object which is the 'O' in ORM?

[nbrady-techempower]

@an-tao Raw for ORMs simply means you’re not using a traditional ORM, but you still have to map the result of the query to an object per the test rules. And update that object in the updates test before persisting to the database.

I see the difference in wording for JSON serializer vs serializing but I’m not sure that was intentional. Nonetheless, some people have built their own very simple serializers and we’ve allowed that. I will get the wording on this cleared up next week. Most of our team is on vacation this week.

I understand that this feels like a competition to most people, but the focus for us with these benchmarks is to help build stronger frameworks together and for a community of developers to help each other improve. With that in mind, please be respectful of each other here. I know people are trying to find the best edges possible and it’s our job to remove the gray areas from the rules.

Also, this is my fault for letting this slip through. I usually pull in other reviewers because C/C++ makes me dizzy. ;)

[an-tao]

@nbrady-techempower Thanks for your clarification.
I want to know do I need to map the results to World objects in all Database tests or just in the Updates test?
below is the Single Database Query test requirements:

vi. The row should be converted to an object using an object-relational mapping (ORM) tool. Tests that do not use an ORM will be classified as "raw" meaning they use the platform's raw database connectivity.
vii. The object (or database row, if an ORM is not used) must be serialized to JSON.

It seems that if ORM is not used, I just need to serialize the database row (not the World object) to JSON.

And I will use the jsoncpp lib for serialization no matter how you clarify the rule, the sprintf comes from an optimization that has little effect (It is easy to know that the database side is the bottleneck). I was just too lazy to change it back.

I hope the developers here are friendly, and you know, I also questioned other frameworks, but I was very careful in my wording. I may have misunderstood the rules, but I will never cheat on purpose. To be honest, I don't care what others say, as long as you are fair, I will continue to submit my tests here. Thanks for your work!

[nbrady-techempower]

@an-tao I think we’d want that to be an object as well for consistency. When I get back next Monday I’ll check to make sure that all frameworks are already doing that, clarify with the rest of the team, and update the rules.

And always feel free to point out any frameworks that have inconsistencies. Often times it’s just a misunderstanding and the test maintainers fix when asked. Thanks!

[an-tao]

@nbrady-techempower Thanks! I'll fix it.

[bhauer]

I see the difference in wording for JSON serializer vs serializing but I’m not sure that was intentional.

Indeed, any difference in meaning here was not intentional. The original expectation was that a JSON serialization library or component would be used in all tests that render to JSON. Implementing hand-crafted rendering of JSON within the test implementation was not what we had in mind originally.

I understand that this feels like a competition to most people, but the focus for us with these benchmarks is to help build stronger frameworks together and for a community of developers to help each other improve. With that in mind, please be respectful of each other here. I know people are trying to find the best edges possible and it’s our job to remove the gray areas from the rules.

This is a very important point and thank you for making it!