Frameworkbenchmarks: Rules clarification about Kotlin/hexagon-jetty-mysql and the Database Updates test

[zloster]

The current new champion of the Database Updates test in Round 14 is returning dirty data for the client requests and deferring the updates execution after the result is sent to the client. The data is dirty because when the result is returned to the client the updates haven't hit the DB yet: they are executed asynchronously later.
The code is in BenchmarkStorage.kt:

• https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Kotlin/hexagon/src/main/kotlin/co/there4/hexagon/BenchmarkStorage.kt#L28

• https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Kotlin/hexagon/src/main/kotlin/co/there4/hexagon/BenchmarkStorage.kt#L146-L158

• ExecutorService documentation - https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ExecutorService.html

This is not explicitly forbidden by the rules but I don't think this is following the spirit of the Database Updates test. So I would like to hear what TFB team thinks about this.

[nbrady-techempower]

@zloster Yes, agreed.

@jaguililla Can you speak to this?

[jaguililla]

Right now, the queries are executed, results modified and update is done in a batch in a worker thread. The results are good (the returned worlds are the updated ones) but the update is finished after the connection with the client is closed. So @zloster is right.

You can delete the results from the round if you think is not fair and I will change the code for the next round.

Sorry if this caused any trouble, but I though it was allowed.

[bhauer]

@jaguililla Thanks for the explanation. As @zloster points out, this particular case is not addressed in the requirements, but he is correct that the spirit of the test is for the updates to have been committed prior to returning the results.

I can imagine a pathological (or maybe straight-up diabolical) case where an implementation queues all of the updates created by the entire test run and commits them after the test is completed. Obviously that's not what is happening here, but we'd rather not have that gray area.

We'll clarify the requirements on this.

Looking forward to the revised version for Round 15. I am confident Hexagon will still do very well.

[bhauer]

I've just posted a clarification to the requirements that includes the following:

All updates should be executed prior to generating the HTTP response.

If anyone has a rewording that communicates the notion better, I'm happy to change it. Thanks!

[jaguililla]

I would change executed by completed and generating by returning... Regarding my results, could you please delete them (only the data updates)? thank you!

[bhauer]

Thanks! Will do.

[bhauer]

Thanks again @jaguililla. Rewording of requirement done and Hexagon results for Updates (and updates only) have been hidden.

[jaguililla]

Thank you very much and sorry for the inconvenience!