Frameworkbenchmarks: Review and revise low-concurrency levels on ServerCentral

[bhauer]

I just updated the results web site to make it easier to immediately visualize the lowest and highest performance recorded across the range of concurrency levels we measure.

We have three "bands" of concurrency levels we use during measurement:

• On JSON, single-database query, and fortunes, we measure at concurrency levels 8, 16, 32, 64, 128, and 256.
• On multi-query and updates, we use 256-concurrency only and vary the number of queries.
• On plaintext, we use a higher-concurrency band of 256, 1024, 4096, and 16384.

I propose reviewing and revising the low-concurrency levels in light of the number of cores in the ServerCentral app server (80 HT cores). The concurrency levels were specified when we were testing on our in-house i7 workstations, but have not changed since.

The highlighting as illustrated below leads to some quick thoughts:

1. We should drop the 8 concurrency level. It is the lowest-performing tier on the vast majority of frameworks and doesn't provide much illumination even when it's not the lowest-performing tier.

2. Several frameworks are showing maximum performance at the 256 concurrency level, suggesting we should at least add a 512 level.

3. For the reason of total suite execution time, I'd like to keep the total number of concurrency tiers constrained, so I don't want to add tiers without removing tiers. With that in mind, if we decide a tier above 512 is warranted, I would propose dropping both 8 and 16 to make room for 512 and 1024.

4. We could also consider increasing the singular concurrency level of the multi-query and updates tests to 512 or 1024 depending on what behavior we see across the board from the above changes.

[bhauer]

I've been informed by @nbrady-techempower that the first Preview of Round 15 will likely swap the 8 level out and replace it with 512. We'll see how that ends up looking.

[methane]

How about "Cloud" hardware?
I think concurrency same to "Physical" is too high for "Cloud" hardware.
Maybe, 1/4 is good.

[nbrady-techempower]

@methane thanks! We will review for Round 15. We did not test on "Cloud" for Round 14. Relying on contributors in that space to provide hardware.

[zloster]

When deciding to bump the default load for the tests please note the following change:
https://github.com/TechEmpower/FrameworkBenchmarks/pull/2586#discussion_r111472008
The ability to choose the applied load is no more. IMO this change is making the life harder for a contributor working in a VM: Yes, there is a workaround but it's not as convenient as the removed options for concurrency-levels and query-levels.

[nbrady-techempower]

@zloster concurrency and query levels are still pulled in from the benchmark.cfg
https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/benchmark.cfg.example#L19

[zloster]

@nbrady-techempower I didn't know about this. Thank you.

[msmith-techempower]

Work has been committed against this effort - can this issue be closed?

[nbrady-techempower]

I think this can be closed