Frameworkbenchmarks: Changes / Updates for Upcoming Round

[nbrady-techempower]

I'll be editing this pinned issue to link to other issues or pull requests that may affect tests for this round.

We are working toward Round 18 now. Official results have been captured and will be available on the results website soon.

4291 - Responses including cached date header

4493 - Limit the amount of memory the framework container can use

4601 - Cached queries tests were using the wrong levels

4622 - Stricter Multiple Query test verification

4636 - Enable accelerated networking in Azure

4642 - PostgreSQL bumped to v11

4834 - Actix fortunes explanation of results

We've also clarified some rules:

• Use of single update statements that affect multiple rows is acceptable
• Use of caching/memoization within or on top of JSON serializers is not allowed.

Tests that were in violation of the rules when captured:

• #4842 #4845 onyx multi-query & updates
• #4391 perfect-mysql updates

[mkurz]

Any estimation when you will start the round?
BTW: https://tfb-status.techempower.com/ is down...

[nbrady-techempower]

We'd like to pull the next round from continuous runs this week. Looks like there was an outage at the office this weekend; we'll get things back online Monday morning.

[mkurz]

@nbrady-techempower Thanks. Can you please merge #4671 so it makes it into the next round? Thanks!

[volyrique]

This probably got lost before, but I found another difference between the Terraform scripts and the Azure portal defaults - the OS disk type is "Standard HDD" instead of "Standard SSD". Are you going to change that too in the upcoming Azure runs?

[nbrady-techempower]

@volyrique I want to point out that we're not necessarily shooting for the most optimized environment for azure; I think some of the point to the azure tests is to see how tests perform in a less optimized environment. With that being said, I believe we're using the temp storage drive in azure which is SSD (https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sql/virtual-machines-windows-sql-performance#temporary-disk) I'll make sure to get the final specs posted soon.

[fafhrd91]

seems tfb is stuck

[msmith-techempower]

seems tfb is stuck

I checked the logs - it looks like it's running fine to me.

[fafhrd91]

maybe something with cache?
https://tfb-status.techempower.com/results/019d3fd5-b7e0-41b2-b401-c42a382abf85 shows 16 completed for last 5 hours

[msmith-techempower]

It has only been running for 2 hours.

It is still going and I see no reason to think anything is working improperly based on the logs and expected duration.

[nbrady-techempower]

It does appear to be stuck; we'll take a look at it.

[volyrique]

I want to point out that we're not necessarily shooting for the most optimized environment for azure...

@nbrady-techempower Sure, my main point was about reproducibility - I don't mind if a particular Azure option that may improve performance is not used, as long as it is clearly documented (especially if it is not the same as the default), so that people who try to reproduce the results are not left wondering why their data differs so much.

[jsongte]

I've added some more info on the wiki https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Environment

Once I finalized the azure terraform setups and the scripts I'll make a reference of that on the wiki as well, since all the details can be found there.

[wangkaish]

Seems Azure is stuck!

[nbrady-techempower]

We may be pushing this off for just a little bit until we can resolve the Azure issues. Hopefully this week.

[nbrady-techempower]

Still working on Azure. We're hoping to have Round 18 published by the end of next week, barring any more unforeseen issues (there are always some!)

Feel free to continue making adjustments, but I wouldn't make any experimental changes as they may get captured in official round results.

[zloster]

Another note regarding Java versions - most images are using the Docker openjdk image. For quite long time it was based on the OpenJDK code in the Debian repositories. Very recently they've switched to AdoptOpenJDK builds.
11.0.3 tag is republished with the AdoptOpenJDK. This is not visible yet in the TravicCI builds.

[volyrique]

It seems that the CVE fixes had a pretty significant effect on the JSON serialization, single query, and fortunes results in the latest Citrine run. I don't think the framework standings changed much, but the maximum requests per second dropped by 22.04%, 16.75%, and 22.05% respectively.

[zloster]

Another important point about the MDS mitigations - it seems that the results are with activated hyper-threading.

h2o: Number of processors: 28

Search for the text above here. The Xeon Gold 5120 is having 14 physical cores.

I suspect there will be a run with hyper-threading disabled.

[volyrique]

Apparently other projects suffered from regressions as well.

[msmith-techempower]

It seems that the CVE fixes had a pretty significant effect on the JSON serialization, single query, and fortunes results in the latest Citrine run. I don't think the framework standings changed much, but the maximum requests per second dropped by 22.04%, 16.75%, and 22.05% respectively.

Yikes.

[joanhey]

In the plaintext bench, the difference is near to 50% in some frameworks/platforms.

fw | before | after
|---|----:|-----:|
| h2o | 5,724,772 | 3,085,341
| nginx | 3,668,042 | 2,267,821
| php-ngx | 2,679,720 | 1,746,828
| php | 282,415 | 172,840

I think like @zloster : run with hyper-threading disabled.
Info:
https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/MDS
https://www.phoronix.com/scan.php?page=news_item&px=MDS-Zombieload-Initial-Impact

[bhauer]

If everything goes as planned, Round 18 will be published next week (approximately July 8). It will be based on the most recent Azure run and the Citrine run that used the same commit. These were captured prior to application of the CVE-2019-1147x mitigations.

• Citrine: https://tfb-status.techempower.com/results/52587667-bee1-4b9b-b34d-929b616d5e4c
• Azure: https://tfb-status.techempower.com/results/9ce91254-04d7-4455-9742-aa6a1f673e86

[fafhrd91]

@bhauer those runs still contain "onyx" framework, which violates db bench requirenments

[nbrady-techempower]

@fafhrd91 See the original post. Anything that's listed as violating the rules won't be displayed on the official results page.

[nbrady-techempower]

For those just following along here, Round 18 has been released. https://www.techempower.com/blog/2019/07/09/framework-benchmarks-round-18/

[bhauer]

@nbrady-techempower Thanks for tracking this for the round. Should we close this one and start a similar tracking for Round 19?

[nbrady-techempower]

@bhauer Absolutely; will get one set up.