Frameworkbenchmarks: Changes / Updates / Dates for Round 20

[nbrady-techempower]

_We will be locking PRs for Round 20 on 11/11_

I'll be editing this pinned issue to link to other issues or pull requests that may affect tests for this round.

Changes

• [ ] #5648 We will be enforcing the size of route names for tests for Round 20. (i.e. the plaintext route must be at least the same length as /plaintext and cannot be /p.

Request for Comments

• #5673 Server Response Header Change
• #5844 Out-of-process Caches

Previous Round Information

See: https://github.com/TechEmpower/FrameworkBenchmarks/issues/4966 for details about the previous round released on 5/28/2020

[nbrady-techempower]

* Placeholder *

[bhauer]

Round 20 will be adding visualization of the cached queries test type. In fact, it's very likely that we'll push this visualization out soon since Round 19 already has some results, and doing so will make these render for continuous runs.

If your framework doesn't yet include an implementation of this new test type, please consider adding one!

[volyrique]

The visualization of the cached queries results is a great addition, especially given that the test has been defined (and the first implementations have appeared) 3 years ago. I think the lesson here is that there won't be a significant number of implementations of a future test type unless the results are being shown properly on the results dashboard.

There is just one small issue right now - while a partial set of cached queries results (i.e. for a run that hasn't completed yet) is shown properly, the visualization doesn't work for runs that have completed - I just get "No data available for this test".

As for the changes for the next round, are there any plans to upgrade to Ubuntu 20.04? Updating the kernel version has had a significant performance impact in the past.

[michaelhixson]

There is just one small issue right now - while a partial set of cached queries results (i.e. for a run that hasn't completed yet) is shown properly, the visualization doesn't work for runs that have completed - I just get "No data available for this test".

I think this is a side effect of renaming the test type from cached_query to cached-query. Runs that started after this change in the toolset will have visualizable data, runs from before that will not. The TFB website only understands the new name. For Round 19, the new name was manually edited into the website's copy of the results.

[nbrady-techempower]

As for the changes for the next round, are there any plans to upgrade to Ubuntu 20.04? Updating the kernel version has had a significant performance impact in the past.

I don't have a strong opinion on this. Looking at the ubuntu release schedule, 20.04 is still in "Hardware and Maintenance" mode until 2022 and as far as I understand it, the underlying kernel would be the same version, right? We could probably do this will little effort if it made sense to do.

[volyrique]

... the underlying kernel would be the same version, right?

By default Ubuntu Server 18.04 uses kernel version 4.15 (there's also the HWE stack; I am not sure how it works, but it might be an option) and 20.04 gets 5.4, so definitely not. I think it is reasonable to stick with LTS releases, so upgrading now means that this question won't come up for another 2 years. Obviously I can't say for certain what kind of impact a newer kernel would have on the benchmarks - it might even cause regressions. However, I have the feeling that generally speaking the negative effects of the security countermeasures tend to be ameliorated in newer versions and there's always optimization work in the network stack, for example.

[nbrady-techempower]

Hello everyone!

Some Round 20 news:

We will be locking PRs for Round 20 on 11/11 so that we can kick off official round runs by that weekend. Please have your PRs in before then if you'd like to be sure that they make it into the official rounds.

About two weeks after Round 20 is posted, we will be moving to the new Toolset. Nothing will be required from framework maintainers for their existing frameworks to use the new toolset. We have a tool that will automagically replace the current benchmark configs with the new config. There will be plenty of questions during this transition phase, and we'll be here to help. See #6064 for more information about this.

Around this time, we will be taking down Citrine to do some kernel updates and get the new toolset working properly. Exciting times ahead!

[nbrady-techempower]

Hey everyone! Quick update here. We're still working on kicking off the Azure run, since it's been some time since we've noticed an automated run going. We'll have to figure out what's going on there and in the meantime, you have a few extra days to get some PR's in. Let's go with locking PR's by Thursday 11/19 so we can try to capture round 20 over next weekend.

[joanhey]

Add a message to tfb-status with some advance notice before PR submissions are closed for the round (From Round 19)

[nbrady-techempower]

@joanhey you're absolutely right. I forgot to do this. Hopefully nobody is in a last minute rush to get things in and they've at least looked at this issue in the last 20 days when I first posted that comment. I'll put up a notice now. We're planning on kicking off the run on Tuesday. Had some trouble with Azure and mongo.

[nbrady-techempower]

Due to some technical difficulties and a busy client schedule plus the holidays, we're going to postpone Round 20 until after the holiday break. I'll change the notice on TFB-Status. Let's set the last day of PRs 12/28.

[nbrady-techempower]

Without CI right now, I have to go through these PRs and test locally so please get your PRs in early and make sure to test locally before opening. Thanks!