Frameworkbenchmarks: Tests that need updating before next round

[nbrady-techempower]

This is a living list of tests that need to be updated or removed before we record our next official round results. I'll do my best to ping the most recent maintainers as well as list the problem. Tests that aren't updated/fixed prior to the next round will be removed to free up resources and speed up the testing cycle.

Tests that are broken

C++/ffead-cpp
Crystal/spider-gazelle
Dart/hunt-dmd
Go/chi-*
Go/revel
Haskell/servant-beam
Haskell/spock

Tests that are failing query verification #5145

C++/cpoll_cppsp-postgres-raw
C++/cpoll_cppsp-postgres-raw-threadpool
C++/cpoll_cppsp-raw
C++/cppcms
C++/silicon
C++/treefrog-hybrid
Dart/hunt-http
Elixir/phoenix

Tests that are not reading the id column in the queries or db tests #5167

D/hunt
D/vibed

This list is still growing. I'm not even a 3rd of the way through my audit. Please feel free to contribute in the comments section.

[jcheron]

I had already taken a little lead starting from the end of the Travis run:

Tests that are broken

• Swift/perfect-mongodb (Queries & Update tests)
• Swift/perfect-mysql
• Swift/perfect-postgresql
• Scala/play2-scala-anorm (Fortune test)
• Scala/akka-http
• Ruby/hanami
• Ruby/hanami-unicorn
• Ruby/rack
• Ruby/rack-falcon
• Ruby/rack-unicorn
• Python/vibora
• PHP/hhvm
• Perl/dancer
• Perl/kelp
• Perl/kelp-mongodb
• Perl/web-simple
• Nim/httpbeast
• Nim/jester
• Lua/octopus
• PHP/cakephp (Fortune test)
• Javascript/express-graphql-mysql (Update test)
• Java/spring-webflux
• Java/servlet-mysql
• Java/minijax
• Java/redkale-postgres (Db test in Travis)

Tests that are failing query verification https://github.com/TechEmpower/FrameworkBenchmarks/pull/5145

• TypeScript/typescript-rest (Update test ->no reading before update)
• Swift/kitura-mongodb (Update test)
• Scala/play2-scala-anorm-netty (Fortune test)
• Rust/rocket (Update test -> rows updated)
• Ruby/rails-postgresql (Queries test)
• Ruby/rails-unicorn
• Ruby/sinatra (Fortune test)
• Ruby/sinatra-postrges (Fortune test)
• Python/morepath (Query, Update tests)
• Python/pyramid (Update test)
• Python/pyramid-py2 (Update test)
• Python/quart (Update test)
• Python/spyne (Query, Update tests)
• Python/spyne-nginx-uwsgi (Db, Query, Update tests)
• Python/bottle (Db, update tests)
• Python/bottle-nginx-uwsgi
• Python/cherrypy
• PHP/imi (Update test)
• Perl/mojolicious (Query, Update tests)
• Kotlin/pronghorn (Db, Query, Update tests)
• Javascript/hapi (Query test)
• Javascript/koa (Update test)
• Javascript/express-postgres (Update test ->no reading before update)
• Javascript/express-graphql-postgres (Update test ->no reading before update)
• Javascript/express-chakra (Update test)
• Java/spring-webflux-mongo (Update test)
• Java/quarkus (Update test)
• Java/officefloor (Query, Update tests) -> Hibernate pb? (see https://github.com/TechEmpower/FrameworkBenchmarks/pull/5145#issuecomment-544735202)
  
  
  
  • officefloor-netty
  
  
  • officefloor-rapidoid
  
  
  • officefloor-spring_data
  
  
• Java/dropwizard (Update test)
• Java/dropwizard-jdbi-postgres (Db, Query, Fortune tests)
• Java/dropwizard-postgres(Db, Query, Update & Fortune tests)
• Groovy/hot-* (All db tests)
• C++/ulib (Update test, blocking for Travis on Query)
• C++/wt-postgres (Query, Update tests)

Tests that are not reading the id column in the queries or db tests https://github.com/TechEmpower/FrameworkBenchmarks/issues/5167

• Python/quart
• Python/aiohttp
• PHP/hhvm

[joanhey]

All php frameworks are correct after PR #5192 for #5167.

The only that need the change is hhvm, but it is broken a long time ago.

@jcheron You marked frameworks that are correct.
php, php-ngx, fat-fee, workerman, kumbiaphp, ...

They only omit the id in the update tests that is permited.

[joanhey]

php/amp is not broken, only hhvm.

[joanhey]

PHP/cakephp the fortunes test is broken after the last version update.

[joanhey]

https://www.techempower.com/benchmarks/#section=test&runid=59583d86-f7b6-4e86-b86a-a7e8eebf89fb&hw=ph&test=fortune&l=zik073-v the last row cakephp - Did not complete
and
https://tfb-status.techempower.com/unzip/results.2019-10-28-04-17-30-832.zip/results/20191024072803/cakephp/fortune/raw.txt

But the run log is empty https://tfb-status.techempower.com/unzip/results.2019-10-28-04-17-30-832.zip/results/20191024072803/cakephp/run

That happens after update the cakephp version.

[jcheron]

Maybe it was just an accident, it looks good on Travis.

[joanhey]

It looks good on travis, but the fortunes test always failed for the last 6 months in citrine and azure.

In azure:
https://www.techempower.com/benchmarks/#section=test&runid=45c588f9-9968-4330-934f-bd39ed92442c&hw=ph&test=fortune&l=zik073-v

I think that we could create an issue for the php frameworks, to clean this issue.
Created issue #5193

[an-tao]

@nbrady-techempower ,Hey, I have updated drogon with the PR #5191 , I think you can remove drogon from the list, thanks.

[michaelhixson]

@jcheron You commented in your PR that several Java frameworks using Hibernate were failing query verification, but they're not in your list in this thread. Why is that?

[jcheron]

@michaelhixson
I continue the pointing, starting from the bottom, I'm close to java...

[joanhey]

Tests that are not reading the id column in the queries or db tests #5167

C++/cpoll_cppsp (all variants)
C++/ulib
Crystal/crystal
Crystal/kemal
Crystal/raze
D/hunt (all variants)
Perl/mojolicious
Perl/plack
Python/aiohttp
Python/quart
Racket/racket