Frameworkbenchmarks: Include memory consumption for the benchmark

[ufoscout]

I often hear complains about the TechEmpower benchmarks not reporting any information about frameworks memory consumption. I agree that indeed it is an interesting information and that it should be somehow provided.
Ticket #1295 was closed four years ago but still nothing has been implemented.
Is there any real plan to have this fixed in the coming rounds?

[nbrady-techempower]

Thanks for opening an issue. We've discussed this recently here https://github.com/TechEmpower/FrameworkBenchmarks/issues/3940

[ufoscout]

@nbrady-techempower
Is there a reason to explain the fact that you close tickets are not implemented or fixed?
The ticket you linked was also closed without any real answer.
This clearly communicates, no matter what you say, that you are ignoring the request. In fact, all the issues opened in the last 4 years requesting for this feature were closed without any fix.

[msmith-techempower]

@ufoscout The toolset does, in fact, capture memory statistics, but we don't currently render it in any meaningful way on the results website.

[cjnething]

Hi @ufoscout we haven't forgotten about the requests! We just moved them to the roadmap (under TODOs: Improve dstat usage). Hope that clears things up. Thanks!

[dbohdan]

I was also interested in this information, so I wrote a script that calculates the average memory usage of every framework in a given benchmark. Here are the results for the "fortune" benchmark in https://tfb-status.techempower.com/results/9ce91254-04d7-4455-9742-aa6a1f673e86 (Azure, 2019-06-04). The value of -1.0 means the script detected abnormal data. You should keep in mind that some of these numbers are for tests that failed or had errors.

| Memory usage (MiB) | Framework |
| ------------------ | --------- |
| -1.0 | angel |
| -1.0 | angel-postgres |
| -1.0 | falcore |
| -1.0 | goji |
| -1.0 | hanami |
| -1.0 | sailsjs-postgres |
| 832.8 | aah-mysql |
| 837.0 | aah-postgresql |
| 910.6 | openresty |
| 1016.9 | echo |
| 1101.9 | padrino-unicorn |
| 1240.5 | actix-pg |
| 1247.8 | phoenix |
| 1250.7 | actix-core |
| 1278.9 | blacksheep |
| 1305.7 | atreugo-quicktemplate |
| 1306.8 | atreugo |
| 1319.6 | cherrypy |
| 1322.3 | express-postgres |
| 1331.0 | atreugo-mongo |
| 1332.4 | cherrypy-py3 |
| 1333.0 | padrino |
| 1333.3 | express-graphql-postgres |
| 1334.9 | atreugo-prefork-quicktemplate |
| 1337.4 | amber |
| 1338.8 | actix-diesel |
| 1342.8 | peachpie |
| 1350.5 | atreugo-prefork |
| 1355.5 | express-graphql-mysql |
| 1359.1 | express-graphql-mongodb |
| 1385.8 | chi-gojay |
| 1390.3 | chi |
| 1391.2 | chi-sjson |
| 1399.1 | atreugo-mongo-prefork |
| 1440.1 | cpoll_cppsp-raw |
| 1448.5 | chi-gojay-prefork |
| 1455.6 | chi-sjson-prefork |
| 1459.1 | crystal |
| 1461.2 | chi-prefork |
| 1464.9 | crystal-radix |
| 1483.7 | iron |
| 1503.2 | amp |
| 1524.6 | act-eclipselink-pgsql |
| 1527.4 | act-eclipselink-mysql |
| 1539.9 | cppcms |
| 1545.9 | cppcms-postgres |
| 1549.7 | cutelyst-thread-my-grantlee |
| 1553.7 | cutelyst-thread-pg-grantlee |
| 1554.7 | bottle-nginx-uwsgi |
| 1557.3 | api_hour |
| 1558.1 | act-eclipselink-mysql-rythm |
| 1565.4 | cutelyst-pf-my |
| 1565.8 | cutelyst-thread-pg |
| 1571.9 | cutelyst-pf-pg |
| 1573.6 | cutelyst-thread-my |
| 1574.7 | cutelyst-pf-my-nodelay |
| 1575.1 | cutelyst-thread-pg-nodelay |
| 1577.5 | cutelyst-pf-pg-nodelay |
| 1579.4 | cutelyst-thread-my-nodelay |
| 1579.5 | act-morphia-mongo |
| 1579.7 | act-morphia-mongo-rythm |
| 1581.5 | kami |
| 1583.2 | act-hibernate-mysql-rythm |
| 1586.3 | act-eclipselink-pgsql-rythm |
| 1592.4 | silicon |
| 1594.1 | act-hibernate-pgsql |
| 1597.5 | act-hibernate-mysql |
| 1602.0 | workerman |
| 1606.8 | act-hibernate-pgsql-rythm |
| 1610.7 | webware |
| 1618.7 | cutelyst-nginx-pg |
| 1621.5 | express-mysql |
| 1633.9 | cutelyst-nginx-my |
| 1639.3 | api_hour-mysql |
| 1639.8 | starlette |
| 1642.3 | woo |
| 1650.1 | raze |
| 1652.4 | rack-sequel-postgres-unicorn-mri |
| 1656.0 | rack-sequel-unicorn-mri |
| 1660.9 | swoole |
| 1663.0 | sinatra-sequel-postgres-unicorn-mri |
| 1667.4 | hunt-dmd |
| 1672.6 | weppy-nginx-uwsgi |
| 1673.7 | warp-hasql |
| 1681.2 | bottle-raw |
| 1684.1 | quart |
| 1692.7 | warp |
| 1711.7 | kitura-mongodb |
| 1712.3 | lapis |
| 1713.9 | kemal |
| 1719.2 | sinatra-sequel-unicorn-mri |
| 1734.7 | quart-uvicorn |
| 1735.7 | warp-mysql-haskell |
| 1736.0 | aspcore-vb-mw-ado-my |
| 1736.6 | sinatra-unicorn-mri |
| 1738.9 | express-mongodb |
| 1740.9 | h2o |
| 1746.9 | kitura-gcd-postgres-orm |
| 1747.8 | kitura-postgres |
| 1748.1 | kitura-gcd-postgres |
| 1748.9 | drogon |
| 1750.3 | rocket |
| 1750.6 | aspcore-rhtx-pg |
| 1751.5 | drogon-raw |
| 1756.5 | servant |
| 1760.5 | fasthttp |
| 1765.3 | kitura-postgres-orm |
| 1767.4 | tornado-py3-uvloop |
| 1770.6 | kelp |
| 1774.7 | sinatra-sequel-postgres-passenger-mri |
| 1775.3 | aspcore-mw-ef-pg |
| 1775.7 | fasthttp-postgresql |
| 1776.5 | kitura-nio-postgres-orm |
| 1777.9 | rack-sequel-passenger-mri |
| 1780.7 | aspcore-vb-mw-ado-pg |
| 1782.1 | rack-sequel-postgres-passenger-mri |
| 1783.2 | kelp-mongodb |
| 1784.0 | wt |
| 1786.2 | sinatra-postgres-unicorn-mri |
| 1788.0 | servant-postgres-wire |
| 1790.0 | fasthttp-prefork |
| 1790.8 | klein |
| 1809.1 | kitura-nio-postgres |
| 1810.0 | fasthttp-postgresql-prefork |
| 1811.4 | sinatra-passenger-mri |
| 1816.9 | sinatra-sequel-passenger-mri |
| 1821.1 | sinatra-postgres-passenger-mri |
| 1822.2 | blade |
| 1823.1 | tornado-py3 |
| 1826.1 | servant-mysql-haskell |
| 1827.0 | treefrog-hybrid |
| 1827.1 | hunt |
| 1834.2 | tornado |
| 1842.1 | hamlet-swoole |
| 1847.2 | treefrog |
| 1851.0 | responder |
| 1851.8 | treefrog-postgres |
| 1859.1 | aspcore-mono-my |
| 1859.9 | roda-sequel-postgres-unicorn-mri |
| 1860.0 | fastapi |
| 1860.1 | lumen-swoole |
| 1863.8 | aspcore-mono-mw-pg |
| 1864.4 | laravel-swoole |
| 1866.4 | rack-sequel-postgres |
| 1866.4 | pyramid |
| 1868.8 | mochiweb |
| 1877.6 | treefrog-mongodb |
| 1879.5 | aspcore-mono-mvc-my |
| 1884.9 | rails-unicorn |
| 1889.4 | sinatra-sequel-postgres |
| 1892.5 | urweb |
| 1893.0 | wt-postgres |
| 1893.4 | aspcore-mono-mw-my |
| 1893.9 | urweb-mysql |
| 1896.4 | bottle |
| 1899.3 | aspcore-mono-pg |
| 1900.1 | vibed |
| 1909.3 | vibed-ldc |
| 1910.0 | roda-sequel-unicorn-mri |
| 1912.5 | roda-sequel-postgres-passenger-mri |
| 1922.2 | django |
| 1928.4 | rack-sequel |
| 1934.1 | dart |
| 1936.3 | roda-sequel-passenger-mri |
| 1947.3 | pyramid-py2 |
| 1955.3 | sinatra |
| 1957.5 | aspcore-mono-mw-ef-pg |
| 1969.2 | jooby2-undertow |
| 1969.7 | sinatra-postgres |
| 1970.1 | django-postgresql |
| 1970.5 | aspcore-mono-mvc-pg |
| 1973.3 | racket |
| 1979.6 | vibed-ldc-pgsql |
| 1985.4 | django-py3 |
| 1995.2 | redstone |
| 2002.0 | aspcore-mono-mvc-ef-pg |
| 2002.5 | ulib-mysql |
| 2012.4 | uvicorn |
| 2014.7 | aspcore-mw-dap-my |
| 2015.4 | aspcore-mw-ado-my |
| 2016.3 | polkadot-postgres |
| 2024.5 | aspcore-mw-dap-pg |
| 2027.9 | ulib-mongodb |
| 2030.6 | web2py-optimized |
| 2033.0 | spyne-nginx-uwsgi |
| 2034.3 | aspcore-mvc-dap-my |
| 2036.3 | vibed-dmd-pgsql |
| 2038.6 | jooby |
| 2041.3 | aspcore-mvc-dap-pg |
| 2042.4 | polkadot-mysql |
| 2048.4 | redstone-mongodb |
| 2049.3 | roda-sequel-postgres |
| 2050.0 | aspcore-mw-ado-pg |
| 2052.5 | web2py |
| 2058.8 | ubiquity-react |
| 2065.8 | sinatra-sequel |
| 2070.5 | dropwizard-postgres |
| 2074.7 | http4k-netty |
| 2075.7 | aspcore-mvc-ado-my |
| 2080.9 | spyne-raw |
| 2085.2 | spyne |
| 2087.6 | ffead-cpp |
| 2093.6 | mojolicious |
| 2098.0 | rails |
| 2099.9 | php |
| 2100.5 | jooby2 |
| 2104.2 | aspcore-mvc-ado-pg |
| 2109.4 | gin |
| 2113.9 | turbogears |
| 2120.8 | ffead-cpp-apache-mongo |
| 2121.1 | hanami-unicorn |
| 2127.8 | aspcore-ado-my |
| 2131.9 | go-pgx-quicktemplate |
| 2136.5 | go-pgx-prefork-quicktemplate |
| 2148.6 | php-php5-raw |
| 2150.5 | polkadot-mongodb |
| 2150.7 | ningle |
| 2152.2 | go-pgx |
| 2153.4 | pedestal |
| 2155.7 | go-pgx-prefork |
| 2157.0 | go-my-prefork |
| 2159.6 | http4k-apache |
| 2161.0 | ninglex |
| 2164.3 | aspcore-ado-pg |
| 2167.6 | go-my |
| 2171.8 | koa-mysql |
| 2174.6 | jooby2-jetty |
| 2176.6 | octopus |
| 2182.8 | fastify |
| 2194.1 | ktor-jetty |
| 2201.5 | fastify-postgres |
| 2204.0 | dropwizard |
| 2206.8 | hexagon-jetty-postgresql |
| 2208.7 | go-mgo |
| 2219.0 | macchiato |
| 2219.0 | http4k |
| 2219.5 | ktor |
| 2222.8 | http-kit |
| 2226.6 | light-4j |
| 2227.4 | fastify-mysql |
| 2229.8 | weppy-py3 |
| 2232.2 | roda-sequel |
| 2233.3 | jlhttp-postgres |
| 2234.9 | hapi |
| 2235.0 | duct-httpkit |
| 2240.7 | aqueduct |
| 2240.9 | koa |
| 2243.8 | http-kit-raw |
| 2256.3 | go-mgo-prefork |
| 2261.8 | aspcore-mvc-ef-pg |
| 2265.4 | ffead-cpp-mysql |
| 2266.4 | ffead-cpp-postgresql |
| 2269.3 | servlet-mysql |
| 2271.8 | roda-sequel-postgres-torquebox-jruby |
| 2273.9 | roda-sequel-torquebox-jruby |
| 2298.0 | hapi-mysql |
| 2299.8 | php-h2o |
| 2302.5 | ffead-cpp-nginx-postgresql |
| 2302.9 | ffead-cpp-nginx-mongo |
| 2303.6 | ffead-cpp-nginx-mysql |
| 2305.8 | servlet-postgresql |
| 2307.9 | flask-nginx-uwsgi |
| 2310.8 | zebra |
| 2312.0 | undertow-postgresql |
| 2314.8 | activeweb |
| 2315.8 | nodejs-mongodb-raw |
| 2317.9 | nodejs-chakra |
| 2319.4 | http4k-undertow |
| 2319.6 | nodejs |
| 2323.3 | phalcon-micro |
| 2323.4 | koa-postgres |
| 2326.9 | spring-mongo |
| 2335.7 | hapi-postgres |
| 2339.0 | php-pools |
| 2361.8 | grizzly-jersey |
| 2371.4 | php-raw7-tcp |
| 2379.9 | nodejs-postgres |
| 2381.0 | php-pgsql-raw |
| 2386.2 | weppy |
| 2392.7 | dropwizard-jdbi-postgres |
| 2401.8 | rapidoid-postgres |
| 2407.1 | ffead-cpp-apache-postgresql |
| 2409.0 | rapidoid-mysql |
| 2412.9 | hexagon |
| 2420.8 | undertow-jersey |
| 2437.3 | hapi-nginx |
| 2439.4 | ffead-cpp-apache-mysql |
| 2454.0 | httpserver-postgres |
| 2462.4 | sailsjs |
| 2473.8 | undertow-jersey-hikaricp |
| 2473.9 | flask-raw |
| 2484.2 | flask |
| 2491.8 | ringojs-convenient |
| 2505.0 | morepath |
| 2515.8 | hexagon-resin-mongodb |
| 2525.5 | phalcon-mongodb |
| 2526.9 | http4k-ktorcio |
| 2539.1 | vertx-web-susom-postgres |
| 2552.2 | nodejs-mysql |
| 2554.5 | giraffe-stripped |
| 2562.7 | duct |
| 2587.0 | giraffe |
| 2600.3 | phpixie |
| 2604.9 | hexagon-resin-postgresql |
| 2606.0 | ktor-reactivepg |
| 2629.9 | kumbiaphp-raw |
| 2639.3 | compojure-raw |
| 2646.1 | wicket |
| 2650.8 | spring-webflux-pgclient |
| 2656.0 | phalcon |
| 2660.6 | ktor-jasync |
| 2680.3 | kumbiaphp |
| 2695.5 | spring-webflux-jdbc |
| 2697.6 | jawn |
| 2707.9 | compojure |
| 2719.1 | spring-webflux-mongo |
| 2719.8 | luminus |
| 2720.6 | sinatra-sequel-torquebox-jruby |
| 2749.7 | minijax |
| 2760.1 | vertx-postgres |
| 2774.6 | slim-php5 |
| 2776.4 | spark |
| 2788.4 | redkale-postgres |
| 2807.4 | slim |
| 2831.4 | proteus-mysql |
| 2833.3 | php-eloquent |
| 2838.8 | ktor-cio |
| 2839.0 | vertx-web-postgres |
| 2886.1 | tapestry |
| 2915.9 | play2-java-ebean-hikaricp |
| 2920.9 | play2-java-jooq-hikaricp-netty |
| 2927.2 | ratpack-pgclient |
| 2930.6 | play2-java-ebean-hikaricp-netty |
| 2938.0 | sinatra-sequel-postgres-torquebox-jruby |
| 2945.6 | play2-java-jooq-hikaricp |
| 2954.2 | play2-scala-anorm |
| 2961.5 | spring |
| 2963.4 | play2-java-jpa-hikaricp |
| 2968.4 | play2-scala-anorm-netty |
| 2977.6 | tornado-pypy2 |
| 2995.0 | pronghorn |
| 2998.4 | play2-scala-reactivemongo-netty |
| 3002.5 | play2-java-jpa-hikaricp-netty |
| 3003.7 | gemini-postgres |
| 3013.7 | duct-mongodb |
| 3020.2 | play2-scala-reactivemongo |
| 3038.1 | rack-sequel-torquebox-jruby |
| 3040.0 | play2-scala-slick-netty |
| 3042.7 | ubiquity |
| 3043.6 | aspnet-mono-ngx-my |
| 3049.9 | vertx-web-mongodb |
| 3058.1 | ratpack-jdbc |
| 3071.3 | play2-scala-slick |
| 3088.0 | dropwizard-mongodb |
| 3093.5 | ninja-standalone |
| 3122.8 | duct-aleph |
| 3132.5 | hamlet |
| 3134.8 | ulib-postgres_fit |
| 3231.1 | rack-sequel-postgres-torquebox-jruby |
| 3255.8 | spring-webflux-rxjdbc |
| 3260.5 | lithium |
| 3283.1 | officefloor-tpr |
| 3284.2 | gemini-mysql |
| 3309.8 | officefloor-thread_affinity |
| 3347.3 | wildfly-ee |
| 3404.8 | fintrospect |
| 3425.8 | micronaut |
| 3427.1 | grails |
| 3449.8 | aspnet-mono-ngx-my-ef |
| 3458.9 | officefloor-spring_data |
| 3468.9 | officefloor-micro |
| 3471.2 | es4x |
| 3498.3 | yii2-raw |
| 3573.0 | officefloor |
| 3612.3 | spring-webflux |
| 3619.9 | fuel |
| 3621.1 | ulib-postgres |
| 3716.6 | akka-http |
| 3737.0 | officefloor-netty |
| 3826.0 | yii2 |
| 4098.4 | http4s |
| 4122.9 | proteus |
| 4168.8 | officefloor-rapidoid |
| 4172.4 | lumen |
| 4310.3 | codeigniter |
| 4322.1 | hot-mysql |
| 4328.2 | symfony |
| 4340.4 | hot-mongodb |
| 4440.0 | hot-postgres |
| 4570.9 | bottle-pypy2 |
| 4686.4 | cakephp |
| 4770.2 | laravel |
| 4772.6 | fat-free-raw |
| 4960.1 | weppy-pypy2 |
| 5045.6 | akka-http-slick-postgres |
| 5143.9 | helidon |
| 5296.1 | flask-pypy2 |
| 5331.6 | flask-pypy2-raw |
| 6824.7 | greenlightning |
| 8829.2 | onion |

To run the following script you will need to install a recent version of Tcl and jq (e.g., sudo apt install tcl jq on Debian and Ubuntu). Download and unzip results.zip from tfb-status.techempower.com, then run the script against its contents (see the usage message in the script's body for an example). Please comment about any bugs or problems you encounter.

#! /usr/bin/env tclsh

proc memory-usage-average statsFile {
    exec jq -r {
        if length == 6 and (map(length) | add) > 0 then
            (map(.[] | .["memory usage"].used) | (add / length) | floor)
        else
            -1
        end
    } $statsFile
}

proc summarize {resultsDir {benchmark {fortune}}} {
    foreach subdir [lsort [glob -dir $resultsDir *]] {
        set f $subdir/$benchmark/stats.txt.json
        if {![file exists $f]} continue

        puts [list [memory-usage-average $f] [file tail $subdir]]
    }
}

set badUsage [expr {[llength $argv] ni {1 2}}]
if {$argv in {-h -help --help /?} || $badUsage} {
    set me [info script]
    puts stderr "usage: $me path \[benchmark\]\ne.g.,\n\$ $me\
                 mnt/tfb/FrameworkBenchmarks/results/20190605011349/ fortune"
    exit $badUsage
}
summarize {*}$argv

[bhauer]

@dbohdan Very cool data! Thanks for gathering that.

One caveat: we've not applied much (if any) effort in tuning the memory usage of any implementations. We've also not asked contributors to do memory tuning. Just keep that in mind when evaluating memory usage. For example, tests on the JVM may be allocating a conservative amount of memory without much concern.