Frameworkbenchmarks: Help reproducing poor Haskell Servant Data Updates Results

[codygman]

I saw these poor results for Haskell where it only got 49 req/s:

https://www.techempower.com/benchmarks/#section=data-r14&hw=ph&test=update&l=8vmx1b&w=4zspc-0&d=e3&f=4ftiww-0-0-4e0w-0-6bk-pao

And the equivalent ruby got 1362.

I ran these codebases locally and got vastly different results. Is there some factor I'm overlooking that running locally would skew so much? Was Haskell performing so poorly just a fluke? I'm not sure how to investigate further at the moment.

Notes below:

* Database update results
** haskell
*** 1
**** 256
cody@zentop:~$ wrk -t4 -c256 -d10s http://localhost:7041/updates
Running 10s test @ http://localhost:7041/updates
  4 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    23.85ms    9.15ms 233.72ms   88.59%
    Req/Sec     2.71k   382.10     3.85k    82.71%
  107548 requests in 10.02s, 18.64MB read
Requests/sec:  10729.77
Transfer/sec:      1.86MB
** sinatra
*** 1
**** 256
cody@zentop:~$ wrk -t4 -c256 -d10s http://localhost:8080/updates
Running 10s test @ http://localhost:8080/updates
  4 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   185.62ms   43.17ms 377.42ms   82.13%
    Req/Sec   355.46    187.42   646.00     60.53%
  13685 requests in 10.03s, 2.99MB read
Requests/sec:   1363.94
Transfer/sec:    304.72KB

[nbrady-techempower]

Sure it could be any number of things. Looks like your Ruby results are the same, which is curious. Did you only run this once? Different thread counts, different concurrencies? Any server warmup? What are the specs? Also, should try running wrk from a different machine. Basically, data, data, and more data, would be the only way to proceed.

[codygman]

I'll run wrk from a different machine tomorrow and post more test results with different concurrency configurations.

[zloster]

Edit: about 2) and 3) the numbers for JSON don't show decrease at the higher concurrences so maybe you should investigate reconfiguration of the DB connection pool

@codygman Here are some pointers that could help you (IMO):

1. You can get the wrk command line from here: http://tfb-logs.techempower.com/round-14/final/servant/update/raw. This file is the log from where wrk is executed i.e. the client machine. The logs are for Round 14. You are interested in the following type of logs (scroll to the right for all options):

 Queries: 5 for servant
 wrk -H 'Host: localhost' -H 'Accept: application/json,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 15 -c 256 --timeout 8 -t 32 http://TFB-server:7041/updates?queries=5
---------------------------------------------------------

There will be one such section for each test data column (queries = 1, queries = 5 and so on). In the beginning you will also see the two warm up runs which have different parameters for wrk.

1. The Database update test is very heavy on queries to the database - for the 20 updates case there are total of 40 queries executed for each HTTP request. I've checked the source code and the code is not using batch updates but I don't know Haskell so maybe my statement is not true. However I noticed _interesting problem_ with servant implementation.
   All Database update tests are run always on 256 concurrency which is not the case for Fortunes and Single Query. Only Multiple queries is running also at that concurrency. If you open the test Data table for both test you will notice that the numbers align perfectly. For example Multiple queries = 1 servant gives 2434. You will have to press Data table just bellow Results because I can't find a way to get direct link to the results table. We divide it by 2 and get 1217. For Data updates queries = 1 the result is 1187. The two numbers are almost the same (within 3% which is very small deviation). This observation holds for each data point (queries=1, 5 and so on) for the Multiple queries and Data updates tests. So for the given concurrency and number of queries the results are aligned well for the two tests and the behaviour is the same.
   Next you have to look at Data table for Single query and Fortunes tests. The results for higher concurrences in both tests suggest there is problem with the libraries/source code implementation. Notice the catastrophic decrease in the numbers:
   Single query, concurrency = 32 gives 32274 requests/s
   Single query, concurrency = 256 gives 2416 requests/s
   Fortunes, concurrency = 64 gives 30409 request/s
   Fortunes, concurrency = 256 gives 2240 request/s
   If there are no problems with the implementation usually there is a graceful decrease in the results for higher concurrences. Just check the Data tables for the other two frameworks that you are interested in.
   _My conclusion_ is that you should investigate why servant is having a _problem_ handling higher concurrences in the tests that are using the database (see the update above). The better way to reproduce the higher concurrences will be to use separate machine for the wrk execution and make sure the machine has adequate power to handle the wrk load because of the coordinated omission problem.

2. You should investigate bumping up the number of connections in the DB connection pool. If I understand the code right the current number is _30_. The well performing Java frameworks are using numbers around _128_ to _256_. For example revenj-jvm is not bothering at all with DB connection pool: it just attaches a DB connection to each request processing thread for the duration of request processing. The total number of threads for the resin servlet container is limited to 256 or 512 (I don't remember).
   This setting will hurt the numbers a little for the cloud benchmarks but the queries are very lightweight and the DB seems not to have a problem with the higher connection numbers.

Good luck

[codygman]

@zloster Thanks a ton for all of the advice! I'll look into those and use the command log to ensure I'm using the same params and running the "same" test.

@nbrady-techempower Here is some more comprehensive data I gathered so far:

* server system stats
                          ./+o+-       cody@zentop
                  yyyyy- -yyyyyy+      OS: Ubuntu 16.04 xenial
               ://+//////-yyyyyyo      Kernel: x86_64 Linux 4.4.0-75-generic
           .++ .:/++++++/-.+sss/`      Uptime: 7h 49m
         .:++o:  /++++++++/:--:/-      Packages: 2646
        o:+o+:++.`..```.-/oo+++++/     Shell: bash 4.3.46
       .:+o:+o/.          `+sssoo+/    CPU: Intel Core i5-6600K CPU @ 4.3GHz
  .++/+:+oo+o:`             /sssooo.   RAM: 2440MiB / 15963MiB
 /+++//+:`oo+o               /::--:.  
 \+/+o+++`o++o               ++////.  
  .++.o+++oo+:`             /dddhhh.  
       .+.o+oo:.          `oddhhhh+   
        \+.++o+o``-````.:ohdhhhhh+    
         `:o+++ `ohhhhhhhhyo++os:     
           .o:`.syhhhhhhh/.oo++o`     
               /osyyyyyyo++ooo+++/    
                   ````` +oo+++o\:    
                          `oo++.   
* wrk system stats  
                          ./+o+-       cody@cody-G46VW
                  yyyyy- -yyyyyy+      OS: Ubuntu 16.04 xenial
               ://+//////-yyyyyyo      Kernel: x86_64 Linux 4.4.0-51-generic
           .++ .:/++++++/-.+sss/`      Uptime: 7d 8h 36m
         .:++o:  /++++++++/:--:/-      Packages: 2531
        o:+o+:++.`..```.-/oo+++++/     Shell: bash 4.3.46
       .:+o:+o/.          `+sssoo+/    Resolution: 1366x768
  .++/+:+oo+o:`             /sssooo.   DE: XFCE
 /+++//+:`oo+o               /::--:.   WM: Xfwm4
 \+/+o+++`o++o               ++////.   WM Theme: Default
  .++.o+++oo+:`             /dddhhh.   GTK Theme: Greybird [GTK2]
       .+.o+oo:.          `oddhhhh+    Icon Theme: elementary-xfce-dark
        \+.++o+o``-````.:ohdhhhhh+     Font: Sans 10
         `:o+++ `ohhhhhhhhyo++os:      CPU: Intel Core i5-3230M CPU @ 3.2GHz
           .o:`.syhhhhhhh/.oo++o`      GPU: GeForce GTX 660M
               /osyyyyyyo++ooo+++/     RAM: 1860MiB / 7869MiB
                   ````` +oo+++o\:    
                          `oo++.      

* Benchmarks
** Updates
*** 32 (wrk -c32 -d1m -t4)
**** summary
     Haskell handles 6x as many requests with 5x lower latency
**** data
***** haskell
****** 1
    4 threads and 32 connections
    Thread Stats   Avg      Stdev     Max   +/- Stdev
      Latency    94.36ms   64.65ms 847.40ms   87.60%
      Req/Sec    92.25     46.30   190.00     55.42%
    21833 requests in 1.00m, 19.67MB read
  Requests/sec:    363.47
  Transfer/sec:    335.24KB
***** ruby (mri passenger)
****** 1
    4 threads and 32 connections
    Thread Stats   Avg      Stdev     Max   +/- Stdev
      Latency   551.02ms   84.35ms 859.53ms   71.80%
      Req/Sec    18.86     13.98    70.00     78.11%
    3461 requests in 1.00m, 3.28MB read
  Requests/sec:     57.60
  Transfer/sec:     55.83KB
*** 256
**** summary
     Haskell handles about 7x more requests and has 89 timeouts vs 3344 timeouts in ruby
**** data
***** haskell
****** 1 
   4 threads and 256 connections
   Thread Stats   Avg      Stdev     Max   +/- Stdev
     Latency   686.11ms  392.00ms   2.00s    56.28%
     Req/Sec    92.44     32.69   250.00     74.81%
   22114 requests in 1.00m, 19.92MB read
   Socket errors: connect 0, read 0, write 0, timeout 89
 Requests/sec:    368.11
 Transfer/sec:    339.51KB
***** ruby (mri passenger)
****** 1
   4 threads and 256 connections
   Thread Stats   Avg      Stdev     Max   +/- Stdev
     Latency   965.80ms  539.53ms   1.95s    57.69%
     Req/Sec    17.47     11.07    90.00     73.62%
   3474 requests in 1.00m, 3.29MB read
   Socket errors: connect 0, read 0, write 0, timeout 3344
 Requests/sec:     57.81
 Transfer/sec:     56.03KB

[codygman]

Hm, after looking over the logs that @zloster provided I wasn't using the same wrk arguments. I'll see if that helps replicate the poor performance.

[codygman]

After updating the wrk arguments, I still couldn't reproduce the official results where ruby had 200x more responses per second than Haskell:

#haskell
cody@cody-G46VW:~$ wrk -H 'Host: localhost' -H 'Accept: application/json,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 15 -c 256 --timeout 8 -t 32 http://192.168.1.9:7041/updates?queries=20
Running 15s test @ http://192.168.1.9:7041/updates?queries=20
  32 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   455.36ms  315.64ms   1.86s    56.62%
    Req/Sec    19.72     11.46    70.00     83.13%
  Latency Distribution
     50%  442.61ms
     75%  731.26ms
     90%  879.40ms
     99%    1.17s 
  8461 requests in 15.10s, 6.34MB read
Requests/sec:    560.48
Transfer/sec:    429.98KB

#ruby 
cody@cody-G46VW:~$ wrk -H 'Host: localhost' -H 'Accept: application/json,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 15 -c 256 --timeout 8 -t 32 http://192.168.1.9:8080/updates?queries=20
Running 15s test @ http://192.168.1.9:8080/updates?queries=20
  32 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.18s   897.86ms   3.95s    85.56%
    Req/Sec    19.85     21.61    70.00     82.72%
  Latency Distribution
     50%    3.56s 
     75%    3.73s 
     90%    3.84s 
     99%    3.91s 
  1046 requests in 15.10s, 851.35KB read
Requests/sec:     69.27
Transfer/sec:     56.38KB

[codygman]

Maybe I'll do better focusing on benchmarking Haskell vs Go for this test... unless my machine can't run wrk fast enough to exhibit the performance degradation after scaling up. Perhaps the max throughput of my "wrk client computer" is too slow.

Should I run wrk on my faster computer maybe?

[nbrady-techempower]

@codygman if you have the ability to that would be helpful. When I have a chance this week, I'll try and run some tests in between the continuous benchmarking in our SC environment and try and get you more information.

[msmith-techempower]

Is this issue still relevant? there has been some Haskell work in recent months (since May last year >_<) that may have addressed this? Can this issue be closed?