Talk: Performance Issues

[hjanuschka]

we shipped coral to www.krone.at

right no we are using 4 VMS with 2 containers each serving the coral app in production mode.
however, http server from node does not seem to be able to handle the traffic.

questions:

• https://github.com/coralproject/talk/pull/1808 any ETA when this is merged, especially looking to the cache-headers
  
  
  
  • we tried tunneling it through varnish and adding headers, however varnish itself has issues with websockets. (reference: https://github.com/varnishcache/varnish-cache/issues/2748)
  
  
• how many nodes/VMs are you folks running on WashingtonPost, new york times?

we are trying to get our akamai config to work with websockets, to be able to add the cache headers, because 90% of the assets would be cachable on the client,and this would reduce the overall requests to the http. server.

[hjanuschka]

update: we ended up setting expires headers on nginx. - and added a couple of more nodes
right now at 4 VMS à 3 containers, with 4 cores each VM
but still response is sometimes really slow.

we added healthchecks to the containers, and auto restart them when they start failing.

at the end of the day it seems that node js segfaults, it produces a corefile (size of available ram) - however the core file seems to be incomplete, so i cannot really gdb it.

any advices?

[hjanuschka]

its driving me nuts, the node server is soo poor. now on 8VMS à 4 container with 5 cores per VM. the node process just stops responding under 5s, without notice 2 minutes after start.

what is going on here!

[hjanuschka]

we even scaled our SSL infrastructure, because of the slow coral responses, the nginx containers started to freak out

[kgardnr]

What we typically recommend is to provide a number set of nodes performing job processing and the rest serving http traffic. This would ensure that nodes processing emails and scraping tasks will operate successfully.

In this situation we'd recommend configuring a single Talk node as the job processor. You can set a node to not handle job processing by changing the Docker command executed for the image to:

./bin/cli-serve -w

(note the lack of -j which would have enabled jobs)

You can then create a new Talk instance where you set the Docker command to:

./bin/cli-serve -j

(note that http traffic will be handled by this node, you should not route http traffic to this particular node to keep it processing only jobs)

[cristiandean]

Hello @hjanuschka, we also have performance issues with Coral Talk. The last one was when we updated from version 4.4.0 to 4.5.1 and the CPU average consumption doubled #1773 . We did some tests and and we found that the villain is the websocket.

So, one of our ideas is disable the websocket losing freshness in favor of network stability #1615 (we're working on it).

[kgardnr]

@christiandean can you outline your Talk architecture for us?

[cristiandean]

@kgardnr we are using 10 VMs with 4 CPU / 8GB each. We also have 6 VMs of front end with 4 CPUs / 8GB each.

With that we are attending 1.500 request per second, about 30 inserts/updates per second on database.

We tried to isolate our websocket in 20 VMs but we didn't succeed.

[wyattjoh]

As I mentioned in the #1773 issue @cristiandean, is it possible to attach a debugger to the node process on an instance for a few seconds to gather a CPU profile? I'm groking the changes from 4.4.0 -> 4.5.1 now to try to identify the specific issue, but so far aren't finding anything standing out.

[hjanuschka]

@kgardnr thx, currently trying in production to use all nodes only with -w and just a single new one with -j (w/o web traffic)

first few minutes look good:

while ( true ); do time curl localhost:81; echo ""; done

its almost stable arround 80-200ms, whereas with -j there are spikes to 2-33sec. ⭕️

however, ill have to wait, before the party, up until tomorrow 06:00-13:00 CEST - to verify (there is our traffic peak)

is there any statsd/influxdb - middleware that we get a couple of metrics out of it? if not one of the next steps is that we'd create one.

@cristiandean what you mean with frontend? did you split the admin UI from the end user (commenter) part? (thinking about that too)

@wyattjoh thx, i ll try to get a profile tomorrow

[hjanuschka]

@wyattjoh - as of writing my perf. dropped again, now without -j:( back to 33s spikes.
could you provide a quick description on how to gather a profile?

[hjanuschka]

@wyattjoh @kgardnr ok figured it out, currently running it.
any chances i can drop the resulting profile on a private channel?

[wyattjoh]

Hi @hjanuschka, you can email the profile to me directly at wyatt at mozillafoundation.org.

[cristiandean]

@wyattjoh I'm programming to put the profiler next week.

@hjanuschka Yes, we splited the admin UI, backend application and static files (js, css, etc). With frontend I mean nginx servers.

[hjanuschka]

@wyattjoh just sent you a tgz - with 2 profiles, different size, both under circumstances where the webserver started responding really slow, avg 2-7 seconds on / which is just the 301 -> /admin

i also have a invalid core dump, seems like a cutoff or whatever, it has 1.6G and i definitly conains sensitive data, so even if it would be usefull, its near impossible to share.

[hjanuschka]

sent a couple of more profiler logs.

we are also trying to lazy load the Coral embed widget. with the bet on the fact that the majority of the users does not scroll down to the comment box.

also trying to find a way to kill the wss:// connection, on tabs that are open and unused for N minutes.

[wyattjoh]

Based on the feedback so far, I've created a potential patch that I think will definitely improve the performance around web-socket traffic.

You can see the changes here: https://github.com/coralproject/talk/pull/1817

Unfortunately, we don't have a great way to publish the changes as a easy docker image, so if possible, if you're already running v4.6.1, that branch should be a pretty simple switch. It's been lightly debugged and tested, but it's a work in progress, and will likely evolve as we get more feedback. To use this, you can build the docker image from the branch:

docker build -t coralproject/talk:latest -f Dockerfile .
docker build -t coralproject/talk:latest-onbuild -f Dockerfile.onbuild .

Followed by building your custom image and then manually pushing that to validate with.

Let me know if this improves anything for anyone!

[hjanuschka]

@wyattjoh many many thx, currently building new base image from the PR branch, and building on top of that my custom image and shipping it :shipit:

[hjanuschka]

looks amazing so far.

however before going into full party mode, i ll wait for the high peak hours tomorrow.
but from right now, 10 minutes after shipping it, it looks like i could even downsize my VMS, response is now constant between 0.10-0.015ms 🙌

@wyattjoh many many thx for your work and patience ❤️ really appreciated.

[wyattjoh]

Please note there was a fix applied to that branch, anyone using should redeploy with that fix.

[hjanuschka]

@wyattjoh i sent you another archive with profile data, it starts to be slow again. now that its morning here in CEST and traffic is growing.

would be great if you could have a look.

[hjanuschka]

just a little update, sorry for spamming this thread. going to do a how-to-run-at-scale.md once everything is cleared and running.

added 3 more machines, so we are now at:

6 machines each 3 instances of coral in -w mode.
1 machine 1 instance in -j mode.
2 machines 1 instance of nginx (for ssl termination).

where as one of the recent bottlenecks where the ssl nodes

those kernel params seem to help atleast for now

sysctl -w net.netfilter.nf_conntrack_max=131072
sysctl -w fs.file-max=12000500
sysctl -w fs.nr_open=20000500
sysctl -w net.ipv4.tcp_mem='10000000 10000000 10000000'
sysctl -w net.ipv4.tcp_rmem='1024 4096 16384'
sysctl -w net.ipv4.tcp_wmem='1024 4096 16384'
sysctl -w net.core.rmem_max=16384
sysctl -w net.core.wmem_max=16384
sysctl -w net.ipv4.tcp_max_orphans=400000

[hjanuschka]

still having slow responses good damn, what are we doing wrong.

[hjanuschka]

its going to be lunch time in our country, which means traffic peak at us!
we added a few cors, right now at

8 machines à 3 node instances + 3 ssl nodes + job container.
seems to settle now 👍 using the #1817 branch! 👏

[wyattjoh]

I'd also recommend setting the TALK_STATIC_URI @hjanuschka to your Akamai url (note that you'll need to do this across any node serving HTTP traffic right now, as it'll render the templates differently as a result), you could then set the origin to any of your public facing Talk HTTP servers. This way, websocket traffic won't go across it, and it'll lift the responsibility of sending static assets off of Talk, which should lessen the load some more even.

Final recommendation would be to instruct your Nginx nodes to cache the /embed/stream page for about 5 minutes a piece (force if you can via a similar Nginx config to below):

proxy_cache_path /data/nginx/cache keys_zone=my_cache:10m use_temp_path=off;

# ...

server {

  # ...

  location /embed/stream {
      proxy_cache my_cache;
      proxy_ignore_headers Cache-Control;
      proxy_cache_valid any 5m;
      add_header X-Cache-Status $upstream_cache_status;
      proxy_pass http://127.0.0.1:5000;
      proxy_http_version 1.1;
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-Proto $scheme;
  }

  # ...

}

I'd also suggest running N instances of Talk on each VM where N is the number of CPU cores you have available, until I can get to #1816, it'll be a more manual process to scale.

That should also hopefully lessen the load some more!

[hjanuschka]

thx i am going to play around with the TALK_STATIC_URI
it already tried to send expire headers on /embed/stream - but somehwo it does not work
and well it includes dnymic response (nonce) could that even lead to problems?

sample article: https://www.krone.at/1737960 - search for talk.krone.at in the network debugger.

 location /static {

      expires 1d;
      proxy_set_header        Host $host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-Proto "removed-at-edge";
      proxy_set_header        X-KRN-BY-SSL   1;
      proxy_pass          http://coral;

      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto $scheme;
      proxy_set_header        X-Forwarded-Host   $host:443;
      proxy_set_header        X-Forwarded-Server $host;
      proxy_set_header        X-Forwarded-Port   443;
      proxy_set_header        X-KRN-Real-IP   $remote_addr;

#force http 1
      proxy_read_timeout  900;
# google api sometimes takes 9+
      #proxy_intercept_errors on;



      }
      location /embed {

expires 1d;
      proxy_set_header        Host $host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-Proto "removed-at-edge";
      proxy_set_header        X-KRN-BY-SSL   1;
      proxy_pass          http://coral;

      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto $scheme;
      proxy_set_header        X-Forwarded-Host   $host:443;
      proxy_set_header        X-Forwarded-Server $host;
      proxy_set_header        X-Forwarded-Port   443;
      proxy_set_header        X-KRN-Real-IP   $remote_addr;

#force http 1
      proxy_read_timeout  900;
# google api sometimes takes 9+
      #proxy_intercept_errors on;




      }

we had a upstream {} block in nginx, which did the loadbalancing - i just changed that to only one backend, which is a haproxy which then does the loadbalancing, looking 💯 so far!
i suspect the nginx is not recognizing dead nodes fast enough and passing on requests.

haproxy is a bit more flexible when it comes to health checks. (yes we are using OSS version of nginx so no health_check for us)

we are now at 10 VMS with 4 instances each, by having 5 cores (one left for the system)
and we implemented lazyloading, calling Coral.render only when comment section is in viewport.

overall we are on a good path, and looking forward, weekend should be fine (less traffic)
monday is the week-peak (beside breaking news) - lets hope.

[hjanuschka]

ohhh and one vm with one instance for the jobs 😬

[hjanuschka]

@wyattjoh one more thing, any chances we can get #1817 merged and released? so we can switch back to official images? because that seemed to be the biggest performance gain, beside all other tweaks.

[wyattjoh]

I plan on getting #1817 merged shortly yes! It's just undergoing a short review first.

I'll review the other info you passed and continue to iterate on the work we've done so far.

Many thanks @hjanuschka!

[wyattjoh]

Good news @hjanuschka, after a review from @cvle, I was able to additionally, greatly improve the performance of the subscription system with batching.

We'll be releasing 4.6.2 today with this fix in it.

[wyattjoh]

Released v4.6.2 https://github.com/coralproject/talk/releases/tag/v4.6.2

Let us know how it goes!

[hjanuschka]

@wyattjoh going to pull the new images, really appreciate your help! looking forward to.
will post updates, once i have a bit more data 👍

[hjanuschka]

@wyattjoh so my token (besides the really bad code style; havent yet found an accaptable beautifier for vim and ES6) - script is ok? (the docs arround that where a bit confusing. (another PR we'll do, once everything is fine)

[hjanuschka]

@wyattjoh shipped and running, but for whatever reason it still shows 4.6.1 where as i verified, the container contains the code from the PR in #1817 - you forgot a bump somwhere?

[wyattjoh]

Yeah :( forgot to bump the version in the package.json... Will re-release v4.6.2 shortly, it won't impact anything though!

[kgardnr]

Solved in https://github.com/coralproject/talk/pull/1817

[hjanuschka]

weekend was good. we even had our first sad breaking event, and talk kept running.
however i am a bit curious while trying to cache the /stream/embed/

we are now using, TALK_STATIC_UR - via akamai/cdn
and 10minutes nginx caching (fixed cache key, excl. query string), as @wyattjoh mentioned, overall it looks great, just a little nitpick on >further improvements.

if i get it right, the page/response itself is 100% static, so no matter what comes in it could be cached by cdn, as the react/js part inside the iframe extracts the parameters from the query string?.

however the query string contains information that is a little bit dynamic (asseturl, asset-title...)

on akamai/cdn/nginx - i could easily remove the query string and keep a single copy in the cache, reducing the origin traffic. but the end-user still downloads alot, and wastes his own bandwidth, and in the end wastes my bandwidth which we pay too. w/o query string i could cache the tiny html for atleast 1 day on the users browser :)

however - TALK_STATIC_URI - does not change it, and if i try it hardcorded it fails due to cors restrictions.

ideas.

• call fixed url /embed/stream
  
  
  
  • pass parameters with postMessage/receiveMessage
  
  
• try to fix cors responses, and provide TALK_STREAM_FRAME_URI - to seperate it
• in general (might be the overall solution) - make websockets cors ready, and provide TALK_WEBSOCKET_URI
  
  
  
  • so we could divide traffic by CDN - and non CDN traffic (aka websockets)
  
  

what do you think of it? any quick win? if not i would love to go for a PR. i personally would prefer the TALK_WEBSOCKET_URI way.

[kgardnr]

Hi @hjanuschka! Wonderful news that Talk performance has improved - woohoo!

Most of our dev team is off this week for summer holidays, but someone will get back to you as soon as we can!

[hjanuschka]

@kgardnr thx for the info.

going to close the issue, since it runs butter smooth now.

a quick re-cap - for others stumbling upon this issue:

• the patches in #1817 fixed the root cause inside the software itself
• using TALK_STATIC_URI to server assets via CDN - reduced the amount of connections to the express app (incl. setting cache headers there)
• scaling SSL nodes - to use 3 nginx+haproxy machines
• using IPVS to loablance on layer 3 to ssl nodes.
• scaling to 8 VMS with 5 cores and each of them with 4 coral instances
• splitting webservers (-w) and job servers (-j) helped alot with random long running requests
• caching /embed/stream directly on nginx, helped also on reducing backend connections
• switching from nginx based loadbalancing to haproxy - was the last key to sucess
  
  
  
  • we now use nginx (for caching, and some other stuff) - and nginx has a backend defined, which goes to haproxy, where all the real application servers are configured
  
  
• tuning kernel parameters on the nginx/ssl nodes

Kernel Parameters

#SSL tweak

sysctl -w net.netfilter.nf_conntrack_max=131072
sysctl -w fs.file-max=12000500
sysctl -w fs.nr_open=20000500
# Set the maximum number of open file descriptors
ulimit -n 20000000

# Set the memory size for TCP with minimum, default and maximum thresholds 
sysctl -w net.ipv4.tcp_mem='10000000 10000000 10000000'

# Set the receive buffer for each TCP connection with minumum, default and maximum thresholds
sysctl -w net.ipv4.tcp_rmem='1024 4096 16384'

# Set the TCP send buffer space with minumum, default and maximum thresholds 
sysctl -w net.ipv4.tcp_wmem='1024 4096 16384'

# The maximum socket receive buffer sizemem_max=16384
sysctl -w net.core.rmem_max=16384

# The maximum socket send buffer size
 sysctl -w net.core.wmem_max=16384

HAProxy conf

        #log /dev/log   local0
        #log /dev/log   local1 notice
        #chroot /var/lib/haproxy
        #stats socket /run/haproxy/admin.sock mode 660 level admin
        #stats timeout 30s
        #user haproxy
        #group haproxy
        maxconn 40000
        #daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL). This list is from:
        #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
        # An alternative list with additional directives can be obtained from
        #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
  timeout check 2000
  timeout connect 8000
  timeout client  8000
  timeout server  8000
frontend coral
    bind *:87
    mode http
    default_backend coral_nodes
backend coral_nodes
    mode http
    balance roundrobin
    option forwardfor
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    option httpchk HEAD / HTTP/1.1\r\nHost:localhost
    server coral1-1 backend-coral1:80 check inter 2s fastinter 1s downinter 2s rise 3 fall 2
    server coral1-2 backend-coral1:81 check inter 2s fastinter 1s downinter 2s rise 3 fall 2
    server coral1-3 backend-coral1:82 check inter 2s fastinter 1s downinter 2s rise 3 fall 2
    server coral1-4 backend-coral1:83 check inter 2s fastinter 1s downinter 2s rise 3 fall 2
..... all other nodes

NGINX conf

http {
    proxy_cache_path /var/cache/nginx/coral keys_zone=coral_app:20m use_temp_path=off;
    server_tokens off;
  client_max_body_size 300M;

  sendfile           on;
  tcp_nopush         on;
  tcp_nodelay        on;
  keepalive_timeout  5;


  upstream coral {
      least_conn;
      server haproxy.krn.krone.at:87;
  }

  server {
    listen 443 ssl http2;
    server_name ~^.*(talk|talk-cdn)\.krone\.at$;

    ssl_certificate    /keys/server.crt;
    ssl_certificate_key /keys/server.key;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
#    ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
    ssl_prefer_server_ciphers on;
    #ssl_session_cache shared:SSL:10m;

    ssl_session_cache shared:SSL:20m;
    ssl_session_timeout 180m;
    ssl_buffer_size 4k;


    ssl_dhparam /keys/dhparam.pem;
    error_page 503 /503.html;
    error_page 502 /503.html;

    location = /503.html {
      root /usr/share/nginx/html/;
    }
    # coral websocket
    location /api/v1/live {
      proxy_set_header        Host $host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-Proto "removed-at-edge";
      proxy_set_header        X-KRN-BY-SSL   1;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
      # legacy www01
      proxy_pass          http://coral;

      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto $scheme;
      proxy_set_header        X-Forwarded-Host   $host:443;
      proxy_set_header        X-Forwarded-Server $host;
      proxy_set_header        X-Forwarded-Port   443;
      proxy_set_header        X-KRN-Real-IP   $remote_addr;

#force http 1
      proxy_read_timeout  900;
# google api sometimes takes 9+
      #proxy_intercept_errors on;


    }

   location / {
      proxy_set_header        Host $host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-Proto "removed-at-edge";
      proxy_set_header        X-KRN-BY-SSL   1;
      proxy_pass          http://coral;

      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto $scheme;
      proxy_set_header        X-Forwarded-Host   $host:443;
      proxy_set_header        X-Forwarded-Server $host;
      proxy_set_header        X-Forwarded-Port   443;
      proxy_set_header        X-KRN-Real-IP   $remote_addr;

#force http 1
      proxy_read_timeout  900;
# google api sometimes takes 9+
      #proxy_intercept_errors on;

      location /static {

      expires 1d;
      proxy_set_header        Host $host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-Proto "removed-at-edge";
      proxy_set_header        X-KRN-BY-SSL   1;
      proxy_pass          http://coral;

      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto $scheme;
      proxy_set_header        X-Forwarded-Host   $host:443;
      proxy_set_header        X-Forwarded-Server $host;
      proxy_set_header        X-Forwarded-Port   443;
      proxy_set_header        X-KRN-Real-IP   $remote_addr;

#force http 1
      proxy_read_timeout  900;
# google api sometimes takes 9+
      #proxy_intercept_errors on;



      }
      location /embed/stream {
        proxy_cache coral_app;
        proxy_cache_key $scheme$proxy_host$uri;
        proxy_ignore_headers Cache-Control;
        proxy_cache_valid any 5m;
        add_header X-NGINX-CACHE-Status $upstream_cache_status;


expires 1d;
      proxy_set_header        Host $host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-Proto "removed-at-edge";
      proxy_set_header        X-KRN-BY-SSL   1;
      proxy_pass          http://coral;

      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto $scheme;
      proxy_set_header        X-Forwarded-Host   $host:443;
      proxy_set_header        X-Forwarded-Server $host;
      proxy_set_header        X-Forwarded-Port   443;
      proxy_set_header        X-KRN-Real-IP   $remote_addr;
      proxy_read_timeout  900;
      #proxy_intercept_errors on;




      }


    }



  }

additionally as we use akamai as our CDN - we force to downstream client cachability of JS/CSS to 30 days - as in case of code change, the hash inside the filename changes anyway

keep in mind that the nginx config is a hell of a copy and 🍝 weirdo. since we are happy that it now works, we are going to further optimize it using includes. was just a hell of a ride launching it into production.

we build a custom plugin, to send metrics out to grafana, whenever a comment is created (probably going to publish it)

we locked down the comments (also via custom plugin) from 23:00 to 06:00 (as there is no human to do the review, and every comment is reviewed)

graph group/time-window is 1h, so the current peak is around 500 something per hour

[hjanuschka]

MANY thx to all involved! and special thx for @wyattjoh (let me know when you are around vienna to get a 🍺 or a ☕️ ) for investing your time to look at the profiler data, and comming up with a fix that fast!