Kong: Kong 1.0.3 vs 1.2.1 Load Test CPU Utilization Benchmark Diff
Summary
I just ran a 120 TPS load-test against a proxy(service/route pair) running just a plain request termination plugin on-top of the route throwing {"message": "Success"} HTTP 200 on two different Kong versions
On 1.0.3 with a few modifications to src outlined here:
https://github.com/Kong/kong/issues/4769#issuecomment-513873111
I am seeing utilization hit .41 CPU utilized at a stable rate
On 1.2.1 I am seeing CPU utilization hit .53 CPU utilized at a stable rate(note I do run 1 mod on 1.2.1, outlined here):
https://github.com/Kong/kong/issues/4194#issuecomment-498001854
This is our sandbox env and no other active traffic would be hitting these environments. Both times I also had global plugins enabled like rate-limit,statsd,http log, correlation id as well.
I figured the 29.27% increase would be of interest to yall. Maybe a quick internal load-test Kong side could check for similar findings?
Steps To Reproduce
- Run load test as prescribed above on the various Kong versions.
Additional Details & Logs
- Kong version 1.0.3(with slight mods) and 1.2.1(1 modification from standard)
- Leveraging C* as DB
- Kong Alpine Docker
jeremyjpj0916
All 9 comments
Kong 1.2.1 ENV Var Config:
- name: KONG_HEADERS
value: latency_tokens
- name: KONG_PLUGINS
value: >-
kong-plugin-oauth,zipkin,kong-error-log,kong-oidc-implicit-token,kong-response-size-limiting,request-transformer,kong-service-virtualization,kong-cluster-drain,kong-upstream-jwt,kong-splunk-log,kong-spec-expose,kong-path-based-routing,kong-oidc-multi-idp,correlation-id,oauth2,statsd,jwt,rate-limiting,acl,request-size-limiting,request-termination,cors
- name: KONG_DNS_ORDER
value: 'LAST,SRV,A,CNAME'
- name: KONG_CASSANDRA_CONTACT_POINTS
value: 'server8171,server8176,server8180,server8173,server8175,server8172'
- name: KONG_LOG_LEVEL
value: notice
- name: KONG_PROXY_ACCESS_LOG
value: 'off'
- name: KONG_ADMIN_ACCESS_LOG
value: 'off'
- name: KONG_PROXY_ERROR_LOG
value: /dev/stderr
- name: KONG_ADMIN_ERROR_LOG
value: /dev/stderr
- name: KONG_ANONYMOUS_REPORTS
value: 'off'
- name: KONG_PROXY_LISTEN
value: '0.0.0.0:8000, 0.0.0.0:8443 ssl http2'
- name: KONG_ADMIN_LISTEN
value: '0.0.0.0:8001'
- name: KONG_MEM_CACHE_SIZE
value: 1024m
- name: KONG_SSL_CERT
value: /usr/local/kong/ssl/kongcert.crt
- name: KONG_SSL_CERT_KEY
value: /usr/local/kong/ssl/kongprivatekey.key
- name: KONG_SSL_CERT_DER
value: /usr/local/kong/ssl/kongcertder.der
- name: KONG_CLIENT_SSL
value: 'off'
- name: KONG_UPSTREAM_KEEPALIVE
value: '60'
- name: KONG_TRUSTED_IPS
value: '0.0.0.0/0,::/0'
- name: KONG_REAL_IP_HEADER
value: X-Forwarded-For
- name: KONG_REAL_IP_RECURSIVE
value: 'on'
- name: KONG_CLIENT_MAX_BODY_SIZE
value: 100m
- name: KONG_CLIENT_BODY_BUFFER_SIZE
value: 50m
- name: KONG_ERROR_DEFAULT_TYPE
value: text/plain
- name: KONG_DATABASE
value: cassandra
- name: KONG_PG_SSL
value: 'off'
- name: KONG_CASSANDRA_PORT
value: '9042'
- name: KONG_CASSANDRA_KEYSPACE
value: kong_dev
- name: KONG_CASSANDRA_TIMEOUT
value: '5000'
- name: KONG_CASSANDRA_SSL
value: 'on'
- name: KONG_CASSANDRA_SSL_VERIFY
value: 'on'
- name: KONG_CASSANDRA_USERNAME
value: *****
- name: KONG_CASSANDRA_PASSWORD
value: *****
- name: KONG_CASSANDRA_CONSISTENCY
value: LOCAL_QUORUM
- name: KONG_CASSANDRA_LB_POLICY
value: RequestDCAwareRoundRobin
- name: KONG_CASSANDRA_LOCAL_DATACENTER
value: DC1
- name: KONG_DB_UPDATE_FREQUENCY
value: '5'
- name: KONG_DB_UPDATE_PROPAGATION
value: '5'
- name: KONG_DB_CACHE_TTL
value: '0'
- name: KONG_DNS_HOSTSFILE
value: /etc/hosts
- name: KONG_DNS_STALE_TTL
value: '4'
- name: KONG_DNS_NOT_FOUND_TTL
value: '30'
- name: KONG_DNS_ERROR_TTL
value: '1'
- name: KONG_DNS_NO_SYNC
value: 'off'
- name: KONG_LUA_SSL_TRUSTED_CERTIFICATE
value: /usr/local/kong/ssl/kongcert.pem
- name: KONG_LUA_SSL_VERIFY_DEPTH
value: '3'
- name: KONG_LUA_SOCKET_POOL_SIZE
value: '30'
- name: KONG_ROUTER_CONSISTENCY
value: eventual
- name: KONG_DB_CACHE_WARMUP_ENTITIES
value: 'services,plugins,consumers'
Kong 1.0.3 ENV Config:
- name: KONG_PLUGINS
value: >-
kong-plugin-oauth,zipkin,kong-error-log,kong-oidc-implicit-token,kong-response-size-limiting,request-transformer,kong-service-virtualization,kong-cluster-drain,kong-upstream-jwt,kong-splunk-log,kong-spec-expose,kong-path-based-routing,kong-oidc-multi-idp,correlation-id,oauth2,statsd,jwt,rate-limiting,acl,request-size-limiting,request-termination,cors
- name: KONG_HEADERS
value: latency_tokens
- name: KONG_DNS_ORDER
value: 'LAST,SRV,A,CNAME'
- name: KONG_CASSANDRA_CONTACT_POINTS
value: 'server8342,server8344,server8346,server8348,server8419,server8350'
- name: KONG_LOG_LEVEL
value: notice
- name: KONG_PROXY_ACCESS_LOG
value: 'off'
- name: KONG_ADMIN_ACCESS_LOG
value: 'off'
- name: KONG_PROXY_ERROR_LOG
value: /dev/stderr
- name: KONG_ADMIN_ERROR_LOG
value: /dev/stderr
- name: KONG_ANONYMOUS_REPORTS
value: 'off'
- name: KONG_PROXY_LISTEN
value: '0.0.0.0:8000, 0.0.0.0:8443 ssl http2'
- name: KONG_ADMIN_LISTEN
value: '0.0.0.0:8001'
- name: KONG_MEM_CACHE_SIZE
value: 1024m
- name: KONG_SSL_CERT
value: /usr/local/kong/ssl/kongcert.crt
- name: KONG_SSL_CERT_KEY
value: /usr/local/kong/ssl/kongprivatekey.key
- name: KONG_SSL_CERT_DER
value: /usr/local/kong/ssl/kongcertder.der
- name: KONG_CLIENT_SSL
value: 'off'
- name: KONG_UPSTREAM_KEEPALIVE
value: '60'
- name: KONG_CLIENT_MAX_BODY_SIZE
value: 100m
- name: KONG_CLIENT_BODY_BUFFER_SIZE
value: 50m
- name: KONG_ERROR_DEFAULT_TYPE
value: text/plain
- name: KONG_DATABASE
value: cassandra
- name: KONG_PG_SSL
value: 'off'
- name: KONG_CASSANDRA_PORT
value: '9042'
- name: KONG_CASSANDRA_KEYSPACE
value: kong_stage
- name: KONG_CASSANDRA_TIMEOUT
value: '5000'
- name: KONG_CASSANDRA_SSL
value: 'on'
- name: KONG_CASSANDRA_SSL_VERIFY
value: 'on'
- name: KONG_CASSANDRA_USERNAME
value: *****
- name: KONG_CASSANDRA_PASSWORD
value: *****
- name: KONG_CASSANDRA_CONSISTENCY
value: LOCAL_QUORUM
- name: KONG_CASSANDRA_LB_POLICY
value: RequestDCAwareRoundRobin
- name: KONG_CASSANDRA_LOCAL_DATACENTER
value: DC1
- name: KONG_DB_UPDATE_FREQUENCY
value: '5'
- name: KONG_DB_UPDATE_PROPAGATION
value: '5'
- name: KONG_DB_CACHE_TTL
value: '0'
- name: KONG_DNS_HOSTSFILE
value: /etc/hosts
- name: KONG_DNS_STALE_TTL
value: '4'
- name: KONG_DNS_NOT_FOUND_TTL
value: '30'
- name: KONG_DNS_ERROR_TTL
value: '1'
- name: KONG_DNS_NO_SYNC
value: 'off'
- name: KONG_LUA_SSL_TRUSTED_CERTIFICATE
value: /usr/local/kong/ssl/kongcert.pem
- name: KONG_LUA_SSL_VERIFY_DEPTH
value: '3'
- name: KONG_LUA_SOCKET_POOL_SIZE
value: '30'
- name: KONG_TRUSTED_IPS
value: '0.0.0.0/0,::/0'
- name: KONG_REAL_IP_HEADER
value: X-Forwarded-For
- name: KONG_REAL_IP_RECURSIVE
value: 'on'
@jeremyjpj0916 since youre keeping a stable rate of requests in this benchmark, are you noticing any difference in p9x latencies?
p0pr0ck5
on 30 Jul 2019
Edit - This is irrelevant info since the most recent test below.
jeremyjpj0916
on 30 Jul 2019
Edit - So I threw wrk on the local nodes running kong to remove network:
Dev Core 1.2.1 Latency:
/ $ wrk -c 1 -t 1 -d 30 http://localhost:8000/F5/status
Running 30s test @ http://localhost:8000/F5/status
1 threads and 1 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 2.11ms 286.91us 11.35ms 92.58%
Req/Sec 477.33 32.10 530.00 72.33%
14266 requests in 30.03s, 3.18MB read
Requests/sec: 475.09
Transfer/sec: 108.49KB
/ $
Stage Core 1.0.3 Latency:
$ wrk -c 1 -t 1 -d 30 http://localhost:8000/F5/status
Running 30s test @ http://localhost:8000/F5/status
1 threads and 1 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 2.08ms 356.26us 18.84ms 95.84%
Req/Sec 483.32 29.77 550.00 75.67%
14440 requests in 30.02s, 3.22MB read
Requests/sec: 481.09
Transfer/sec: 109.86KB
/ $
@p0pr0ck5 to answer your question, no there is no concerning latency difference, only CPU. I watched MEM too and saw nothing concerning there either in my limited study of it.
jeremyjpj0916
on 30 Jul 2019
With our Stage environment upgraded to 1.2.1 as well I can confirm there too we went from the .41 CPU utilization rate to .525-.53 rate as well @ 120 tps. I will also test 1.2.1 without ANY of my modifications in sandbox and see if not having concurrency.lua modified reduces cpu utilization(my guess is that it won't as its not really a hot path thing for standard proxy traffic without bg changes occuring). Could just be the 1.0.3 I was running with the Bungle mods just has some efficiency advantages in my case, that maybe the OG 1.0.3 didn't have and 1.2.1 subsequently lack.
Would be nice to get that CPU back to the levels we saw though on 1.0.3 with those tweaks we been running for 4-5 months that have served well 馃槅. I will also try to test 1.0.3 without the Bungle mods and see what CPU utilization looks like there too to get a better picture. If it was the mods then all hail sir bungle and the crazy perf benefits we were seeing 馃ご, rip bungle homebrew 1.0.3 .
jeremyjpj0916
on 1 Aug 2019
Edit - not needed anymore.
jeremyjpj0916
on 7 Aug 2019
Actually I am changing my view here. Just ran a bunch of better/more extensive tests in an environment I am a bit more confident in the data on(leveraged an unused Stage environment 馃槅 )
By literally taking that environment and deploying bare 1.0.3 vs 1.2.1 with none of the patch work we run I notice these levels of CPU differences:
Kong 1.2.1: .37-.38 CPU used @ 112 TPS
Kong 1.0.3: .345-.35 CPU used @ 115 TPS
So while the difference is ever so slight in an environment I am more confident on than the data collected in my first trial run when I raised the issue, it is not enough to warrant investigation over that amount of difference.
Closing this so focus can be diverted elsewhere, there are better issues to dig into 馃憤 .
jeremyjpj0916
on 7 Aug 2019
@jeremyjpj0916 thanks for getting back to this and giving us an update!
hishamhm
on 7 Aug 2019
Related issues
hbagdi
路
3Comments
ghost
路
3Comments
Tieske
路
4Comments
hos7ein
路
3Comments
antoniott15
路
3Comments
Most helpful comment
Actually I am changing my view here. Just ran a bunch of better/more extensive tests in an environment I am a bit more confident in the data on(leveraged an unused Stage environment 馃槅 )
By literally taking that environment and deploying bare 1.0.3 vs 1.2.1 with none of the patch work we run I notice these levels of CPU differences:
Kong 1.2.1: .37-.38 CPU used @ 112 TPS
Kong 1.0.3: .345-.35 CPU used @ 115 TPS
So while the difference is ever so slight in an environment I am more confident on than the data collected in my first trial run when I raised the issue, it is not enough to warrant investigation over that amount of difference.
Closing this so focus can be diverted elsewhere, there are better issues to dig into 馃憤 .