Fasthttp: High number of context switches - Why?

[bullardla]

I made a simple server using fasthttp which accepts a request and simply returns a 200 response code. However, at a throughput of ~200,000 requests per second, I'm seeing around ~250,000 ctxsw/s. Even at ~100,000 requests per second, I'm seeing ~200,000 ctxsw/s. Is there an explanation for this? From my understanding, context switches are computationally expensive, and given the use of goroutines, I'm not fully understanding why context switches are necessary. Am I fundamentally misunderstanding something?

Server specs -
RAM - 2x 4GB Kingston 4GB DDR3 1Rx8
Processor - 3.5Ghz Intel XEON-E3-1270-V3-Haswell

[erikdubbelboer]

What exactly is ctxsw/s here? OS process context switches? How do you measure this? How are you running your server process? How are you generating the ~200,000 requests per second, are these generated on the same server? How many cores does your CPU have?

[bullardla]

ctxsw/s is context switches per second. I'm measuring this with the vmstat command.
I have a dedicated baremetal which is only running a fashttp server and vmstat, and I have a dedicated baremetal serving as the client. These are involuntary OS context switches. The baremetal server has 8 cores, but changing the number of cores still results in a large amount of context switches for the corresponding throughput.

For example, if I only enable 2 cores (echo 0 > /sys/devices/system/cpu/cpux/online), I'll still see ~80,000 ctxsw/s at a throughput of 20,000 requests per second. With four cores on, I'll see ~175,000 ctxsw/s at a throughput of 50,000 requests per second.

Also, yes, with no requests per second, the number of context switches is 0. Additionally, the number of context switches per second for a Vert.x server is, at most, ~2,500 ctxsw/s, whereas the maximum number for fasthttp is ~500,000 ctxsw/s.

[erikdubbelboer]

I want to say upfront that this has nothing to do with fasthttp (as it doesn't do anything process or thread related) but with the Golang runtime. But I would still like to help you and learn more about this.

With all 8 cores enabled can you check how many threads the fasthttp process has running? I'm wondering if its switching between those or between something else.
Are you setting GOMAXPROCS? I wonder what happens if you set that to 4 while having the 8 cores enabled.

Which version of Go are you using?

[bullardla]

I didn't override any environment variables, so GOMAXPROCS would've been the default, and I was using Go 1.13.7.

Unfortunately I don't have access to the environment anymore, but yeah, looking back at my results it seems that net/http has similar behavior.

I'll raise a question in the Go forum then, since it is strange to me that Go server frameworks have significantly more context switches than similar Java frameworks, despite apparent consensus that context switches are expensive and goroutines/green threads offer a solution to them.

Thanks for the help :smile:

[erikdubbelboer]

@bullardla do you have a link to the post for me? I would like to know what is happening as well 😄