Opa: Provide compute resource guidance for OPA deployments

[tsandall]

It's common for people to ask how many CPU and memory resources to give to OPA for specific use cases (e.g., admission control, microservice API authorization, etc.) The answer will depend on what the target latency should be (e.g., 1ms for microservice API authorization) and what kind of platform OPA is running on. It would be great if we could provide recommendations for typical platforms (e.g., Kubernetes on AWS) for common use cases.

[omerlh]

Hey, any update about this? I'm also experiencing high CPU usage and I wonder if it's normal. Such documentation could help me understand it better.

[tsandall]

@omerlh if you can provide some more detail? Like the original comment says, it's going to depend on a bunch of variables. There is no single correct answer.

[omerlh]

Yep, I was reading all the issues and we're actually considering to move to use GRPC instead of http to see if it will improve the performance.
We noticed that OPA takes up to 300 milicore per pod - which is a lot. I need to find more details about the impact on timing. Attached is the ppref output.

[omerlh]

I just took another CPU profile today, see it attached. During this time we had ~500 requests per second to the container. Hope this helps :)

[jin09]

Is OPA capable of leveraging multiple CPUs? Looks like opa run -s runs the server in single concurrency mode. How shall we leverage multiple cores with OPA server?

[tsandall]

@jin09 by default, the OPA server (opa run -s) uses all available cores. You can limit this with GOMAXPROCS. What makes you think the server is only capable of using a single core?

[jin09]

@tsandall pardon me for gaps in my understanding. I come from python background and have never worked on go. So when I spin the server (opa run -s) I see only a single process running (ps -elf | grep opa) which is why I thought maybe opa is single threaded. Do we use some prefork model, or is it something specific to golang? Probably you can redirect me to some resources where I could learn about it and get my concepts cleared.

Thanks

[tsandall]

Oh, I see. The wiki article for Go explains the concurrency model a bit
(it's based on what's called CSP--communicating sequential processes):
https://en.wikipedia.org/wiki/Go_(programming_language)#Concurrency:_goroutines_and_channels
.

Do we use some prefork model, or is it something specific to golang?

As far as I know, the Go runtime creates OS threads for your goroutines to
run on (up to one per core) and then schedules your goroutines to execute
on them. Preemption happens when you block on synchronization primitives
like channels (but I think that's an implementation detail). I'm not an
expert so take this with a grain of salt.

Probably you can redirect me to some resources where I could learn about it

and get my concepts cleared.

I'd take a look at the Tour of Go if you want to familiarize yourself with
Go and just search for topics you're interested in.

Happy golang-ing!

On Fri, Jan 10, 2020 at 1:09 PM Gautam Jain [e-mail]
wrote:

@tsandall https://github.com/tsandall pardon me for gaps in my
understanding. I come from python background and have never worked on go.
So when I spin the server (opa run -s) I see only a single process
running (ps -elf | grep opa) which is why I thought maybe opa is single
threaded. Do we use some prefork model, or is it something specific to
golang? Probably you can redirect me to some resources where I could learn
about it and get my concepts cleared.

Thanks

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/open-policy-agent/opa/issues/1601?email_source=notifications&email_token=AAB2KJNOP6UN6OTBUKJVFQLQ5C2XJA5CNFSM4IIB7CAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIUX7VY#issuecomment-573145047,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAB2KJMYU37YJJJKRJC2ZQTQ5C2XJANCNFSM4IIB7CAA
.

--
-Torin

[patrick-east]

Just to chime in on the python concurrency model vs golang. Python is limited to a single core by the interpreter and its global lock. It doesn't _actually_ run >1 thread in parallel, execution of threads can be interleaved but always on the same single CPU. That is why for Python applications to use >1 CPU you need to fork into multiple python processes (or put the python scripts behind something that does it for you like gunicorn or whatever). Essentially in python multi-threaded != multi-core, but multi-process can be multi-core. Hence why you are used to seeing >1 process if you check on the server.

Golang, like c/c++/java/etc, uses real system threads which the OS schedules over any of the CPUs on the host. This means that multi-threaded in these languages does imply it can/will be distributed over multiple CPU's.

There is a good post here https://stackoverflow.com/a/4496918/4789546 which gives more details