Openwhisk: how to remove concurrency limits

[YangZhou1997]

Hi community,

I am trying to run thousands of function instances on OW. However, even I follow this method (https://stackoverflow.com/questions/52210036/openwhisk-increase-number-of-concurrent-requests) to set -e limit_invocations_per_minute=999999 -e limit_invocations_concurrent=999999 when ansible openwhisk.yml, the concurrency limit seems still existing. I find there will still be at most 8 function instances invoked by one invoker concurrently, which corresponds to 500 concurrency invocation per minute.

Any suggestion to remove the concurrency limits?

[style95]

@YangZhou1997 Could you share the spec of your invoker machines?
And how much time does it take to run your actions?

Generally, the number of concurrent actions are configured by invoker`s memory.

If you didn't change this value, the default would be 2GB which means 8 concurrent actions if you utilize the default container(256MB)
https://github.com/apache/incubator-openwhisk/blob/master/ansible/group_vars/all#L178

[YangZhou1997]

I have four machines hosting four invokers, each with 160GB memory and 40 cores.
It would take around 40s to run an action.

Thanks for your answering; that exactly solves my question! I will change the memory setting.

[YangZhou1997]

It seems that I cannot directly set the memory to 163840m, since the CPU-share would become lower than 2 and cause container creation crash. Please see the log from invoker:

[2019-05-13T02:46:09.470Z] [INFO] [#tid_437421f4eeb8e1fc5465be9581c85978] [DockerClientWithFileAccess] running /usr/bin/docker run -d --cpu-shares 1 --memory 256m --memory-swap 256m --network openwhiskOverlay -e __OW_API_HOST=https://10.10.1.2:443 --name wsk33_40_whisksystem_fib --pids-limit 1024 --cap-drop NET_RAW --cap-drop NET_ADMIN --ulimit nofile=1024:1024 --log-driver json-file --env __OW_ALLOW_CONCURRENT=True yangzhou1997/python3action:mpi (timeout: 1 minute) [marker:invoker_docker.run_start:8473] [2019-05-13T02:46:11.681Z] [ERROR] [#tid_437421f4eeb8e1fc5465be9581c85978] [DockerClientWithFileAccess] info: command was unsuccessful, code: 125 (unsuccessful), stdout: a115e5b3be467aa7ea1181111b00dcfe1b8ec8770af2cca58a3afb23dc39347f, stderr: WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. /usr/bin/docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:424: container init caused \"process_linux.go:390: setting cgroup config for procHooks process caused \\\"The minimum allowed cpu-shares is 2\\\"\"": unknown. [marker:invoker_docker.run_error:10684:2210]

40960m is a workable number I have tried, which, however, reduces the maximum memory capacity of my machine four times. I am further wondering if there is a feasible solution to solve this dilemma.

[style95]

Hm, it seems this is the issue.

But I think it should work with 131072 because cpushare is calculated based on this:
https://github.com/apache/incubator-openwhisk/blob/master/common/scala/src/main/scala/org/apache/openwhisk/core/containerpool/ContainerFactory.scala#L47

scala> (1024.0/(131072/256)).toInt
res8: Int = 2

And there is clearly the limit on the number of containers that can run on a host.
https://stackoverflow.com/questions/21799382/is-there-a-maximum-number-of-containers-running-on-a-docker-host

It seems the issue comes out from 1000 containers.
If you configure the memory as 131072, there can be 1024 containers with 128MB actions.
So I think it would be better to configure the smaller memory than this.
And it is obviously better to have a bigger number of invokers with smaller memory as Docker daemon cannot handle a huge number of concurrent requests in time.

Also, you need to leave some memory for the invoker itself as well.

Anyway, I think the value should be 2 in case the calculated value is smaller than 2.

[style95]

closing as #4477 is merged