Conductor: Worker Polling Intervals Guidance

[blueelephants]

I know "it depends" but can you share some "real-world-experience" how often worker microservices should poll for new tasks via GET /tasks/poll/{tasktype} at Conductor server?

What would be a guidance? Should it be every second or each 5 seconds, etc.?

Or to say it differently:

• Are there any performance numbers available how many request a conductor server of a specific hardware can handle (before it gets overwhelmed)?
• What would be a good monitoring way to pro-activley monitor Conductor Server to see whether it gets overwhelmed?

Many thanks

[v1r3n]

Polling interval should be set at less than a second, the poll API also supports long polling, which means, it can wait for set amount of time if no element is available in the task's queue before returning empty response. So I would use it in conjunction to set a very low polling interval (~100-500 millisecond) and long poll timeout to say 1 second. This combination works most effective as if there are items available, then you do not suffer from large polling interval, but if you do not have work items in the queue - excessive polling is avoided.

On the server side, Conductor server can scale horizontally and its main limits are going to be the number of IO operations your Dynomite cluster is configured to handle. We run a sustained load of 1000 requests per second on Conductor server with a 3 to 6 node cluster.

For the monitoring, you want to use the metrics published by the servlet container/application server for few items:

• % of thread utilization. Are the server threads starving?
• JVM heap utilization
• Time spent on GC

For the thread utilization, depending upon the application server in use, you can get the metric out using JMX or similar mechanism. We use Atlas for our metric collection.

[blueelephants]

Thanks for sharing your experience!