Druid: Adding Task Slot Count Metrics to Druid Overlord

[mghosh4]

Motivation

The primary motivation of this work is to provide more visibility into the task slot utilization over time. Monitoring utilization can help cluster administrators determine when to add/remove middlemanager/indexer nodes. With native ingestion, this has become even more important.

Proposed changes

I propose to add a new TaskSlotCountStatsMonitor in Overlord similar to TaskCountStatsMonitor class. It will expose the following metrics:

• taskSlot/total/count: Total number of task slots per emission period
• taskSlot/idle/count: Total number of task slots available for adding tasks
• taskSlot/used/count: Total number of task slots being currently used
• taskSlot/lazy/count: Total number of task slots that have been marked lazy
• taskSlot/blacklisted/count: Total task slots of peons that have been blacklisted

Proposed Design

I am planning to add the following apis in TaskRunner

  /**
   * APIs useful for emitting statistics for @TaskSlotCountStatsMonitor
  */
  long getTotalTaskSlotCount();

  long getIdleTaskSlotCount();

  long getUsedTaskSlotCount();

  long getLazyTaskSlotCount();

  long getBlacklistedTaskSlotCount();

The implementation for TaskSlotCountStatsMonitor will be similar to TaskCountStatsMonitor. It will use the TaskSlotCountStatsProvider interface which will be implemented by TaskMaster. TaskMaster will use taskRunner to emit the required metrics

Operational impact

We are adding some new metrics for better cluster monitoring.

Test plan

Other than unit tests, we are also planning to test this in our local Druid clusters. We will be using the statsd-based emitter framework to collect all the new emitted metrics for visualization.

[RoseDaniel]

This is a fantastic idea @mghosh4 !

[jihoonson]

Hi @mghosh4, thanks for the nice proposal!

• taskSlot/total/count: Total number of task slots per emission period
• taskSlot/idle/count: Total number of task slots available for adding tasks
• taskSlot/used/count: Total number of task slots being currently used

These 3 metrics will be definitely useful! To be honest, I can't believe we don't have these yet.

• taskSlot/lazy/count: Total number of task slots that have been marked lazy
• taskSlot/blacklisted/count: Total task slots of peons that have been blacklisted

I think there 2 metrics will fit better to middleManagers/indexers rather than task slots, since middleManager/indexers can be marked as lazy or blacklisted, not individual task slot. How about worker/lazy/count and worker/blacklisted/count (or indexingWorker/lazy/count and indexingWorker/blacklisted/count)?

[jihoonson]

• taskSlot/lazy/count: Total number of task slots that have been marked lazy
• taskSlot/blacklisted/count: Total task slots of peons that have been blacklisted

I think there 2 metrics will fit better to middleManagers/indexers rather than task slots, since middleManager/indexers can be marked as lazy or blacklisted, not individual task slot. How about worker/lazy/count and worker/blacklisted/count (or indexingWorker/lazy/count and indexingWorker/blacklisted/count)?

On a second thought, these 2 metrics seem useful as well since it is easier to compare with other metrics such as total or idle. But I would suggest fixing the descriptions since task slots are not marked as lazy or blacklisted, but middleManagers and indexers are. Maybe they can be "total number of task slots of lazy/blacklisted middleManagers and indexers".