Druid: [Proposal] Dynamic prioritization and laning

[gianm]

Motivation

(Side note- at various points in this proposal I talk about 'historicals', but by that I mostly mean 'historicals and any other nodes that brokers can fan out to, including task peons'. It would have been a mouthful to type out every time.)

Clusters sometimes have heterogenous workloads: imagine both 'light' queries that could run at interactive speeds and 'heavy' resource-intensive queries. Busy clusters with heterogenous workloads can become unresponsive, even to 'light' queries, when cluster resources are all tied up processing 'heavy' queries. It's frustrating for end users who are accustomed to 'light' queries typically running quickly. Starvation can happen for multiple kinds of resources:

1. The @Processing thread pool on historical nodes (druid.processing.numThreads). Druid does make an effort to have each processing task be bite-sized (each one just processes one segment), so even though they are not preemptible, they are amenable to prioritization. Currently, query prioritization (setting priority in the query context) only does one thing: control the execution order of tasks in the processing pool. When priorities are set properly, this is reasonably effective at avoiding starvation.
2. The HTTP server thread pool on historicals and brokers (druid.server.http.numThreads). The model here is that each query gets one thread.
3. The merge buffer pool on historicals and brokers (druid.processing.numMergeBuffers). Each groupBy query needs one of these on historicals, and may need some on brokers as well.
4. The HTTP client connection limit from brokers to historicals (druid.broker.http.numConnections). Each connection can only carry one query at a time.

Items 2, 3, and 4 are allocated per-query, meaning that they effectively act as limits on the number of concurrent queries that can run. If each historical node has, let's say, 30 HTTP server threads, then no more than 30 queries can run concurrently at a time on that server. If all 30 of these queries are 'heavy' then they will starve out 'light' ones.

In practice, the ways that people prevent starvation of the resources of items 2, 3, and 4 are to either set the limits high enough that they exceed the number of 'heavy' queries that might run at once, or to try to separate 'heavy' and 'light' queries onto different nodes (by issuing heavy vs. light queries to different brokers, and using historical tiers to create a tier that only handles 'light' queries). Neither of these is really a very satisfying approach, since it means cluster architecture must be closely adapted to the expected query workload, creating more work for cluster operators and reducing stability when workloads change unexpectedly.

There is one more problem: even though query prioritization (1) is effective at preventing starvation of the @Processing pool, it can be difficult to set priorities appropriately. It's generally doable when end users are accessing Druid through an API layer that is under tight control. But when end users access Druid directly, or are accessing Druid through a third party UI like Superset or Looker, it is not likely that priorities will be properly set.

Proposed changes

Concept

The idea is to establish dynamic prioritization and laning.

Dynamic prioritization (adjusting query properties automatically) is meant to address the problem that end users are not always in a good position to be able to set priorities effectively. Laning, as in fast and slow lanes, is meant to cap the number of concurrently-running low-priority queries, and thereby reserve guaranteed resources for high-priority queries. This makes query prioritization effective at preventing starvation across the entire query stack, not just in the @Processing thread pool.

New properties

|Property|Description|Where?|Default|
|-|-|-|-|
|druid.broker.priority.periodThreshold|An ISO8601 period. Queries whose intervals fall outside [now - periodThreshold, ∞) will have their priority adjusted by periodAdjustment.|Broker|null (ignored)|
|druid.broker.priority.periodAdjustment|Amount to adjust priority for queries who fall outside the periodThreshold. Negative numbers mean lower priorities and positive numbers mean higher. Generally a negative number would make more sense.|Broker|-1|
|druid.server.http.maxLowPriorityThreads|Maximum number of HTTP server threads to devote to queries with negative priority. This parameter must be set lower than all the limits in items 2, 3, and 4 above in "motivation" in order to prevent starvation.|Broker, Historical, Task|Integer.MAX_VALUE|

Altered query behavior

When a query comes in with priority less than zero, and the low-priority pool is full (there are maxLowPriorityThreads already running), reject the query with an HTTP 429 "Too Many Requests" code. Clients would be expected to do automatic backoff and retry. HTTP 503 "Service Unavailable" might be a more appropriate return code, but a less-common code like 429 makes it easier to know when to trigger the backoff/retry loop in Druid clients.

Don't ever reject queries with priority zero or higher.

This ensures that queries with non-negative priority cannot be starved out by queries with negative priority.

Rationale

For dynamic prioritization, the goal in this proposal is to keep it as simple as possible while still being useful. I think basing it totally off the interval does that.

For laning, while arriving at the idea to reject low-priority queries, I thought of a few other options and decided against them:

• Queue up queries past maxLowPriorityThreads rather than rejecting them. I didn't propose this since Druid's HTTP server design is thread-per-request, so a server thread is allocated upfront, and the only way I could see to give up the thread is to abort the query. The HTTP server could potentially be refactored to make it possible to queue up queries without hogging threads, but I think that would make more sense as future work. Of course, at some point, queries would still need to be rejected, since you can't queue an infinite number of requests.
• Allow low-priority queries to run past maxLowPriorityThreads, but preempt them if higher-priority queries come in. This is how the @Processing thread pool works, but I don't think it'd work for the other levels of the query stack, since things like HTTP threads, merge buffers, and broker -> historical connections are allocated upfront and cannot be reclaimed without aborting the query.

Operational impact

The new features would be optional and off by default, so there will be no operational impact to people that don't want to turn them on.

Test plan

This sort of thing would need to be tested on real clusters to provide some assurance that it is effective. I have a few in mind that would be good candidates.

Future work

• Improving the cost function that differentiates 'heavy' and 'light' queries, possibly by taking into account the number of rows, number of columns, expected filter selectivity, amount of computation estimated per row, and so on.

• Queue up low priority queries somehow, rather than rejecting them.

• Some way of setting the low-priority concurrency limit that is easier than druid.server.http.maxLowPriorityThreads. Setting a number of threads correctly requires cluster operators to think about many other different parameters on multiple node types.

[peferron]

I really like this proposal!

For context: we plan to make large interactive queries more responsive by running many low-priority background queries in order to populate a remote cache such as Redis or Memcached. This works for us because many of our large interactive queries are predictable, while smaller interactive queries are less predictable but are fast anyway so don't need to be cached.

Laning will ensure that these background queries don't impact interactive queries, and responding with 429 will make it much easier to throttle the submission of background queries in our caching service, so I love that.

Could you perhaps elaborate a bit more about why the dynamic prioritization scheme based on periodThreshold makes sense, as opposed to a durationThreshold for example? In our case, users may issue interactive queries that have a small interval duration (e.g. one hour) but are far in the past (e.g. six months ago).

[dylwylie]

Sounds great.

Instead of using the interval to decide how to change the priority of the query, what do you think about using the number of segments that the query operates over? Appreciate that this mechanism is to be expanded upon in future work but in the short term I think this works better because it more directly correlates with the amount of work a query has to do.

E.G.

Query A queries a huge datasource with 10 segments per-hour over 6 months, this requires ~40k segments to be queried and will be relatively slow without a huge cluster..

Query B queries a relatively sparse datasources that's been compacted nicely to have a single segment per day over 6 months, this requires ~200 segments to be queried and so in most clusters will be relatively quick.

[gianm]

@peferron,

Could you perhaps elaborate a bit more about why the dynamic prioritization scheme based on periodThreshold makes sense, as opposed to a durationThreshold for example? In our case, users may issue interactive queries that have a small interval duration (e.g. one hour) but are far in the past (e.g. six months ago).

I was thinking a period threshold is likely to align with how people often set up historical tiers: a 'hot' tier for the latest 30 days and a 'cold' tier for older data.

@Dylan1312,

Instead of using the interval to decide how to change the priority of the query, what do you think about using the number of segments that the query operates over? Appreciate that this mechanism is to be expanded upon in future work but in the short term I think this works better because it more directly correlates with the amount of work a query has to do.

Well, same :)

Maybe it's worth creating more than one threshold even in the first version. It sounds like there are is demand for ways of thinking about prioritization beyond a 'hot' / 'cold' tier setup. How about starting with the three that have been suggested: period (from now), duration (of the query interval), and number of segments? In this case, if a user sets multiple, I'd suggest applying the adjustment _once_, if _any_ of the thresholds triggers. I don't see a reason that a query that only triggers one threshold to be prioritized above one that triggers two. Concretely the properties would be,

• druid.broker.priority.periodThreshold
• druid.broker.priority.durationThreshold
• druid.broker.priority.segmentCountThreshold
• druid.broker.priority.adjustment

I think we'll want to add number of rows in the future, but that information is tougher to come by right now (the broker doesn't cache it unless Druid SQL is enabled, and it only lives in the SQL module). So I would still leave that one for the future.

[peferron]

I was thinking a period threshold is likely to align with how people often set up historical tiers: a 'hot' tier for the latest 30 days and a 'cold' tier for older data.

Got it. It makes sense if the goal is to be an alternative to tiering in certain situations, as you mentioned in your proposal.

Some of the confusion, I think, is coming from the fact that the proposal matches "light" with "high priority", and "heavy" with "low priority", but then goes on to propose a period threshold that's intuitively not the best way to achieve that: heavy queries hitting recent data will remain high priority, while light queries hitting older data will be adjusted down to low priority.

That's why the comments went on to suggest thresholds that are intuitively better correlated, such as duration or segment count (and we could keep going until we end up applying ML to it).

Let me ask you this: if there was an omniscient function available in the broker that could exactly predict how light or heavy a query is going to be, would you remove the period threshold, or leave it in as an optional configuration option?

If you remove it, then this proposal is indeed about classifying light vs heavy queries.

If you leave it in, then this proposal is rather about letting users dynamically adjust priorities based on abitrary criteria, with light or heavy just being one of them; and it's OK to sometimes classify a heavy query as high-priority and vice-versa.

I hope this doesn't sound like nitpicking—it would be helpful (at least to me) to clear this up.

BTW, my suggestion of a duration threshold was only as a review comment—in our case, we fully control the query priority and don't have any need for dynamic prioritization, so laning is the only useful feature for us here. My selfish interest would even be to ship laning first and push dynamic prioritization to a follow-up PR. 😛

[gianm]

Let me ask you this: if there was an omniscient function available in the broker that could exactly predict how light or heavy a query is going to be, would you remove the period threshold, or leave it in as an optional configuration option?

It depends on how omniscient the function is :)

Depending on how omniscient the function is, I still think there'd be value in overriding its decisions. Imagine you have a 'hot' and 'cold' tier and you really do want to apply a priority penalty to anything that'd hit the 'cold' tier, even if it looks like a cheap query. The reason might be because you have set up the cold tier to be storage-dense, and even seemingly-cheap queries might take a long time to run there due to needing to swap data in and out of disk. And you want those to go in the 'heavy' lane even if they look cheap.

If you leave it in, then this proposal is rather about letting users dynamically adjust priorities based on arbitrary criteria, with light or heavy just being one of them; and it's OK to sometimes classify a heavy query as high-priority and vice-versa.

I think this is the best way to think about what I'm proposing. The main motivation for dynamic prioritization is to differentiate heavy vs light queries, but I don't think that's going to be the only way people will want to do it. Maybe people will want to adjust priorities for specific datasources, or queries from specific users, or whatever.

BTW, my suggestion of a duration threshold was only as a review comment—in our case, we fully control the query priority and don't have any need for dynamic prioritization, so laning is the only useful feature for us here. My selfish interest would even be to ship laning first and push dynamic prioritization to a follow-up PR. 😛

That's a good point, the two features don't need to be done at the same time, and probably shouldn't be. (Although there is value in designing them together to make sure they align with each other.)

[leventov]

Since HTTP threads don't do any serious work on historicals (and indexing nodes?) (it's all delegated to @Processing threads) could we move to async in this particular part of the system and eliminate the druid.server.http.numThreads option?

[gianm]

I think we could, but it should be a different proposal. Even in an async http world, we still need some kind of limits on how many concurrent queries can be running simultaneously, since each one does need to allocate per-query memory for merging / etc and there is not an infinite supply. So most of the stuff in this proposal would still be useful.

I think the main thing I'd change about this proposal if async http existed is that instead of going straight to returning HTTP 429 when maxLowPriorityThreads queries are running, I would instead queue them up but delay them starting for a while. We will still need to reject at some point, but it'd be later. I'd probably also call the config maxLowPriorityQueries rather than maxLowPriorityThreads.

[sascha-coenen]

I LOVE LOVE LOVE the motion to work on dynamic prioritisation and laning and I also like these two terms.

Actually I was thinking about the same feature for a long time. I agree to your problem statement BUT I would model the problem quite differently. I believe to have an alternative proposal to make which would be similarly easy to implement but lead to a superior system behaviour (more automatic fairness, more control authority by admin, more self-healing) I think it might be looked at as being combinable into the existing proposal and some parts of my proposal below can be considered optional.

---

As was already stated, internally, Druid already has a query prioritisation mechanism, but out of the box, this priority can only be specified by a client as a static value.

Asking for a "dynamic prioritisation" is already implicitly formulating that there are several aspects that influence a query's priority.

(Some of these are intrinsic - meaning that Druid can compute them itself. Some are extrinsic - they are known only by an admin/client/user.)

(a) [extrinsic] the user context/intention

• how urgent is the query
  is the query originating from an interactive system or is it a report query that may take a long time.
• how important is the user - CEO/VIP/customer/employee/backend-process

(b) [intrinsic] the query's weight

• an estimation of how long it takes to execute or how much computational power is needed

(c) [intrinsic] the system state

• how much of the query result is already cached
• is the system idle or under heavy load

(d) [intrinsic] the system/user's history

• how much strain has a user put onto the system recently. One might want to down-prioritise queries originating from a heavy-hitter over time or after a quota is reached to give other more modest users also a fair share.

---

Alternative Proposal
Let me say for each of the above how I would model them individually and then how those pieces could be put together:

(a) "the extrinsic context"
To model the extrinsic input, a query should be associated with a logical token that an admin can map to certain parameterizations. I'm used to calling such a logical token a "resource-pool" although I don't consider it a good name. Basically, instead of directly setting a "priority" inside of a query's context object, I would instead propose to specify a freely chosen logical resource pool name in the same manner.

{
    "type": "topN",
    "context": {
        "resource-pool": "interactive-customer-query"
    }
}

An admin can pre-configure resource-pools and then queries can refer to them. An unknown resource pool or missing declaration could then map to a "default" resource pool.

(b) "computing a query weight"

• given an incoming query, a broker would first determine a weight for it.
• one could make this mechanism pluggable to have a smooth migration path.
  -- A simple strategy would assign each query the same weight.
  -- A legacy strategy could set the query weight to be equal to the query priority set in the query context object.
  -- A similarly simple strategy could look up a static query weight from a resource-pool.
  (This would then be functionally equivalent to the current way of setting priorities in the query context)
  -- Even a realistic query weight is rather straightforward and easy to implement:

    weight := #segments * #metrics * #splits * selectivity

    # stands for number-of
    selectivity means which fraction of records need to be scanned due to the given filter condition.

(c) "the system state"

• similar to a query weight, the system state can also be modelled as another weight that can be combined with a query-weight and other inputs to form an eventual query priority.
• this is where Gian's proposal plugs in.
• (c1) "system is under heavy load"
  --this increases the weight and ends up mapping more queries to low priorities.
• (c2) "query results are partially cached"
  -- if a certain fraction of a query is cached, this should reduce the query weight.
  -- if a broker can determine which fraction of a query is cached it would reduce the query weight accordingly
  query-weight := query-weight * %cached
  -- the broker then sends the query along with its weight to the historical, which can optionally further discount the given query weight by observing it's local cache situation.

(d) "the system/user's history"

• an additional weight can be computed based on a user's or resource-pool's query history this weight acts as a penalty.
• a broker could keep a temporally-discounted sum over all query-weights per connection or per resource pool (or both)
• due to a mathematical argument, there exists only a single temporal discounting function which is rational/consistent, which is exponential temporal discounting. This is nice two times over because a) it means one does not need to make this weighting strategy pluggable necessarily and b) the exponential function has nice computational properties which allow for very efficient incremental computation of an "eternal running sum"
  temporal-weight := sum ( weight(query[i]) * exp( now() - time(query[i]) ) )

---

Putting those pieces together:

• a client annotates a query with a resource pool name like "my-pool" instead of directly specifying a static priority.
• a broker receives the query and computes a weight
  weight := query-weight * system-weight * temporal-discount
• broker sends query and weight to the historicals. Alternatively, the broker can turn the weight into a priority and send this.
• historicals receive query along with weight and either treat the weight as a priority and follow their current scheduling logic, or alternatively, in a sophisticated setup, a historical could further modulate the weight based on its local conditions like "local load condition" or "local cache"
• the broker also uses the resource pool name to lookup configuration attributes which can influence/modulate the weight computations, for instance:
  -- base-offsets == laning
  -- weighting factors
  -- directives as to whether to engage or ignore a certain criteria or stage in the weight computation.
  -- specification of which pluggable strategy to use.
  -- half-life for exponential temporal discounting.
  -- in short, any parameterization for the process of computing a dynamic query priority.

---

Although it might seem like a nice-to-have component, it is especially the incorporation of an exponential temporal discounting that would lead to a self-healing system that also has automatic fairness.
There would never have to any query that need to be rejected. Queries might simply get such high weights that it will be never their turn to compute which might eventually time them out.
Likewise, users pinging the system to death with massive amounts of queries will not be able to impact other more modest users.

All of the above might sound complex on paper but even if all got implemented, this would still be very straightforward and easy to do and would constitute a minor effort.

There are more sophisticated solutions, which allow a query to move from one lane to another one.

Lastly, although I agree that one might want to start small, I consider it useful and necessary to try to formulate a "true north" vision nonetheless - an ideal, fully fledged solution. One can then still implement only part of it, but it allows for validating whether that first step is well aligned with what potential later steps one might want to take.

[gianm]

@sascha-coenen, thanks for your feedback. After reading it, it sounds to me like most of it can be done as additions to my original (and intentionally minimalist) proposal, so I am not experiencing concern that the first step would be misaligned with a more sophisticated solution. But would you please let me know what you think, and in particular, what you think of the below comments?

On resource pools,
I like the resource pool idea as a way of incorporating extrinsic information. I think it maps nicely onto how people want to think about query priorities: numbers don't really make sense, but interactive or api or scheduled-report do make sense. It's more of a pain to configure, though. Rather than a single number on the query context, there's an indirection from the query context to an identifier defined somewhere else (broker configuration? metadata db?).

But even though it's a pain, I think it's a useful feature. In the context of my original proposal they would be a good place to define overrides for default priority and for the druid.broker.priority.* properties. (You might have more aggressive or more lenient thresholds for different kinds of queries.)

You're right about the name, it's not a good one based on how you've conceived it (it's not really a pool of resources, it's more like a way to group together queries that should be treated similarly).

I don't think they'd need to be added immediately, it makes sense to me to do them as a follow-on that lets you override the systemwide defaults.

On incorporating multiple sources of intrinsic information,
I think while my proposal is extraordinarily basic as far as what intrinsic information it incorporates (intentionally so, to keep the initial implementation simple), I hope more could be added later. I think all the stuff in your proposal could be done as follow-ons if we wanted. Since mine is so simple, it won't leave much legacy cruft behind that a more sophisticated system would have to deal with. Just the druid.broker.priority.* properties.

On rejecting queries,
I think this is the biggest area where I'm not sure how to align your proposal with mine.

The main conflict I see is that you envision not needing to reject queries, but I think we do need to reject queries, if only to prevent http server thread starvation. Due to this issue, I don't think we can do more than two 'lanes': a fast lane (queries in this category are always accepted for processing) and a slow lane (queries are rejected if the lane is 'full'). This constrains how beautiful the solution can be, since a binary decision must be made by the broker (reject a query, or accept it and begin execution).

By the way, http server thread starvation isn't the only resource that I think we need a binary laning system for: others include concurrent connections from brokers to historicals, and memory used for merging and storing partial results on historicals (both offheap merge buffers + on-heap structures). All of these resources are limited and must be reserved early in the query processing lifecycle, and cannot be released without canceling the query.

[leventov]

Two somewhat related issues: #8356 and #8357.

[ccaominh]

I'm planning to implement this proposal.

I think one potential area is of future improvement (that @sascha-coenen already alluded to), is allowing queries to move between lanes. For example, the heuristics mentioned may misidentify a heavy query as a light one, which may cause starvation of later light queries. In this case, the heavy query could possible be moved to the correct lane after it's consumed too many resources. Similarly, if a light query is initially misidentified as a heavy one, its priority would need to be corrected either before timing out or when it is retried later.

[leventov]

I believe that "Dynamic prioritization and laning" and "Query execution interleaving" (#8356) can help each other so the implementations for them should be better co-designed, even if the implementations will come in separate PRs.

More generally, this pair of issues and https://github.com/apache/incubator-druid/issues/4773 (to control which segments can and can not be in the file cache) should make separating Druid clusters into multiple tiers completely unnecessary.

[sascha-coenen]

On resource pools,
You're right about the name, it's not a good one based on how you've conceived it (it's not really a pool of resources, it's more like a way to group together queries that should be treated similarly).

True. How I got into referring to this feature as "resource pools" is that I was borrowing the name from Vertica. (https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/AdministratorsGuide/ResourceManager/ManagingWorkloads.htm?tocpath=Administrator%27s%20Guide%7CManaging%20the%20Database%7CManaging%20Workloads%7C_____0)

In Vertica one can define separate resource pools that have attributes like how much bandwidth they may use etc. A user account can be associated with a resource pool and resource pools can be set up in such a way that a query can start in one resource pool and can be moved to a different resource pool on certain conditions like if it hasn't completed within a given amount of time.
This goes along the same lines of what ccaominh has described above.

Recently, Amazon announced "Automatic workload management and query priorities " for their AWS Redshift product:
https://aws.amazon.com/about-aws/whats-new/2019/09/amazon-redshift-announces-automatic-workload-management-and-query-priorities/
They even boast to use machine learning. I don't like the product but the naming "automatic workload management" is nice and it is also interesting that it has "automatic" in it, suggesting that resource management is actually something that doesn't require that much tuning and administration but something that kind of can be done almost in a mathematically optimal way.

I think the current Druid mechanism with query priorities is quite nice, but a priority that is 1 higher than another priority is already completely dominating the query prioritisation.
So whether using names or numbers to denote a priority, the prioritization should be gradual, not absolutist. Currently, as long as there are segments for a prio:10 query in the processing queue, no single segment for a prio:9 query will be processed. Instead I would think that a probabilistic interpretation of a numeric priority would be smoother. One could then put a higher-level construct on top of that. For instance a query analyzer which can compute how heavy a query is and can map that to a priority which makes sure that the query is neither allowed to starve nor to slow down interactive queries, simply by mapping it to the right numeric priority.
Such an approach would be nice in that it is such a generic concept that sophisticated features could be realised later on on that foundation. If on the other hand an "engineered" approach is taken as a foundation, like a fixed, predefined number of lanes with fixed, predefined behaviours, then this limits how many later innovations could be put on top of it.

--

More generally, this pair of issues and #4773 (to control which segments can and can not be in the file cache) should make separating Druid clusters into multiple tiers completely unnecessary.

Interesting! Is it really true that tiers could be made "completely" unnecessary? We are using tiers for fail safety, having one tier running in one availability zone and a tier holding replicas running in another availability zone. Would k-safety and multi-AZ cases also be possible in a world without tiers?

[leventov]

@sascha-coenen

Is it really true that tiers could be made "completely" unnecessary? We are using tiers for fail safety, having one tier running in one availability zone and a tier holding replicas running in another availability zone. Would k-safety and multi-AZ cases also be possible in a world without tiers?

After #4773 is done (or even before that), it's pretty straightforward to implement the transparent lazy download of segments from the deep storage on Historicals upon they are first queried. This, in turn, opens up the technical possibility to implement spinning up Historical tiers which are exact clones of other tiers in a matter of minutes rather than dozens of minutes or hours.

When you are able to clone the whole Historical tier in a few minutes, I think keeping a backup tier in a different AZ is not optimal in almost all cases except the rare ones with very stringent availability requirements (this is not typical for Druid which is used in analytics workloads which are rarely that critical).

[gianm]

True. How I got into referring to this feature as "resource pools" is that I was borrowing the name from Vertica. (https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/AdministratorsGuide/ResourceManager/ManagingWorkloads.htm?tocpath=Administrator%27s%20Guide%7CManaging%20the%20Database%7CManaging%20Workloads%7C_____0)

The Vertica resource pools are a bit more pool-y, they have things like MAXCONCURRENCY, MAXQUERYMEMORYSIZE, and PRIORITY that look like they correspond pretty closely to Druid concepts (see documentation for CREATE RESOURCE POOL).

The RUNTIMECAP + CASCADE TO mechanism doesn't correspond to any current Druid concept, but maybe it should. It is trying to solve the same problem that @ccaominh brought up.

Such an approach would be nice in that it is such a generic concept that sophisticated features could be realised later on on that foundation. If on the other hand an "engineered" approach is taken as a foundation, like a fixed, predefined number of lanes with fixed, predefined behaviours, then this limits how many later innovations could be put on top of it.

Would it really be that tough to make the system more flexible in the future? I'd imagine giving the fast and slow lanes special names and then allowing people to define more. We might have to deprecate a couple of config properties, but that doesn't seem too bad.

[asdf2014]

@gianm I am looking forward to this new feature! :+1:

An ISO8601 period. Queries whose intervals fall outside [now - periodThreshold, ∞) will have their priority adjusted by periodAdjustment.

In addition, I also have some related thinking. Can we calculate the approximate elapsed time of the request before it actually executes? Then we can start adjusting the priority without waiting for the request runtime to reach the periodThreshold threshold.

Recently, Amazon announced "Automatic workload management and query priorities " for their AWS Redshift product

And similar to Amazon's WLM products, it feels that the introduction of machine learning will be a big trend, whether it is automatic load balancing in this proposal, or as a time series anomaly detection that may be developed in the future.