Druid: Broker can lock up due to defunct http connections

Created on 12 Dec 2013 · 19Comments · Source: apache/druid

HTTP connections from the broker to realtime or historical nodes can stop working but not get reset (perhaps they will be reset once something in the network stack times them out, but this could take hours). Any broker jetty threads that happen to be trying to talk to a defunct server will become stuck until the connections reset. It's apparently easy for this to happen to _all_ jetty threads when there are a lot of concurrent queries, which locks up the broker.

We could set a default timeout on broker -> compute requests. Or for slick extra credit, abort outgoing connections when the broker's ServerView realizes that a compute has gone missing.

Area - Querying Bug

Source

gianm

Most helpful comment

There is an undocumented config requirement:

[num of brokers] * broker:druid.processing.numThreads > historical:druid.server.http.numThreads

Otherwise you get unresponsive brokers very quickly under query load.

If you step back a sec, that seems very obvious. It's just not documented explicitly and I ran into it.

sixtus on 20 Jan 2014

👍2

All 19 comments

We've gone with setting a default timeout of 5 mins on broker --> compute requests for the time being. We'll keep this issue open and eventually push better cleanup of outgoing connections.

fjy on 16 Dec 2013

I ran into this immediately after deploying 0.6.42 in production :/

sixtus on 19 Dec 2013

This is getting really really bad now, it's like happening within minutes event though I got 4 brokers (instead of two when using druid-0.4)

sixtus on 14 Jan 2014

This issue only refers to behavior when there is an overabundance of broken connections, like connections to compute nodes that simply don't work or tcp connections stuck half-open due to compute crashes. I don't think any of that should be really common (hopefully not common enough to be "really bad" unless servers are constantly crashing) and am wondering if you are really seeing this issue or if you are seeing something similar but different.

What sort of symptoms are you seeing?

gianm on 14 Jan 2014

I see brokers simply not answering anymore.

My current working theory is the relationship of druid.broker.http.numConnections and druid.server.http.numThreads. Might that be it?

sixtus on 14 Jan 2014

This is also a day 1 druid issue. The default timeout that was introduced
can be turned off by setting druid.broker.http.readTimeout to null or a
large value.
On Jan 14, 2014 7:49 AM, "Gian Merlino" [email protected] wrote:

This issue only refers to behavior when there is an overabundance of
broken connections, like connections to compute nodes that simply don't
work or tcp connections stuck half-open due to compute crashes. I don't
think any of that should be really common (hopefully not common enough to
be "really bad" unless servers are constantly crashing) and am wondering if
you are really seeing this issue or if you are seeing something similar but
different.

What sort of symptoms are you seeing?

—
Reply to this email directly or view it on GitHubhttps://github.com/metamx/druid/issues/325#issuecomment-32276060
.

fjy on 14 Jan 2014

The query path goes like broker http thread -> broker compute connection pool -> compute http threads -> compute computation pools. Can you tell which resource is the one blocking things? Thread dumps and netstats during an outage are good ways of figuring that out.

If one of the broker resources is blocking things, then those properties you mentioned are a good place to start looking for a solution. If the compute resources look more problematic, there are also properties for number of compute http threads (druid.server.http.numThreads) and the size of the computation pool (druid.processing.numThreads).

gianm on 14 Jan 2014

There is an undocumented config requirement:

[num of brokers] * broker:druid.processing.numThreads > historical:druid.server.http.numThreads

Otherwise you get unresponsive brokers very quickly under query load.

If you step back a sec, that seems very obvious. It's just not documented explicitly and I ran into it.

sixtus on 20 Jan 2014

👍2

Do you mean greater than the total number of http threads across all
historical nodes, or greater than then number of http threads on each
historical node?

This requirement may also be very specific to your work-load. I believe it
would greatly depend on your cache hit-rate and how many queries actually
get sent down to historical nodes vs. how many results get pulled from
cache.

On Mon, Jan 20, 2014 at 1:22 AM, Hagen Rother [email protected]:

There is an undocumented config requirement:

[num of brokers] * broker:druid.processing.numThreads >
historical:druid.server.http.numThreads

Otherwise you get unresponsive brokers very quickly under query load.

If you step back a sec, that seems very obvious. It's just not documented
explicitly and I ran into it.

—
Reply to this email directly or view it on GitHubhttps://github.com/metamx/druid/issues/325#issuecomment-32744317
.

xvrl on 20 Jan 2014

Hey guys, is there still a problem or can I resolve it?

fjy on 14 Apr 2014

It seems it can be avoided by configuration. sum of brokers' http connections vs. historical's threads. broker locks up if larger.

sixtus on 14 Apr 2014

There was a problem at one time where requests were processing in the http thread pool instead of the processing thread pool. I thought there was a PR that fixed it but can't find it at this time. Is this still an issue?

drcrallen on 28 May 2015

I wonder could this become not an issue by design (and even configurations like "numConnections" and "numThreads" are removed) if we use asynchronous computations everywhere, because asynchronous piece of work doesn't lock the thread.

leventov on 11 Jan 2017

This issue was specifically about tcp connections to a dead historical node not getting reset or timed out in a timely manner, leading to queries involving that historical node (which might be all queries) getting stalled and then potentially stalling the entire broker (if all its query threads were taken up by such stalled queries).

@leventov I think you're suggesting that if we remove the "one query thread per query" limit, then even if we have a bunch of stalled queries, we could prevent the entire broker from getting stalled. That's true but we do need _some_ limit on concurrent queries, because each one requires memory (for result buffers, tracking involved data nodes, etc) and if we allow unlimited concurrent queries we will eventually run out of memory. See related discussion on #3810. We could probably increase the number of possible concurrent queries substantially though.

gianm on 11 Jan 2017

@leventov did you mean using Async request handling so that jetty threads serving requests are not blocked. In order to do that with Jersey, we would need to migrate to Jersey 2.x .

himanshug on 12 Jan 2017

👍1

@gianm in async model, we can have a limited number of processing threads on a historical/realtime node (we already have), and requests=events wait in the queue. After all it is not so much different from what we already have on historicals (not sure about realtime nodes). It is different on brokers, which don't queue/limit requests, delegating this to processing nodes.

leventov on 16 Jan 2017

@leventov I was referring to the brokers, they also need to allocate some memory per query for results merging (& other things, but results buffers are the biggest). A fixed set of brokers cannot have infinite capacity for concurrent queries, at some point they have to either reject new queries, block on new queries, or crash. But with better resource management the limits could probably be higher than they are today.

gianm on 16 Jan 2017

Is this still an issue in 0.15.0?

vogievetsky on 9 Jun 2019

I think it still exists in 0.18.1. We have three brokers. After we run benchmark with some heavy queries, I see one of the broker getting into a stale/unresponsive state, meaning, the broker process is running on that node but it doesn't respond. I tried executing query directly on this broker which times out and even /status check doesn't produce a response. When I stop the broker process to restart the service again, I see a bunch of WARN log messages about jetty QueuedThreadPool and an ERROR saying JettyServerModule - Jetty lifecycle event failed.

The complete error log is given below.

2020-06-11T08:26:27,889 INFO [Thread-43] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/internal-discovery/BROKER/prod-euclid-druid-queryserver1-1735672:8082]
2020-06-11T08:26:27,899 INFO [NodeRoleWatcher[BROKER]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher - Node[http://prod-euclid-druid-queryserver1-1735672:8082] of role[broker] went offline.
2020-06-11T08:26:27,900 INFO [Thread-43] org.apache.druid.curator.discovery.CuratorDruidNodeAnnouncer - Unannounced self [{"druidNode":{"service":"druid/broker","host":"prod-euclid-druid-queryserver1-1735672","bindOnHost":false,"plaintextPort":8082,"port":-1,"tlsPort":-1,"enablePlaintextPort":true,"enableTlsPort":false},"nodeType":"broker","services":{"lookupNodeService":{"type":"lookupNodeService","lookupTier":"__default"}}}].
2020-06-11T08:26:27,901 INFO [Thread-43] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER]
2020-06-11T08:26:57,908 INFO [Thread-43] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@3909a854{HTTP/1.1,[http/1.1]}{0.0.0.0:8082}
2020-06-11T08:26:57,909 INFO [Thread-43] org.eclipse.jetty.server.session - node0 Stopped scavenging
2020-06-11T08:26:57,913 INFO [Thread-43] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@186dcb05{/,null,UNAVAILABLE}
2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-162[timeseries_[euclid_cdo_live_hourly]_18f82248-c0a4-4a1b-b163-7a9626c9d91b],5,main]
2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-165[topN_[euclid_cdo_live_hourly]_549ddbd3-f42e-4f69-b75f-ee053d5bb0d5],5,main]
2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-177[topN_[euclid_cdo_live_hourly]_c99ddc98-0077-4d09-a53d-efd9f21eb5d9],5,main]
2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-159[topN_[euclid_cdo_live_hourly]_2cbbaac8-171b-4039-b8ec-78439c0ca563],5,main]
2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-175[topN_[euclid_cdo_live_hourly]_748433e0-294f-4df3-837b-3836b20d5ef5],5,main]
2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-163[topN_[euclid_cdo_live_hourly]_e41aeaac-5830-4e3e-9f78-1c5fbfca4656],5,main]
2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-182[topN_[euclid_cdo_live_hourly]_0381d5c9-d858-4c9a-aa57-6353e0334453],5,main]
2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-150[topN_[euclid_cdo_live_hourly]_51dd4cda-9fcf-4533-af6f-71d11568e4ae],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-169[topN_[euclid_cdo_live_hourly]_9a40b9f3-e77c-4393-af88-a949df285d61],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-181[topN_[euclid_cdo_live_hourly]_f3a2af6d-78cd-4ff5-8df6-909edb8040bb],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-184[topN_[euclid_cdo_live_hourly]_4ef89502-6e54-436a-b54e-47776a11ee31],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-155[topN_[euclid_cdo_live_hourly]_a4e2f32b-62a7-40b1-9160-54b36205908d],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-173[topN_[euclid_cdo_live_hourly]_c38ae97c-fb65-4b12-91c2-3fb8511d3939],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-180[topN_[euclid_cdo_live_hourly]_2049db4c-6796-4a69-acbc-f38a4fa1a666],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-149[topN_[euclid_cdo_live_hourly]_85f8f14e-7eda-4176-9408-b723728f2a8e],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-179[topN_[euclid_cdo_live_hourly]_6d83af84-35eb-4ecd-b40a-4e615061e0a0],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-174[topN_[euclid_cdo_live_hourly]_1e84dba4-80d5-4bbf-bc52-7373a3adc2fe],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-186[topN_[euclid_cdo_live_hourly]_2c476c10-7e41-45f7-8437-02bd6907d96d],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-161[topN_[euclid_cdo_live_hourly]_312791d2-704d-4b76-8104-8a87c2baa42a],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-166[topN_[euclid_cdo_live_hourly]_a2a2ddc7-cd92-4f99-85af-77cb4b7d050c],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-168[topN_[euclid_cdo_live_hourly]_b352dcb8-6dbc-4b1c-83bf-54f37427a122],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-171[topN_[euclid_cdo_live_hourly]_65d58a84-b67d-444a-b261-5947dec1def9],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-178[topN_[euclid_cdo_live_hourly]_37b5c479-cb6d-4e9e-9917-4b99d510653a],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-157[topN_[euclid_cdo_live_hourly]_14a9ad98-f3a5-407e-b183-f953f9f93df1],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-154[topN_[euclid_cdo_live_hourly]_aa2697cd-5e1c-419b-891b-1c1563dc07b2],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-167[topN_[euclid_cdo_live_hourly]_848e3e21-5da2-4449-a96f-9a79e8581cfa],5,main]
2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-183[topN_[euclid_cdo_live_hourly]_2b2727fa-bc35-455e-b947-bb90463965bb],5,main]
2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-160[topN_[euclid_cdo_live_hourly]_b3f4badf-f5da-4103-820e-37d97afba7d5],5,main]
2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-170[topN_[euclid_cdo_live_hourly]_714624b8-fd38-46a3-8044-17cb91f54acd],5,main]
2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-164[topN_[euclid_cdo_live_hourly]_1fd07432-53bd-433b-9cda-714660401bcc],5,main]
2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-156[topN_[euclid_cdo_live_hourly]_cf25e506-acd1-4979-bfe4-07c0b67b5547],5,main]
2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-158[topN_[euclid_cdo_live_hourly]_e3c5d7c0-0dc1-4098-a8cb-2d0ee3c04a4f],5,main]
2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-176[timeseries_[euclid_cdo_live_hourly]_7755967f-6778-4064-8a23-0b3bdb8742ac],5,main]
2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn't stop Thread[qtp510793091-172[topN_[euclid_cdo_live_hourly]_c20848de-6dde-43ae-86e2-4cfe54f8f126],5,main]
2020-06-11T08:27:27,920 ERROR [Thread-43] org.apache.druid.server.initialization.jetty.JettyServerModule - Jetty lifecycle event failed [class org.eclipse.jetty.server.Server]
java.util.concurrent.TimeoutException: null
    at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:129) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
    at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:30) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
    at org.eclipse.jetty.server.handler.AbstractHandlerContainer.doShutdown(AbstractHandlerContainer.java:181) ~[jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
    at org.eclipse.jetty.server.Server.doStop(Server.java:439) ~[jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) [jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
    at org.apache.druid.server.initialization.jetty.JettyServerModule$2.stop(JettyServerModule.java:450) [druid-server-0.18.1.jar:0.18.1]
    at org.apache.druid.java.util.common.lifecycle.Lifecycle.stop(Lifecycle.java:368) [druid-core-0.18.1.jar:0.18.1]
    at org.apache.druid.java.util.common.lifecycle.Lifecycle$1.run(Lifecycle.java:401) [druid-core-0.18.1.jar:0.18.1]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
2020-06-11T08:27:27,920 WARN [Thread-43] org.apache.druid.server.initialization.jetty.JettyServerModule - Unable to stop Jetty server.
java.util.concurrent.TimeoutException: null
    at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:129) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
    at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:30) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
    at org.eclipse.jetty.server.handler.AbstractHandlerContainer.doShutdown(AbstractHandlerContainer.java:181) ~[jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
    at org.eclipse.jetty.server.Server.doStop(Server.java:439) ~[jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
    at org.apache.druid.server.initialization.jetty.JettyServerModule$2.stop(JettyServerModule.java:450) [druid-server-0.18.1.jar:0.18.1]
    at org.apache.druid.java.util.common.lifecycle.Lifecycle.stop(Lifecycle.java:368) [druid-core-0.18.1.jar:0.18.1]
    at org.apache.druid.java.util.common.lifecycle.Lifecycle$1.run(Lifecycle.java:401) [druid-core-0.18.1.jar:0.18.1]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
2020-06-11T08:27:27,920 INFO [Thread-43] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL]
2020-06-11T08:27:27,924 INFO [Thread-43] org.apache.druid.sql.calcite.schema.MetadataSegmentView - MetadataSegmentView is stopping.
2020-06-11T08:27:27,924 INFO [Thread-43] org.apache.druid.sql.calcite.schema.MetadataSegmentView - MetadataSegmentView Stopped.
2020-06-11T08:27:27,925 INFO [LookupExtractorFactoryContainerProvider-MainThread] org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited. Lookup notices are not handled anymore.
2020-06-11T08:27:27,937 INFO [Thread-43] org.apache.druid.guice.LifecycleForkJoinPoolProvider - Shutting down ForkJoinPool [org.apache.druid.guice.LifecycleForkJoinPoolProvider@35bd2eec]

@gianm @sixtus Any idea how we can resolve this?