Druid: Fresh Docker install results in "Uh... I have no servers. Not assigning anything... "

[bbpaulvanharen]

I have a fresh VM running Ubuntu 18.04 LTS configured with 4 CPUs and 4 GB of RAM. I've installed Docker and downloaded the Druid archive from github.

After the command docker-compose -f distribution/docker/docker-compose.yml up the system starts loading and the container boots up. The Druid console is accessible at http://localhost:8888/. Configuring the data source to load the tutorial data goes as expected. After submitting the data spec, an error is displayed in the Ingestion window: _HTML Error: org.apache.druid.java.util.common.ISE: No default server found!_. Similarly, no data sources are shown in the Datasources window.
In the console of the Docker container there is frequent the message _Uh... I have no servers. Not assigning anything..._

Affected Version

Latest-n-greatest: Druid-0.19.0

Console snippet

coordinator | 2020-08-28T11:35:13,805 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments. coordinator | 2020-08-28T11:35:13,805 WARN [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorRuleRunner - Uh... I have no servers. Not assigning anything... coordinator | 2020-08-28T11:35:13,805 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues: coordinator | 2020-08-28T11:35:18,809 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments. coordinator | 2020-08-28T11:35:18,809 WARN [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorRuleRunner - Uh... I have no servers. Not assigning anything... coordinator | 2020-08-28T11:35:18,809 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues: coordinator | 2020-08-28T11:35:21,278 DEBUG [qtp23218037-120] org.apache.druid.jetty.RequestLog - 172.18.0.8 GET //172.18.0.4:8081/druid/coordinator/v1/rules HTTP/1.1 coordinator | 2020-08-28T11:35:22,064 INFO [DatabaseRuleManager-Exec--0] org.apache.druid.metadata.SQLMetadataRuleManager - Polled and found 1 rule(s) for 1 datasource(s) coordinator | 2020-08-28T11:35:22,518 INFO [org.apache.druid.metadata.SQLMetadataSegmentManager-Exec--0] org.apache.druid.metadata.SQLMetadataSegmentManager - No segments found in the database! coordinator | 2020-08-28T11:35:22,877 INFO [TaskQueue-StorageSync] org.apache.druid.indexing.overlord.TaskQueue - Synced 0 tasks from storage (0 tasks added, 0 tasks removed). coordinator | 2020-08-28T11:35:23,811 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments. coordinator | 2020-08-28T11:35:23,811 WARN [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorRuleRunner - Uh... I have no servers. Not assigning anything... coordinator | 2020-08-28T11:35:23,811 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues: coordinator | 2020-08-28T11:35:28,814 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments. coordinator | 2020-08-28T11:35:28,815 WARN [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorRuleRunner - Uh... I have no servers. Not assigning anything... coordinator | 2020-08-28T11:35:28,815 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues: coordinator | 2020-08-28T11:35:33,817 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments. coordinator | 2020-08-28T11:35:33,818 WARN [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorRuleRunner - Uh... I have no servers. Not assigning anything... coordinator | 2020-08-28T11:35:33,818 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues: coordinator | 2020-08-28T11:35:38,821 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments. coordinator | 2020-08-28T11:35:38,821 WARN [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorRuleRunner - Uh... I have no servers. Not assigning anything... coordinator | 2020-08-28T11:35:38,822 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues: router | 2020-08-28T11:35:40,046 ERROR [qtp1151370725-68] org.apache.druid.server.router.QueryHostFinder - WTF?! No server found for serviceName[druid/broker]. Using backup router | 2020-08-28T11:35:40,047 ERROR [qtp1151370725-68] org.apache.druid.server.router.QueryHostFinder - WTF?! No backup found for serviceName[druid/broker]. Using default[druid/broker] router | 2020-08-28T11:35:40,047 ERROR [qtp1151370725-68] org.apache.druid.server.router.QueryHostFinder - Catastrophic failure! No servers found at all! Failing request!: {class=org.apache.druid.server.router.QueryHostFinder} router | 2020-08-28T11:35:40,047 WARN [qtp1151370725-68] org.eclipse.jetty.server.HttpChannel - /druid/v2/sql router | org.apache.druid.java.util.common.ISE: No default server found! router | at org.apache.druid.server.router.QueryHostFinder.pickDefaultServer(QueryHostFinder.java:119) ~[druid-server-0.17.0.jar:0.17.0] router | at org.apache.druid.server.AsyncQueryForwardingServlet.service(AsyncQueryForwardingServlet.java:271) ~[druid-server-0.17.0.jar:0.17.0] router | at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0] router | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) ~[jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830] router | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655) ~[jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830]

[kirilgorbachov]

+1

[chintanp]

@bbpaulvanharen : Are all your other services up when the coordinator reports these logs? I found that my broker had exited, and I believe that is why the UI is not working and the coordinator gives this message. I am still trying to debug this, but broker logs say, jvm.config missing.

[graysonchao]

I ran into this issue too. Checking the broker logs, I saw that the last line before crashing was:

broker           | 2021-01-12T00:54:47,777 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[0] of size[500,000,000]

So I suspected the broker was dying due to OOM.

I increased the memory allocated to Docker on my machine from 2GB to 16GB, and it worked. Hope this helps!

[gianm]

Maybe we should update the docker-compose settings to use either the "nano" or "micro" settings from https://druid.apache.org/docs/latest/operations/single-server.html. (Or have two docker-compose config bundles, one for "nano" and one for "micro".) That would probably help. A PR to do that would be great.

By the way, anyone running into this: if you don't strictly _need_ Docker, I'd suggest using those scripts from https://druid.apache.org/docs/latest/operations/single-server.html.