Druid: Error in coordinator/overlord when loading scan-query

[KenjiTakahashi]

I'm getting errors like this

2017-09-21T21:17:34.025034+00:00 monitowl-dev wh_druid_overlord[27859]: Exception in thread "main" com.google.inject.CreationException: Unable to create injector, see the following errors:                                      
2017-09-21T21:17:34.025617+00:00 monitowl-dev wh_druid_overlord[27859]: 1) No implementation for io.druid.query.GenericQueryMetricsFactory was bound.                                
2017-09-21T21:17:34.026017+00:00 monitowl-dev wh_druid_overlord[27859]:   while locating io.druid.query.GenericQueryMetricsFactory                                                   
2017-09-21T21:17:34.026441+00:00 monitowl-dev wh_druid_overlord[27859]:     for the 1st parameter of io.druid.query.scan.ScanQueryQueryToolChest.<init>(ScanQueryQueryToolChest.java:48)                                          
2017-09-21T21:17:34.026811+00:00 monitowl-dev wh_druid_overlord[27859]:   at io.druid.query.scan.ScanQueryDruidModule.configure(ScanQueryDruidModule.java:36) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.query.scan.ScanQueryDruidModule)
2017-09-21T21:17:34.027141+00:00 monitowl-dev wh_druid_overlord[27859]: 2) No implementation for io.druid.query.GenericQueryMetricsFactory was bound.
2017-09-21T21:17:34.027372+00:00 monitowl-dev wh_druid_overlord[27859]:   while locating io.druid.query.GenericQueryMetricsFactory                                                   
2017-09-21T21:17:34.027621+00:00 monitowl-dev wh_druid_overlord[27859]:     for the 1st parameter of io.druid.query.scan.ScanQueryQueryToolChest.<init>(ScanQueryQueryToolChest.java:48)
2017-09-21T21:17:34.027988+00:00 monitowl-dev wh_druid_overlord[27859]:   while locating io.druid.query.scan.ScanQueryQueryToolChest
2017-09-21T21:17:34.028260+00:00 monitowl-dev wh_druid_overlord[27859]:     for the 1st parameter of io.druid.query.scan.ScanQueryRunnerFactory.<init>(ScanQueryRunnerFactory.java:52)
2017-09-21T21:17:34.028492+00:00 monitowl-dev wh_druid_overlord[27859]:   at io.druid.query.scan.ScanQueryDruidModule.configure(ScanQueryDruidModule.java:41) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.query.scan.ScanQueryDruidModule)
2017-09-21T21:17:34.028749+00:00 monitowl-dev wh_druid_overlord[27859]: 2 errors
2017-09-21T21:17:34.028975+00:00 monitowl-dev wh_druid_overlord[27859]: #011at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470)
2017-09-21T21:17:34.029211+00:00 monitowl-dev wh_druid_overlord[27859]: #011at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:155)
2017-09-21T21:17:34.029438+00:00 monitowl-dev wh_druid_overlord[27859]: #011at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:107)            
2017-09-21T21:17:34.029665+00:00 monitowl-dev wh_druid_overlord[27859]: #011at com.google.inject.Guice.createInjector(Guice.java:99)
2017-09-21T21:17:34.029944+00:00 monitowl-dev wh_druid_overlord[27859]: #011at com.google.inject.Guice.createInjector(Guice.java:73)                                                 
2017-09-21T21:17:34.030185+00:00 monitowl-dev wh_druid_overlord[27859]: #011at com.google.inject.Guice.createInjector(Guice.java:62)
2017-09-21T21:17:34.030399+00:00 monitowl-dev wh_druid_overlord[27859]: #011at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:390)               
2017-09-21T21:17:34.030678+00:00 monitowl-dev wh_druid_overlord[27859]: #011at io.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:63)
2017-09-21T21:17:34.030892+00:00 monitowl-dev wh_druid_overlord[27859]: #011at io.druid.cli.ServerRunnable.run(ServerRunnable.java:40)                                               
2017-09-21T21:17:34.031104+00:00 monitowl-dev wh_druid_overlord[27859]: #011at io.druid.cli.Main.main(Main.java:108)

while having scan-query extension in druid.extensions.loadList, which is in the _common/common.runtime.properties file.

I've confirmed that this does not affect historicals, nor brokers and that removing scan-query from extensions fixes the problem.

Is this expected? Should I be splitting the extensions definitions for different nodes now? What I'm worried about that our local tasks [or middle managers, or whatever actually serves incoming data :-)] (spawned by overlord) will not see it, then, which will be no good.

[KenjiTakahashi]

Oh, I forgot! This is on 0.10.1.

[stevenchen3]

Got the same here.

[almike]

yeah this is a major issue for us as well

[vogievetsky]

It appears that in Druid 0.10.1 (vs 0.10.0) non-query nodes (coordinator, overlord) can not load query extensions.
Previously in 0.10.0 they were simply ignored.
This is technically a regression I pray to the Druid gods that it will be fixed in 0.11.0
There is a very simple work-around for now: only load the scan query on the query nodes, broker, historical, etc.

[almike]

thanks @vogievetsky, we will try that out!

[KenjiTakahashi]

@vogievetsky Thanks, that's what I figured out as well. Still not sure about my question at the bottom of the first post, though. It is not clear to me what will happen in such scenario.

[leventov]

It appears that in Druid 0.10.1 (vs 0.10.0) non-query nodes (coordinator, overlord) can not load query extensions.
Previously in 0.10.0 they were simply ignored.
This is technically a regression I pray to the Druid gods that it will be fixed in 0.11.0
There is a very simple work-around for now: only load the scan query on the query nodes, broker, historical, etc.

This is caused by #4429 and IMO it's not a regression, it's a first step towards not loading everything everywhere. So not loading scan-query on coordinator and overlord is not a "workaround", it's "the right thing to do".

[gianm]

@leventov I don't agree, it's a real pain for users if they need to specify a different list of extensions for each node type. Especially since new users often don't fully understand what node types are doing what jobs. In the past, extensions could be loaded everywhere and they would just end up doing nothing if they don't apply to a particular node type.

I guess it does make sense to not actually load everything everywhere for efficiency sake, but for users' sanity, how about instead having extensions be able to declare what kinds of node types they should be loaded on.

[KenjiTakahashi]

I am fine with specifying different extensions for different nodes (although @gianm argument makes sense, too), I just need to know what will happen when a task spawned locally by overlord needs extension that overlord itself cannot load? My understanding is that currently, overlord's config is being passed down to local tasks.

BTW: The scan-query case should "fix itself" with the next release, because it got merged to core.

[gianm]

I just need to know what will happen when a task spawned locally by overlord needs extension that overlord itself cannot load?

@KenjiTakahashi, if the extension must be serialized as part of the task (for example a custom firehose) then the overlord must be able to load it. The scan query isn't needed to be serialized as part of the task, so, it doesn't need to be loaded on the overlord.

[KenjiTakahashi]

Yes, but how will a task know that it has to be loaded? Maybe I'm not being clear here, sorry. I mean we're using the index_realtime tasks spawned locally on overlord. There is no separate properties file for tasks/middlemanagers/\. But still I should be able to query the incoming data, right? How would I use scan-query with that, if it's not specified in overlord config?

[leventov]

@KenjiTakahashi you can configure custom properties for forked task JVMs, but not overlord using druid.indexer.fork.property, see http://druid.io/docs/latest/configuration/indexing-service.html

[KenjiTakahashi]

So I can pass any property through this directly to task, bypassing overlord itself? That makes sense, gonna try it out, thanks.

[gianm]

Should be irrelevant now since scan-query became merged into the core code in #4751.

[jkukul]

It appears that in Druid 0.10.1 (vs 0.10.0) non-query nodes (coordinator, overlord) can not load query extensions.

Could 0.10.1 release notes be updated with a statement like:
Query extensions must be removed from the list of extensions on non-query nodes (coordinator, overlord).

This would save some trouble for users upgrading from 0.10.0 (and earlier).

[lamberken]

hi @gianm, got the same in 0.17.0
https://github.com/apache/druid/issues/9483

[anandgm007]

Same Issue in 0.18.1.
Any Resolution/Work-around for this issue?

[gianm]

@anandgm007 If you are having the issue from the original description, you can fix it by removing scan-query from your extensions list. It is part of core now.

[anandgm007]

@gianm Thanks for the reply. I don;t have scan-query in the extensions List. Found out my issue is exactly mentioned here in 9483. Wanted to add the moving average extension and i changed the load_extension in broker alone.
Here is my load_extention in _common, common.runtime.properties
druid.extensions.loadList=["druid-s3-extensions", "mysql-metadata-storage","druid-hdfs-storage", "druid-kafka-indexing-service", "druid-datasketches","druid-moving-average-query"]

Getting the following error in router.log when i restart the broker service
2020-08-06T13:52:10,538 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.RuntimeException: com.google.inject.CreationException: Unable to create injector, see the following errors:

1) No implementation for org.apache.druid.query.QuerySegmentWalker was bound.
while locating com.google.inject.Provider
for the 1st parameter of org.apache.druid.query.movingaverage.MovingAverageQueryToolChest.(MovingAverageQueryToolChest.java:61)
at org.apache.druid.query.movingaverage.MovingAverageQueryModule.configure(MovingAverageQueryModule.java:48) (via modules: com.google.inject.u
til.Modules$OverrideModule -> org.apache.druid.query.movingaverage.MovingAverageQueryModule)

Should i include "druid-moving-average-query" in the overload also? right now its just the broker.

[gianm]

I think for the moving-average-query in particular, it should only be in the broker/runtime.properties, not the other services.