Druid: Emitted metrics constantly growing

[cesure]

Hi!

We started testing Druid 0.10.0 on our dev cluster and found a possible bug.

We are using the Http Emitter Module to log the metrics. Since 0.10.0 the messages sent to our HTTP API are constantly growing the longer the single nodes are running. There seems to be a problem with the "namespace/deltaTasksStarted" metric which is multiple times in the JSON with different timestamps. The longer the node runs (broker in this case but happens on historical nodes too) the more deltaTasksStarted entries there are per emitted message. See attached a sample JSON.

druid_broker_metrics.json.txt

Please let me know if you need further details.

[gianm]

Thanks for the report @cesure. It looks like this is due to NamespaceLookupExtractorFactory accepting CacheScheduler when instantiated, which is not a singleton and so it's created each time a "cachedNamespace" is deserialized.

@leventov this changed in #3697, where NamespaceLookupExtractorFactory used to take a (singleton) NamespaceExtractionCacheManager but now takes a (not singleton) CacheScheduler. Would it make sense to make CacheScheduler a singleton or is some other approach warranted?

[leventov]

https://github.com/metamx/druid/tree/cacheScheduler-singleton might fix this

[cesure]

Thank you @leventov!

I just backported this single commit locally into 0.10.0 will try it on our dev cluster.

[cesure]

Making CacheScheduler a singleton fixed the problem for us!

I can't comment on any side effects this change eventually brings in for now, though.

[leventov]

@cesure thanks for testing. The fix should be available in Druid 0.10.1.