Graylog2-server: Graylog-server error: "Couldn't calculate index range for index "logstash-2016.11.25"..."

Created on 26 Dec 2016 · 3Comments · Source: Graylog2/graylog2-server

Hello Guys!
We have an elasticsearch node which shared across logstash, ntopng, graylog, etc. Every time we run a curator which deletes indices which are older than 30 days old we see this error message in graylog file log:

2016-12-26T17:24:35.139+02:00 ERROR [MongoIndexRangeService] Couldn't calculate index range for index "logstash-2016.11.25" org.elasticsearch.index.IndexNotFoundException: no such index at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:585) ~[graylog.jar:?] at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:133) ~[graylog.jar:?] at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:113) ~[graylog.jar:?] at org.elasticsearch.action.search.AbstractSearchAsyncAction.<init>(AbstractSearchAsyncAction.java:99) ~[graylog.jar:?] at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.<init>(SearchQueryThenFetchAsyncAction.java:53) ~[graylog.jar:?] at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:93) ~[graylog.jar:?] at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:47) ~[graylog.jar:?] at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:149) ~[graylog.jar:?] at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:137) ~[graylog.jar:?] at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85) ~[graylog.jar:?] at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58) ~[graylog.jar:?] at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359) ~[graylog.jar:?] at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:52) ~[graylog.jar:?] at org.graylog2.indexer.elasticsearch.GlobalTimeoutClient.doExecute(GlobalTimeoutClient.java:58) ~[graylog.jar:?] at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359) ~[graylog.jar:?] at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:348) ~[graylog.jar:?] at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:577) ~[graylog.jar:?] at org.graylog2.indexer.elasticsearch.GlobalTimeoutClient.search(GlobalTimeoutClient.java:64) ~[graylog.jar:?] at org.graylog2.indexer.indices.Indices.timestampStatsOfIndex(Indices.java:593) ~[graylog.jar:?] at org.graylog2.indexer.ranges.MongoIndexRangeService.calculateRange(MongoIndexRangeService.java:142) ~[graylog.jar:?] at org.graylog2.indexer.ranges.MongoIndexRangeService.lambda$handleIndexReopening$1(MongoIndexRangeService.java:203) ~[graylog.jar:?] at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[graylog.jar:?] at com.github.rholder.retry.Retryer.call(Retryer.java:160) ~[graylog.jar:?] at org.graylog2.indexer.ranges.MongoIndexRangeService.handleIndexReopening(MongoIndexRangeService.java:203) [graylog.jar:?] at sun.reflect.GeneratedMethodAccessor384.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_65] at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_65] at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:95) [graylog.jar:?] at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:80) [graylog.jar:?] at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_65] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]

AND

ERROR [Indices] Error while calculating timestamp stats in index <ntopng-2016.11.25> org.elasticsearch.index.IndexNotFoundException: no such index at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:585) ~[graylog.jar:?] at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:133) ~[graylog.jar:?] at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:113) ~[graylog.jar:?] at org.elasticsearch.action.search.AbstractSearchAsyncAction.<init>(AbstractSearchAsyncAction.java:99) ~[graylog.jar:?] at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.<init>(SearchQueryThenFetchAsyncAction.java:53) ~[graylog.jar:?] at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:93) ~[graylog.jar:?] at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:47) ~[graylog.jar:?] at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:149) ~[graylog.jar:?] at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:137) ~[graylog.jar:?] at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85) ~[graylog.jar:?] at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58) ~[graylog.jar:?] at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359) ~[graylog.jar:?] at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:52) ~[graylog.jar:?] at org.graylog2.indexer.elasticsearch.GlobalTimeoutClient.doExecute(GlobalTimeoutClient.java:58) ~[graylog.jar:?] at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359) ~[graylog.jar:?] at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:348) ~[graylog.jar:?] at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:577) ~[graylog.jar:?] at org.graylog2.indexer.elasticsearch.GlobalTimeoutClient.search(GlobalTimeoutClient.java:64) ~[graylog.jar:?] at org.graylog2.indexer.indices.Indices.timestampStatsOfIndex(Indices.java:593) ~[graylog.jar:?] at org.graylog2.indexer.ranges.MongoIndexRangeService.calculateRange(MongoIndexRangeService.java:142) ~[graylog.jar:?] at org.graylog2.indexer.ranges.MongoIndexRangeService.lambda$handleIndexReopening$1(MongoIndexRangeService.java:203) ~[graylog.jar:?] at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [graylog.jar:?] at com.github.rholder.retry.Retryer.call(Retryer.java:160) [graylog.jar:?] at org.graylog2.indexer.ranges.MongoIndexRangeService.handleIndexReopening(MongoIndexRangeService.java:203) [graylog.jar:?] at sun.reflect.GeneratedMethodAccessor384.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_65] at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_65] at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:95) [graylog.jar:?] at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:80) [graylog.jar:?] at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_65] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]

Please advise what is graylog doing with indices which should not be managed by graylog at all?

Graylog Version: 2.1.2-1
Elasticsearch Version: 2.1.1
MongoDB Version: 2.6.11
Operating System: Ubuntu 14.04
Browser version: Firefox 49.0.2

bug triaged

Source

deanilenko

Most helpful comment

We are installing an index change monitor in the embedded Elasticsearch node to detect any changes to indices in Elasticsearch. We do not check if the index that triggered an event is actually managed by Graylog before running a recalculation of the index ranges.

This will show some ugly warnings in the logs but shouldn't do any harm.

The problem will be fixed in the upcoming 2.2. (PR https://github.com/Graylog2/graylog2-server/pull/3259)

Thank you for the report!

bernd on 27 Dec 2016

👍2

All 3 comments

Hi,

We are using GitHub issues for tracking bugs in Graylog itself, but this doesn't look like one. Please post this issue to our public mailing list or join the #graylog channel on freenode IRC.

Thank you!