Graylog2-server: Search query with sort on custom field returns an error if it uses multiple index sets

[2fa]

Expected Behavior

Search query should return sorted result for specified custom field when there are multiple index sets.

Current Behavior

When you use "/search" query on multiple index sets with sort enabled for specific custom field it fails with 500 error code.

It's working if you execute same search query on "All messages" stream.

Steps to Reproduce (for bugs)

1. Create additional index set with some data in it.
2. Run a search query from the "/search" page with sorting on custom field.

Context

Elasticsearch error log:

[2017-11-21T19:54:16,379][DEBUG][o.e.a.s.TransportSearchAction] [graylog-es1-es3] [st_d1_graylog_0][4], node[37wRzeRfTRy46-ppocAQGQ], [P], s[STARTED], a[id=umlVPfLdQAexlxf15BA78A]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[lt_graylog_4, lt_graylog_1, st_d1_graylog_0, graylog_88], indicesOptions=IndicesOptions[id=38, ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_alisases_to_multiple_indices=true, forbid_closed_indices=true], types=[message], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=15, batchedReduceSize=512, preFilterShardSize=128, source={
  "from" : 0,
  "size" : 150,
  "query" : {
    "bool" : {
      "must" : [
        {
          "query_string" : {
            "query" : "event_id:8",
            "fields" : [ ],
            "use_dis_max" : true,
            "tie_breaker" : 0.0,
            "default_operator" : "or",
            "auto_generate_phrase_queries" : false,
            "max_determinized_states" : 10000,
            "allow_leading_wildcard" : false,
            "enable_position_increments" : true,
            "fuzziness" : "AUTO",
            "fuzzy_prefix_length" : 0,
            "fuzzy_max_expansions" : 50,
            "phrase_slop" : 0,
            "escape" : false,
            "split_on_whitespace" : true,
            "boost" : 1.0
          }
        }
      ],
      "filter" : [
        {
          "bool" : {
            "must" : [
              {
                "range" : {
                  "timestamp" : {
                    "from" : "2017-11-21 16:49:16.365",
                    "to" : "2017-11-21 16:54:16.365",
                    "include_lower" : true,
                    "include_upper" : true,
                    "boost" : 1.0
                  }
                }
              }
            ],
            "disable_coord" : false,
            "adjust_pure_negative" : true,
            "boost" : 1.0
          }
        }
      ],
      "disable_coord" : false,
      "adjust_pure_negative" : true,
      "boost" : 1.0
    }
  },
  "sort" : [
    {
      "event_id" : {
        "order" : "asc"
      }
    }
  ]
}}]
org.elasticsearch.transport.RemoteTransportException: [graylog-es1-es1][10.3.1.12:9302][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.index.query.QueryShardException: No mapping found for [event_id] in order to sort on
        at org.elasticsearch.search.sort.FieldSortBuilder.build(FieldSortBuilder.java:262) ~[elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.search.sort.SortBuilder.buildSort(SortBuilder.java:156) ~[elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:634) ~[elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:485) ~[elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:461) ~[elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257) ~[elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:343) ~[elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340) ~[elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1553) ~[elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.6.4.jar:5.6.4]
                at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.6.4.jar:5.6.4]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_151]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

There's also #4363 which can be related to this one.

Your Environment

• Graylog Version: 2.3.2+3df951e
• Elasticsearch Version: 5.6.4
• MongoDB Version: 3.4.7
• Operating System: Irrelevant
• Browser version: Irrelevant

[onyxmaster]

Any progress on this? Sorting is broken for large queries for us.

[bernd]

This happens because the event_id field doesn't exist in all index sets that are part of the query. (actually: doesn't exist in all index mappings) Elasticsearch supports an unmapped_type option which might help to solve this issue. This would need to be added to the search code. The tricky part is to pick the correct value, which probably requires the inspection of the index mappings to get the correct type.

[mpfz0r]

@bernd FYI, that's the issue I saw an Friday. If I restrict the query to indeces that contain the field to sort on,
it works.

[dcddealmeida]

@mpfz0r Hi man, could you help to solve this issue?

I dont understand what you did to correct!

org.elasticsearch.transport.RemoteTransportException: [AvHYDRV][192.168.1.24:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [message] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.

[bernd]

I am unable to reproduce this anymore in the upcoming 4.0 release. It has been fixed in #7637.