Kibana: Discover doesn't show all fields

[jayswan]

I'm seeing a problem with some data where the Discover tab doesn't show all fields, but I can execute queries that match documents with the fields that don't display. For example, I can search:

trans_id:12201

in the query bar and I get exactly the same three documents that I get if I execute the same JSON query in Marvel. However, Kibana only displays a subset of the fields in the document. In this case, the "trans_id" field is NOT displayed by Kibana, even though the documents returned by the search are correct.

Running Kibana4 Beta2 with Elasticsearch 1.4.0.

[rashidkpc]

By default the list only shows the fields that are present in the documents in the result set. Unhide them:

[jayswan]

I may have something horribly wrong, but I don't think that's the problem in this case. With the "Hide Missing Fields" box unchecked, I still don't see the field on which I searched, either in the field list, the expanded table, or the JSON version of the document. For example, here's a search with cURL, based on document ID:

curl -XGET "http://192.168.1.78:9200/logstash-2014.11.20/_search?pretty" -d'

{
"query": {
"match": {

    "_id" : "AUnPLgCT4KoNqCap4_4g"
}

}
}'
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [ {
"_index" : "logstash-2014.11.20",
"_type" : "bro_dns",
"_id" : "AUnPLgCT4KoNqCap4_4g",
"_score" : 1.0,
"_source":{"EventReceivedTime":"2014-11-20 21:49:04","SourceModuleName":"bro_dns_in","SourceModuleType":"im_file","ts":"1416520135.779387","uid":"CPkb5UqMfhfF9t0q3","id_orig_h":"192.168.1.41","id_orig_p":"50787","id_resp_h":"8.8.8.8","id_resp_p":"53","proto":"udp","trans_id":"58990","query":"b._dns-sd._udp.0.74.168.192.in-addr.arpa","qclass":"1","qclass_name":"C_INTERNET","qtype":"12","qtype_name":"PTR","rcode":"3","rcode_name":"NXDOMAIN","AA":"F","TC":"F","RD":"T","RA":"F","Z":"0","answers":"-","TTLs":"-","rejected":"T","@version":"1","@timestamp":"2014-11-20T21:48:58.898Z","host":"192.168.1.251:58261","type":"bro_dns"}
} ] }}

If I use the same query in Kibana4B2, the resulting document doesn't show most of the fields:
(with _id:"AUnPLgCT4KoNqCap4_4g" in the query bar)
{
"EventReceivedTime": "2014-11-20 21:49:04",
"SourceModuleName": "bro_dns_in",
"SourceModuleType": "im_file",
"ts": "1416520135.779387",
"uid": "CPkb5UqMfhfF9t0q3",
"id_orig_h": "192.168.1.41",
"id_orig_p": "50787",
"id_resp_h": "8.8.8.8",
"id_resp_p": "53",
"proto": "udp",
"@version": "1",
"@timestamp": "2014-11-20T21:48:58.898Z",
"host": "192.168.1.251:58261",
"type": "bro_dns"
}

nor does the field list on the left-hand side.

If I search on the trans_id field, which doesn't show up in the Kibana output, I still get the same document using this search string within a time window in which it's unique:

trans_id:"58990"

[rashidkpc]

Ah, I bet you've added new fields to your documents. You need to goto settings -> indices, select your index pattern and refresh its mapping (yellow button). You'll need todo this any time you introduce documents with fields that haven't been seen before.

[jayswan]

Ah, that was it! That never would have occurred to me. Thanks; this has been driving me crazy!

[webmstr]

So it's parsing _source and throwing away any fields that it hasn't seen before? How about a warning on the screen and a link to the refresh function? This problem will bite anyone who does incremental ELK development!

[i3149]

Just spent the last two hours fighting this. +1 for adding a prominent warning.