Kibana: Missing '.raw' fields for terms filter in Kibana 4

[izzy]

I installed the Kibana 4 Beta and it seems to me there is no way of using the field.raw modifier to get the terms filter to not split everything up?! Is there any way to do this with Kibana 4?

[rashidkpc]

.raw is not provided by kibana, but rather by elasticsearch and/or logstash. Logstash automatically appends .raw to string fields. You may want to check your elasticsearch mapping.

[ravivasamsetty]

Rashid, I ran into same problem today. I confirmed that mappings for my logstash-* index have the multi-field settings in elasticsearch.

Still my terms dropdown is not showing operation.raw field.

Note: My apologies if I shouldnt have commented on a closed issue

[knaveen05]

I was running into same Challenge, Modifying the index worked for me.
Here is what i did....

curl -XGET 'http://elasticsearch:9200/project?pretty'
{
"project" : {
"mappings" : {
"details" : {
"properties" : {
"ASSIGNED_TO" : {
"type" : "string"
},
"BUSINESS_REQUIRED_DATE" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"CREATE_DATE" : {
"type" : "date",
"format" : "dateOptionalTime"
"PROJECT_NUMBER" : {
"type" : "string"
},
"REQUEST" : {
"type" : "string"
},
"REQUEST_DETAILS" : {
"type" : "string"
},
"REQUEST_SUMMARY" : {
"type" : "string"
},
"SITE" : {
"type" : "string"
},
"STATUS" : {
"type" : "string"
}
}
}
},
"settings" : {
"index" : {
"creation_date" : "1422420652794",
"uuid" : "xLydDUDEQky1NpjyHpapVw",
"number_of_replicas" : "1",
"number_of_shards" : "5",
"version" : {
"created" : "1040299"
}
}
}
}
}

Having chalalnges with this in order to get raw data.
What i did:
curl -XPOST 'http://elasticsearch:9200/project' -d '
{
"details" : {
"properties" : {
"ASSIGNED_TO" : {
"type" : "string",
"fields": {
"raw": { "type": "string", "index": "not_analyzed" }
}
}
},
"BUSINESS_REQUIRED_DATE" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"CREATE_DATE" : {
"type" : "date",
"format" : "dateOptionalTime"
"PROJECT_NUMBER" : {
"type" : "string",
"fields": {
"raw": { "type": "string", "index": "not_analyzed" }
}
}
},
"REQUEST" : {
"type" : "string",
"fields": {
"raw": { "type": "string", "index": "not_analyzed" }
}
}
},
"REQUEST_DETAILS" : {
"type" : "string",
"fields": {
"raw": { "type": "string", "index": "not_analyzed" }
}
}
},
"REQUEST_SUMMARY" : {
"type" : "string",
"fields": {
"raw": { "type": "string", "index": "not_analyzed" }
}
}
},
"SITE" : {
"type" : "string",
"fields": {
"raw": { "type": "string", "index": "not_analyzed" }
}
}
},
"STATUS" : {
"type" : "string",
"fields": {
"raw": { "type": "string", "index": "not_analyzed" }
}
}
}
}
}'

This worked.... However You may have to reload all data. (It works only with new data)

Hope it works!!!!

[jeremyreeder]

I had this same problem, even though I knew that my mapping contained the .raw fields. It turns out that, in Kibana 4, if you change the mapping so that new fields exist, you have to go to the Settings tab and refresh the list of fields.

[innovia]

for me it was a matter of uploading the custom index template to elasticsearch cluster

logstash config output section

 elasticsearch { 
      manage_template => false
      template_name => "my-index"
      workers => 4
      cluster => "ES"
      host => "{{elasticsearch_host}}"
      port => {{elasticsearch_port}}
      protocol => "node"
      index => "{{type}}-%{+YYYY.MM.dd}"
    }

upload to ES

curl -XPUT http://es.server.com:9200/_template/my-index-* -d '
{
  "template" : "my-index-*",
  "settings" : {
    "index.refresh_interval" : "5s"
  },
  "mappings" : {
    "_default_" : {
       "_all" : {"enabled" : true, "omit_norms" : true},
       "dynamic_templates" : [ {
         "message_field" : {
           "match" : "message",
           "match_mapping_type" : "string",
           "mapping" : {
             "type" : "string", "index" : "analyzed", "omit_norms" : true,
             "fields" : {
              "raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
             }
           }
         }
       }, {
         "string_fields" : {
           "match" : "*",
           "match_mapping_type" : "string",
           "mapping" : {
             "type" : "string", "index" : "analyzed", "omit_norms" : true,
               "fields" : {
                 "raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
               }
           }
         }
       } ],
       "properties" : {
         "@version": { "type": "string", "index": "not_analyzed" },
         "geoip"  : {
           "type" : "object",
             "dynamic": true,
             "properties" : {
               "location" : { "type" : "geo_point" }
             }
         }
       }
    }
  }
}
'

1) this requires reindexing - or starting from scratch

2) you must reload the fields as explained before me

[lucaspottersky]

@rashidkpc but why doesn't the "raw" version of the field show up on Kibana for selection? It does show on "Visualize", but it doesn't on "Discover".

As a begginer user, this looks inconsistent to me.

[Bargs]

@lucaspottersky this is sort of an idiosyncrasy of Elasticsearch. The "raw" version is an indexed field in Elasticsearch, but it's not actually a property in your original source document. Discover revolves around showing you the data in your source documents. That said, you can still use the raw field in searches in the filter bar at the top of the page. If you open the "Available Fields" settings on the left hand side and uncheck "Hide missing fields", you'll see the raw fields listed.

[djschny]

@Bargs this comes up in confusion with Kibana users a lot; There is also the following from Tanya - https://discuss.elastic.co/t/confused-about-how-to-use-raw-fields-and-not-analyze-string-fields/28106/2

From what I can tell there is inconsistency in how Kibana treats this. In the Discover view if it is to not care about fields that are not in _source then why do they show up.

Not sure why this is closed.

[Bargs]

@djschny looking back up the thread it seems that @rashidkpc may have closed this issue due to misinterpreting the question, but I'm not 100% sure I fully understand the original question either, it's pretty light on details.

We should probably open a new ticket for discussion with more concrete details about the use case you'd like to see us support. I can see how hiding the .raw fields might be confusing if the user is looking for them but I'm also not sure how to best expose them. Are users just looking to be able to click the magnifying glass icon to create filters on raw fields in addition to analyzed fields? (I don't know the implementation details of the discover sidebar so I can't say if that's feasible or not off the top of my head).

[djschny]

We should probably open a new ticket for discussion with more concrete details about the use case you'd like to see us support.

Sounds good. I will do that