Kibana: fields starting with underscore ( _ ) doesn't show.

[jeesim2]

hi.

Fields that starting with underscore ( _ ) doesn't show in kibana 4.
for example _class , _EventID_
I've tried reload fields list but doesn't help.
(neither in visualize panel it doesn't show )

At elasticsearch it really exist at every indices(mapping and data)

Versions.
ES 1.4.0
Kibana 1.4 beta 3

[rashidkpc]

Hi,
We reserve fields beginning with _ for use within elasticsearch, outside of the _source of a document. While elasticsearch does not enforce this rule, we do in Kibana.

[jeesim2]

I've never heard about that rule from any documents..
it is an blocking issue to me for upgrade kibana4.
because very large data in production have fields begginig with _

[shangan]

I do not think this is necessary, underscore field is commonly used in practical process.

[ulir]

We also use underscores, actually as the start for most of our fields, because of naming conventions and dynamic templates that go with it.
A way to configure this away and show those _fields would really help.

[leg100]

The vast majority of the fields I have in elasticsearch have an underscore prefix. We consume systemd journal's JSON output and store it in elasticsearch, and the journal names its fields with an underscore prefix, e.g.:

"_SYSTEMD_CGROUP" : "/", "_MACHINE_ID" : "d85bd5b7c13d4549b3d3d1719d3a344a", "_HOSTNAME" : "ip-172.13.3.3-amazonaws.com", "MESSAGE" : "Starting Generate /run/coreos/motd..."

I'd be interested to hear the justification for excluding them.

[algesten]

This is a terrible decision. We use _-prefixed fields to mean "internally for our own processing, not for public consumption". However those fields are _very_ relevant in making our kibanas.

@rashidkpc
You say

elasticsearch does not enforce this rules, we do in kibana

Could you please elaborate on why kibana should enforce it when elastic doesn't?!

[mjgp2]

Come on guys, this has "it's not a bug, it's a feature" written all over it :)

You can't expect people to re-engineer their _valid_ elasticsearch indexes in order to use kibana. Reading between the lines it looks like you have no way of distinguishing between system fields and object properties, so if there's a collison it's ambiguous (on _id for example).

Surely the aim should be to support any _valid_ elasticsearch index rather than a defined subset of valid indexes according to some criteria you choose? If so this is a bug - it may be low priority, but it's still a bug.

If you want to make "_" prefixes disallowed in ES go ahead - then at least it would be consistent and people wouldn't be scratching their heads wondering why this happens.

[andrewwebber]

i pipe journald JSON logs on every app server directly into elastic search via logstash.

journalctl --boot --no-tail -f -o json | /usr/bin/ncat logstashserver json-codec-port

it was quite nice to view all the systemd properties in kibana3. This is no longer possible to visualize in kibana4. So i should map all journald fields and remove underscores? Some examples:

_MACHINEID
_SYSTEMD_UNIT
_PID
_KERNAL_DEVICE
_KERNEL_SUBSYSTEM
__REALTIME_TIMESTAMP
etc.

Notice some have double underscores :(

[algesten]

Hm. If we could locate where in the code this rule is enforced, it would be worth making a fork.

[algesten]

https://github.com/elastic/kibana/pull/4291 and a pull request. Even if it isn't entirely right, I hope we at least can get @rashidkpc or someone to at least enter into a discussion about it.

[andrewwebber]

this was my poor mans work around for systemd however i would need to dynamically generate this as systemd evolves :(

filter {
  mutate {
    rename => { "_MACHINE_ID" => "MACHINE_ID" }
    rename => { "_SYSTEMD_UNIT" => "SYSTEMD_UNIT" }
    rename => { "_SYSTEMD_SLICE" => "SYSTEMD_SLICE" }
    rename => { "_SYSTEMD_CGROUP" => "SYSTEMD_CGROUP" }
    rename => { "_PID" => "PID" }
    rename => { "_EXE" => "EXE" }
    rename => { "_CMDLINE" => "CMDLINE" }
    rename => { "_BOOT_ID" => "BOOT_ID" }
  }
}

I have had to choose between this 'bug' and exporting dashboards it would be the later.

[algesten]

So there is a way forward. https://github.com/elastic/kibana/pull/4291#issuecomment-114666999

[thesmart]

+1 on this, decisions like this make no sense.

[hasJerry]

Is there an open issue for this somewhere? Doesn't seem rational to carry this restriction forward for all time.

[blendle-hubbit]

Heya!

This issue was just mentioned in the Trello card "(3) Store all enriched items in new ES index [MS - 9-6] [blendle-premium]" on the Blendle Data board.

Might be relevant to you, or not...

_Happy Hackin'!_

[raviv]

+1 for removing this restriction

[andrewthad]

This restriction is a problem for a piece of software I am working on. I'm in favor of removing it.

[ptambe]

+1 for removing this restriction

[tugrulcan]

+1 !

[cabsanchez]

Is this aready fixed in a released version of Kibana?

[lanceblais]

I know this is closed, but I'm also curious in whether or not this has been released?

[alxchk]

This creates an issue with rsyslog elasticsearch plugin

[ychennay]

I can confirm this "feature" is NOT "fixed". Just spent the last couple hours retracing my steps as to why certain fields weren't appearing within Kibana until I found this post.

[ederparaiso]

when this fix will be released?

[hanmanhui]

While many confirms that this issue isn't fixed, shouldn't this be reopened?

[wangxiuwen]

5.5.0 not fixed

[Aaronls79]

"Closed" this is a huge pain why!?!?!?

[cherchezlafemme]

Seeing same issue with data from mongodb.
Any updates on when there will be a fix released?

elasticsearch does not enforce this rules, we do in kibana

@rashidkpc Why?

[ialidzhikov]

I am using 6.5.4 and the issue is still relevant.

[Altiano]

7.2.0 and Kibana shows warning on the fields

[sdether]

What does the warning mean though? I can see fields starting with underscore and i can search for them, so other than that UI artifact is there any adverse behavior with fields starting with underscore?