Kibana: Kibana index pattern page times out on index pattern on indices with very large number of fields

[imotov]

Kibana version: 5.2.1

Elasticsearch version: 5.2

Server OS version: ubuntu 16.04.01 LTS

Browser version: Google Chrome 56.0.2924.87 (64-bit)

Browser OS version: Mac OS

Original install method (e.g. download page, yum, from source, etc.): deb

Description of the problem including expected versus actual behavior:

Originally reported by an user on Russian-language discussion forum: https://discuss.elastic.co/t/logstash-json/75314 The user is trying to use kibana on an index with a very large number of fields (tens of thousands). The user claims that after navigating to "Configure an index pattern" page his kibana instance becomes first unresponsive and displays this screen:

https://cdn-enterprise.discourse.org/elastic/uploads/default/original/2X/a/a1832587a29fb1d507e194a9f0670e76d3d5fa56.png

[rashmivkulkarni]

What is the use case behind this tens of thousands of fields. Would be interested to know...

Tried with a data set of ~2K+ fields and it was successful in 6.0.0Beta1 ( setting the limit to 5K fields) steps attached:

1) created an example index doc:

PUT /testhuge/test/1
{
    "date" : "2017-07-03T00:01:00",
    "action" : "start",
    "myid" : 1,
    "myvar1" : 1,
"myvar2" : 2
}

2) Increased the index mapping total field limit to 5000. ( initial is set to a limit of 1000)

PUT testhuge/_settings
{
  "index.mapping.total_fields.limit": 5000
}

3) Used a script to generate 2K + fields using the script attached and it indexed it within seconds.

TestDataHuge.txt

cc /@tanya @LeeDr @bhavyarm

[tbragin]

Marking "not reproducible" for now, but please respond if you are experiencing this issue. Be sure to Iinclude detailed version and repro steps, as shown above ^^ Thank you!

[trevan]

We have an index pattern with ~6K fields. The index-pattern page has always been fairly responsive.

[LeeDr]

@trevan A lot of us are curious about what kind of data you would use that has ~6k fields. Is it something you can easily describe?

[trevan]

@LeeDr, probably the easiest way to describe it would be a really souped up packetbeat. Our product knows how to analyze 100s of different protocols and extracts many attributes for each protocol as well as enriches each document with more data. Put us inside an network with enough protocols and we would probably go over 10k fields.

[chrisronline]

Removing Management as it doesn't seem there are any action items here yet.

[eric-wei1990]

Hi, I faced same issue.
Currently, we have an index-pattern that there are 43k fields.

@trevan said

Put us inside an network with enough protocols and we would probably go over 10k fields.

So if the number of fields is more than 10k, could it work?
Could you have a test? Thanks~

[LeeDr]

@rasroh did you find a limit? I think you went to at least 8k fields? Was there a setting to tweak to go over 10k fields that you know of?

[jindrichskupa]

I am facing the same issue, please see attached screenshot. The problem occurred after the upgrade and reindex .kibana from 5.6.5 to 6.2.3.

http://prntscr.com/j1kj56

It's the same on Discovery and Index pattern page.

[jindrichskupa]

Kibana process on server consumes more than 2G RAM.

strace output looks:
https://gist.github.com/jindrichskupa/3e812589b814b223d51b3797aa1da4a9

It looks like the memory problem.