Beats: Filebeat setup is not creating index pattern in kibana

[liza-mae]

• Version: 7.9.0 BC1
• Operating System: Ubuntu 16
• Steps to Reproduce:
  Install and run ES/Kibana.
  Install Filebeat.

Due to #19964 - I tried running ./filebeat setup --dashboards instead to get around it and it seemed to pass,
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards

However, it is not creating the index and even using filebeat setup does not create index pattern .

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[LeeDr]

I hit the same problem with 7.9.0 BC1 on centOS 7. Kibana debug logs didn't show anything related to it.

[liza-mae]

@LeeDr after looking at this closer, I would consider it a blocker since it is causing wider problems in Kibana when creating this index manually and more so when it is the default index pattern.

[kaiyan-sheng]

I'm seeing the same issue when trying to create a kibana index pattern for Filebeat:

Metricbeat seems to work just fine. Maybe it's related to the field count in Filebeat? I don't think user should have to manually change server.maxPayloadBytes in order to get Filebeat working.

[andrewkroh]

To elaborate on @kaiyan-sheng's last comment, while creating the index pattern in Kibana, it requests all of the fields for filebeat-* and gets back about 6700 fields (almost a 1MB response) then tries to PUT the index pattern it created and fails with HTTP 413. This is same error code we see when Filebeat tries to insert the index pattern that it generates (although the error was silently ignored due to https://github.com/elastic/beats/pull/20121).

[urso]

This is quite a many fields...

Is the error because of too many fields in general, or does it indicate some other error?
I wonder if the API would allow us to add fields in batches to the index pattern. Then we could try to split the index pattern into batches of 1000 fields and call it until all fields are installed.

[andrewkroh]

I tried a small hack and I was able to slim down the size of the index pattern request body from Filebeat by omitting some of the parameters with default values (e.g. count=0, analyzed=false). This allows Filebeat setup to work, but even with that working there are still problems in Kibana.

It seems like Kibana still wants to update the index pattern when visiting Discover or opening the Index Pattern UI and this triggers the HTTP 413 despite have already pushed an index pattern.

[andrewkroh]

I have identified about 2000 fields that should not be present. That's about 1/3 of the fields in the mapping so when we fix that it should help this issue.

$ ./filebeat export index-pattern | jq '.objects[0].attributes.fields | fromjson | .[].name' | sort | uniq | grep -E "(microsoft|cisco|fortinet)\.rsa"

The fields returned in that list were meant to be named ^rsa.* and not be prefixed with their module names (e.g. microsoft). Of the 21 new modules added in #19713, this problem only affected modules that had existing filesets.

[andrewkroh]

Summary of Changes

• https://github.com/elastic/beats/pull/20170 Fix field mappings to reduce overall number of fields
• https://github.com/elastic/beats/pull/20156 Assert size of index patterns is less than 1 MiB
• https://github.com/elastic/beats/pull/20121 Fail in 'setup --dashboards' if index pattern cannot be installed