Beats: Fail to load index template of type `index`

[hendry-lim]

For confirmed bugs, please report:

• Version: 7.10.0
• Operating System: Linux
• Steps to Reproduce:
  Run Beats setup to set up the new Filebeat index template,
  e.g. filebeat setup -e --index-management -E setup.template.type=index

The following error was printed in console:

Exiting: error loading template: could not load template. Elasticsearch returned: couldn't load template: 400 Bad Request:
{
  "error": {
    "root_cause": [{
        "type": "x_content_parse_exception",
        "reason": "[1:2] [index_template] unknown field [settings]"
      }
    ],
    "type": "x_content_parse_exception",
    "reason": "[1:2] [index_template] unknown field [settings]"
  },
  "status": 400
}

Snippet of the index template printed in console:

{
  "index_patterns": [
    "filebeat-7.10.0-*"
  ],
  "mappings": {
  },
  "priority": 150,
  "settings": {
  }
}

mappings and settings should be nested under template object.

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[lliknart]

Hi there,
We observe the same issue when we want to create a new index template
filebeat setup -e --index-management -E setup.template.type=index

Version: 7.10.0
Operating System: Linux

[kvch]

Unfortunately, I cannot reproduce your issue.
Could you please share more details about your Filebeat instance? What Elasticsearch are you trying to connect to? 7.10? Basic licence?

[hendry-lim]

Elasticsearch 7.10.0
Filebeat 7.10.0
I have a Platinum trial license.
Also encountered the same with Platinum license.
Having said that, from what I know license should not matter.

[kvch]

What is the output of ./filebeat export template -E setup.template.type=component? I am not interested in the specific fields, just want to see the structure.

E.g. I get this:

{
  "template": {
    "mappings": {
      "_meta": {
        "beat": "filebeat",
        "version": "7.10.0"
      },
      "date_detection": false,
      "dynamic_templates": [
      ],
      "properties": {}
     }
}

Also, could you share your setup.template.* settings?

[hendry-lim]

{
  "template": {
    "mappings": {
      "_meta": {
        "beat": "filebeat",
        "version": "7.10.0"
      },
      "date_detection": false,
      "dynamic_templates": [
      ],
      "properties": {}
    },
    "settings": {}
  }
}

I don't have any custom configuration for setup.template.*.

I just tried to load using setup.template.type=component. It worked, but using setup.template.type=index failed with the same error as on my first post.
Ran this command
filebeat setup -e --index-management -E setup.template.type=index -E output.elasticsearch.ssl.certificate_authorities=ca.crt -E output.elasticsearch.username=... -E output.elasticsearch.password=...

[lliknart]

Unfortunately, I cannot reproduce your issue.
Could you please share more details about your Filebeat instance? What Elasticsearch are you trying to connect to? 7.10? Basic licence?

Elasticsearch 7.9.3 / 7.10.0
Filebeat 7.10.0

This command works: Create a legacy tempate

$ docker run -it docker.elastic.co/beats/filebeat:7.10.0 filebeat setup --index-management \
-E output.logstash.enabled=false \
-E 'output.elasticsearch.hosts=['"\"${ES_HOST}\""']' \
-E output.elasticsearch.username="${ES_LOGIN}" -E output.elasticsearch.password="${ES_PASSWORD}" \
-E setup.template.name=test-index -E "setup.template.pattern=test-index-*" \
-E setup.template.overwrite=true -E setup.ilm.rollover_alias=test-index \
-E setup.template.fields=fields.yml -E setup.ilm.enabled=true \
-E "setup.ilm.pattern:{now/d}-000001" -E setup.ilm.policy_name=test-index \
-E setup.ilm.overwrite=true

This one does not work: Try to create an index template

$ docker run -it docker.elastic.co/beats/filebeat:7.10.0 filebeat setup --index-management \
-E output.logstash.enabled=false \
-E 'output.elasticsearch.hosts=['"\"${ES_HOST}\""']' \
-E output.elasticsearch.username="${ES_LOGIN}" -E output.elasticsearch.password="${ES_PASSWORD}" \
-E setup.template.name=test-index -E "setup.template.pattern=test-index-*" \
-E setup.template.overwrite=true -E setup.ilm.rollover_alias=test-index \
-E setup.template.fields=fields.yml -E setup.ilm.enabled=true \
-E "setup.ilm.pattern:{now/d}-000001" -E setup.ilm.policy_name=test-index \
-E setup.ilm.overwrite=true -E setup.template.type=index

The error I get:

Exiting: error loading template: could not load template. Elasticsearch returned: couldn't load template: 400 Bad Request: {"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:42] [index_template] unknown field [mappings]"}],"type":"x_content_parse_exception","reason":"[1:42] [index_template] unknown field [mappings]"},"status":400}. Response body: {"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:42] [index_template] unknown field [mappings]"}],"type":"x_content_parse_exception","reason":"[1:42] [index_template] unknown field [mappings]"},"status":400}. Template is: {
  "index_patterns": [
    "test-index-*"
  ],
  "mappings": {
    "_meta": {
      "beat": "filebeat",
      "version": "7.10.0"
    },
    "date_detection": false,
    "dynamic_templates": [
      {
        "labels": {
          "mapping": {
            "type": "keyword"
          },
          "match_mapping_type": "string",
          "path_match": "labels.*"
        }
      }
[...]

[kvch]

I can reproduce it now, thanks for the help.