Helm-charts: Filebeat not creating specified index

Created on 23 Mar 2020  路  8Comments  路  Source: elastic/helm-charts

Chart version: 7.6.1

Kubernetes version: 1.4

Kubernetes provider: EKS

So I am trying to set up filebeat to use an index different to filebeat, below is my config map:

filebeat.inputs:
- type: docker
  containers.ids:
  - '*'
  processors:
  - add_kubernetes_metadata:
      in_cluster: true
  - add_cloud_metadata:

  fields:
    source: eu-west-1
    log_type: kubernetes
    environment_type: production
    environment: eu-west-1
  tags: ['kubernetes']

output.elasticsearch:
  index: "kubernetes"
  hosts: "https://xeu-west-1.aws.found.io:9243/"
  username: "elastic"
  password: "x"

setup.template.name: "kubernetes"
setup.template.pattern: "kubernetes-*"
setup.template.settings:
  setup.ilm.enabled: true
  setup.ilm.rollover_alias: "kubernetes"
  setup.ilm.pattern: "{now/d}-000001"
  setup.ilm.policy_name: "Standard-Index"

however, when the pod starts it appears to treat ILM as if it's set to auto.

2020-03-23T09:50:59.310Z    INFO    [index-management]  idxmgmt/std.go:258  Auto ILM enable success.
2020-03-23T09:50:59.315Z    INFO    [index-management.ilm]  ilm/std.go:139  do not generate ilm policy: exists=true, overwrite=false
2020-03-23T09:50:59.315Z    INFO    [index-management]  idxmgmt/std.go:271  ILM policy successfully loaded.
2020-03-23T09:50:59.315Z    INFO    [index-management]  idxmgmt/std.go:410  Set setup.template.name to '{filebeat-7.6.1 {now/d}-000001}' as ILM is enabled.
2020-03-23T09:50:59.315Z    INFO    [index-management]  idxmgmt/std.go:415  Set setup.template.pattern to 'filebeat-7.6.1-*' as ILM is enabled.

I am using image and chart version 7.6.1 there are zero errors or warnings on fiilebeat startup.

It's definitely reading the config as the logs are reaching our ES cluster in the filebeat index. How do I change this index?

picture runtman

Most helpful comment

You are 100% correct, my apologies

No worries. Glad I could get you working...

picture fatmcgav on 23 Mar 2020
❤1 👍1

All 8 comments

@runtman based on the filebeat ilm docs, I believe the setup.ilm keys in your config map above are incorrectly nested.

They should be top-level config items...

Can you give that a try?

fatmcgav picture fatmcgav on 23 Mar 2020

Hey, so this my config now:

filebeat.yml:
setup.template.name: "kubernetes"
setup.template.pattern: "kubernetes-*"
setup.template.settings:
  setup.ilm.enabled: true
  setup.ilm.rollover_alias: "kubernetes"
  setup.ilm.pattern: "{now/d}-000001"
  setup.ilm.policy_name: "Standard-Index"

filebeat.inputs:
- type: docker
  containers.ids:
  - '*'
  processors:
  - add_kubernetes_metadata:
      in_cluster: true
  - add_cloud_metadata:

  fields:
    source: eu-west-1
    log_type: kubernetes
    environment_type: production
    environment: eu-west-1
  tags: ['kubernetes']

output.elasticsearch:
  index: "kubernetes"
  hosts: "https://x.eu-west-1.aws.found.io:9243/"
  username: "elastic"
  password: "x"

And the output for ILM is:

2020-03-23T11:20:59.294Z    INFO    [index-management]  idxmgmt/std.go:258  Auto ILM enable success.
2020-03-23T11:20:59.299Z    INFO    [index-management.ilm]  ilm/std.go:139  do not generate ilm policy: exists=true, overwrite=false
2020-03-23T11:20:59.299Z    INFO    [index-management]  idxmgmt/std.go:271  ILM policy successfully loaded.
2020-03-23T11:20:59.299Z    INFO    [index-management]  idxmgmt/std.go:410  Set setup.template.name to '{filebeat-7.6.1 {now/d}-000001}' as ILM is enabled.
2020-03-23T11:20:59.299Z    INFO    [index-management]  idxmgmt/std.go:415  Set setup.template.pattern to 'filebeat-7.6.1-*' as ILM is enabled.
2020-03-23T11:20:59.299Z    INFO    [index-management]  idxmgmt/std.go:449  Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.6.1 {now/d}-000001} as ILM is enabled.
2020-03-23T11:20:59.299Z    INFO    [index-management]  idxmgmt/std.go:453  Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
runtman picture runtman on 23 Mar 2020

What is interesting is, if I leave out setup.template.name or setup.template.pattern the container does tell me it's not there and asks me to add it. So it's being read.

runtman picture runtman on 23 Mar 2020

I jumped into the container too and checked the config to ensure it's all parsed correctly, if this is of any use see below:

sh-4.2# cat filebeat.yml 
setup.template.name: "kubernetes"
setup.template.pattern: "kubernetes-*"
setup.template.settings:
  setup.ilm.enabled: true
  setup.ilm.rollover_alias: "kubernetes"
  setup.ilm.pattern: "{now/d}-000001"
  setup.ilm.policy_name: "Standard-Index"

filebeat.inputs:
- type: docker
  containers.ids:
  - '*'
  processors:
  - add_kubernetes_metadata:
      in_cluster: true
  - add_cloud_metadata:

  fields:
    source: eu-west-1
    log_type: kubernetes
    environment_type: production
    environment: eu-west-1
  tags: ['kubernetes']

output.elasticsearch:
  index: "kubernetes"
  hosts: "https://x.eu-west-1.aws.found.io:9243/"
  username: "elastic"
  password: "x"
runtman picture runtman on 23 Mar 2020

@runtman it looks like the setup.ilm.* keys are still nested under the setup.template.settongs: key.

They shouldn't be...

So the config should look like:
setup.template.name:聽"kubernetes" setup.template.pattern:聽"kubernetes-*" setup.ilm.enabled:聽true setup.ilm.rollover_alias:聽"kubernetes" setup.ilm.pattern:聽"{now/d}-000001" setup.ilm.policy_name:聽"Standard-Index"

fatmcgav picture fatmcgav on 23 Mar 2020
❤1

Oh, I understand now, sorry!

runtman picture runtman on 23 Mar 2020 
2020-03-23T11:47:50.036Z    INFO    [index-management]  idxmgmt/std.go:410  Set setup.template.name to '{kubernetes {now/d}-000001}' as ILM is enabled.
2020-03-23T11:47:50.036Z    INFO    [index-management]  idxmgmt/std.go:415  Set setup.template.pattern to 'kubernetes-*' as ILM is enabled.
2020-03-23T11:47:50.036Z    INFO    [index-management]  idxmgmt/std.go:449  Set settings.index.lifecycle.rollover_alias in template to {kubernetes {now/d}-000001} as ILM is enabled.
2020-03-23T11:47:50.036Z    INFO    [index-management]  idxmgmt/std.go:453  Set settings.index.lifecycle.name in template to {Standard-Index {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.

You are 100% correct, my apologies.

runtman picture runtman on 23 Mar 2020

You are 100% correct, my apologies

No worries. Glad I could get you working...

fatmcgav picture fatmcgav on 23 Mar 2020
❤1 👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

naseemkullah picture naseemkullah  路  6Comments
guding picture guding  路  5Comments
ichasco-heytrade picture ichasco-heytrade  路  7Comments
elijahsgh picture elijahsgh  路  7Comments
jmlrt picture jmlrt  路  4Comments