Charts: [stable/filebeat] Can't overwrite output.file key in values.yaml

[sdanbury]

Is this a request for help?:

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Version of Helm and Kubernetes:
helm: 2.8.1
kubernetes: 1.9.3

Which chart:
stable/filebeat

What happened:
Can't override the output.file part of the config in the values.yaml file. Instead, it combines my output.logstash config with the default output.file section, which causes the container to crash because filebeat can't have two outputs.

What you expected to happen:
I expected my provided config to overwrite the default config in the chart values.yaml.

How to reproduce it (as minimally and precisely as possible):

The values file I am using to override the default chart values. I have copied the default config, removing the output.file section and adding an output.logstash section.

rbac:
  create: false

config:
  filebeat.config:
    prospectors:
      # Mounted `filebeat-prospectors` configmap:
      path: ${path.config}/prospectors.d/*.yml
      # Reload prospectors configs as they change:
      reload.enabled: false
    modules:
      path: ${path.config}/modules.d/*.yml
      # Reload module configs as they change:
      reload.enabled: false

  processors:
  - add_cloud_metadata:

  filebeat.prospectors:
  - type: log
    enabled: true
    paths:
    - /var/log/*.log
  - type: docker
    containers.ids:
    - "*"
    processors:
    - add_kubernetes_metadata:
        in_cluster: true
    - drop_event:
        when:
          equals:
            kubernetes.container.name: "filebeat"

  output:
    logstash:
      hosts:
        - <logstash-url>

Running helm install --debug --values my-values.yaml stable/filebeat produces the following debug output:

COMPUTED VALUES:
config:
  filebeat.config:
    modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false
    prospectors:
      path: ${path.config}/prospectors.d/*.yml
      reload.enabled: false
  filebeat.prospectors:
  - enabled: true
    paths:
    - /var/log/*.log
    type: log
  - containers.ids:
    - '*'
    processors:
    - add_kubernetes_metadata:
        in_cluster: true
    - drop_event:
        when:
          equals:
            kubernetes.container.name: filebeat
    type: docker
  http.enabled: false
  http.port: 5066
  output.logstash:
    hosts:
    - <logstash-url>
  output.file:
    filename: filebeat
    number_of_files: 5
    path: /usr/share/filebeat/data
    rotate_every_kb: 10000
  processors:
  - add_cloud_metadata: null

As you can see, there are now two output sections, output.file comes from the default values.yaml and the output.logstash comes from my values file I am passing in on the command line. I expected there just to be output.logstash.

Logs show:

$ kubectl logs <pod-name>
Exiting: error unpacking config data: more than one namespace configured accessing 'output' (source:'filebeat.yml')

Anything else we need to know:

[sdanbury]

Oh, apologies. Turns out you have to set enabled: false in your values.yaml like so:

config:
  output.file:
    enabled: false

  output.logstash:
    ...

Would be good to have some docs for this chart as this wasn't clear. Worked it out from the elastic docs here: https://www.elastic.co/guide/en/beats/filebeat/current/file-output.html#_literal_enabled_literal_6

[sdanbury]

Oh, it does. That's awkward :grin:. Closing.