Beats: Filebeat 6.6.0 breaks docker container prospecting log population, sends json name of log, not content

[jhowe-uw]

Please include configurations and logs if available.

/etc/filebeat/filebeat.yml

...
filebeat.prospectors:

• type: log
  paths:
  
  
  
  • '/var/lib/docker/containers//.log'
    
    json.message_key: log
    
    json.keys_under_root: true
    
    processors:
  
  
  • add_docker_metadata: ~
    
    ...
  
  

For confirmed bugs, please report:

• Version: filebeat 6.6.0 amd64
• Operating System: Debian 9.6
• Discuss Forum URL: -na-
• Steps to Reproduce:

Upgrade from Filebeat 6.5.4 to 6.6.0
Log entries in elastic change from docker json log content to the json name of the log

Expected:

log: "[2019-02-04 23:55:00,142: DEBUG/MainProcess] portal.tasks.celery_beat_health_check sent. id->561054da-fb21-4ba8-8071-3301e9e14ed3"

Actual:

log: "{
"file": {
"path": "/var/lib/docker/containers/f18ccb8c591ff2704a0974cb8de28083f4c590f19507281edada9ebf5ecdb498/f18ccb8c591ff2704a0974cb8de28083f4c590f19507281edada9ebf5ecdb498-json.log"
}
}"

[ruflin]

The problem here is that in 6.6 we started to populate log.file.path field which now conflicts with your config above. You use keys_under_root in combination with the target logs. You must use a different target name or not use keys_under_root.

In general I would recommend you to move to the docker input if possible: https://www.elastic.co/guide/en/beats/filebeat/6.6/filebeat-input-docker.html

I'm closing this as it's not a bug.

[curantes]

@ruflin We see the same issue but we use the docker input type. Our config loks like below:

    - type: docker
      fields_under_root: true
      containers.ids:
      - "*"
      processors:
        - add_docker_metadata:
        - add_kubernetes_metadata:
            in_cluster: true

raw docker logs looks like this:

{"log":"2019-02-26T13:38:21.616Z\u0009WARN\u0009beater/filebeat.go:374\u0009Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.\n","stream":"stderr","time":"2019-02-26T13:38:21.616905952Z"}

As you can see log will always collide with your new usage of log.file.path

Please advise if there is any configuration issues or this is a bug?

[jhowe-uw]

@curantes I changed my filebeat config in our puppet config to this below.

The stanza 'fields_under_root' is where you get the namespace collisions with the new log.file.path tracking behaviour.

With the config below, your docker logs are under the root 'json' hash tree ( json.log ) , and docker metadata is under the root 'docker' hash tree ( docker.container.labels.com.docker... ) .

HTH

filebeat.prospectors:
- type: docker
  containers.ids: '*'
  json.add_error_key: true
  json.message_key: log
  json.ignore_decoding_error: true
  tags:
    - "docker"
  processors:
    - add_docker_metadata: ~

[joshuabaird]

I would just like to state that this was indeed a breaking change for us (since we use the log field to store our fields and also use fields_under_root: true). It would have been helpful if this had actually been listed as a "breaking change" in the Filebeat 6.6.0 CHANGELOG/release notes.

[ruflin]

@joshuabaird I see your point here, as we also had in our docs some examples about message_key: log as far as I remember. The tricky part here about documenting this as a breaking change is that it would mean every time we add a field, it could be breaking for some users.

@dedemorton Some ideas on where we could document something like this that could hit quite a few users?

[dedemorton]

@ruflin We could do a "Release highlights" section in the release notes. I'll check with other folks working on docs and get back to you. I remember that some folks were not keen on this approach.