Beats: filebeat json log input changed in master?

[graphaelli]

While upgrading filebeat to master, ran into an issue collecting logs from docker containers with filebeat. All log lines are rejected with:

{"type":"mapper_parsing_exception","reason":"object mapping for [log] tried to parse field [log] as object, but found a concrete value"}

docker-compose.yml contains:

filebeat:
  command: filebeat -e --strict.perms=false
  depends_on:
    elasticsearch: {condition: service_healthy}
    kibana: {condition: service_healthy}
  image: docker.elastic.co/beats/filebeat:7.0.0-alpha1-SNAPSHOT
  logging:
    driver: json-file
    options: {max-file: '5', max-size: 2m}
  user: root
  volumes: ['./filebeat.yml:/usr/share/filebeat/filebeat.yml', '/var/lib/docker/containers:/var/lib/docker/containers']

and filebeat.yml is:

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  index.number_of_replicas: 0

setup.dashboards.enabled: true

setup.kibana:
  host: "kibana:5601"

output.elasticsearch:
  hosts: ["elasticsearch:9200"]

filebeat.prospectors:
  - type: log
    paths:
     - '/var/lib/docker/containers/*/*.log'
    json.message_key: log
    json.overwrite_keys: true
    json.keys_under_root: true
    json.add_error_key: true

I've run filebeat containers running 6.2.5-SNAPSHOT, 6.3.0 release, and 6.4.0-SNAPSHOT (filebeat version 6.4.0 (amd64), libbeat 6.4.0 [575c6758d3ef0b33d4b1af54f35cadbc87fda08e built 2018-06-20 06:00:39 +0000 UTC]) side by side with this 7.0.0-alpha1-SNAPSHOT and they are able to process the same logs.

[graphaelli]

logged with debug on:

2018-06-20T17:37:55.785Z    DEBUG   [publish]   pipeline/processor.go:291   Publish event: {
  "@timestamp": "2018-06-20T17:37:55.785Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "7.0.0-alpha1"
  },
  "beat": {
    "name": "6a8a5ec7a2a8",
    "hostname": "6a8a5ec7a2a8",
    "version": "7.0.0-alpha1"
  },
  "host": {
    "name": "6a8a5ec7a2a8"
  },
  "log": "  \"offset\": 207251",
  "time": "2018-06-20T17:37:51.0295429Z",
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "stream": "stderr",
  "source": "/var/lib/docker/containers/6a8a5ec7a2a872514dcb9e90ac5feb16d5e4e6533e43536e5cc702f4ea6a3d78/6a8a5ec7a2a872514dcb9e90ac5feb16d5e4e6533e43536e5cc702f4ea6a3d78-json.log",
  "offset": 1581986
}

the log entry for other events:

"log": "}"
"log": "  \"@metadata\": {",
"log": "    \"beat\": \"filebeat\","

[ph]

thats really odd, I will take a closer look.

[ph]

The problem is the following docker log format by default users the "log" key to store the log message. The above configuration was using fields_under_root to parse the docker log format and put all the information at the root of the event.

in 6.3 we have introduced a new field in our templated named log.level which is part of ECS, this makes the 'log' key an object which causes this _bulk_ inserts to fails.

The solution is to use the docker input; this specialized input will do the necessary manipulation on the original events to make use of ECS fields.

ECS: http://github.com/elastic/ecs

[graphaelli]

@ph thanks for the explanation. Setting json.keys_under_root=false worked as you expected. I'll give the docker input a go for this case.