Fluent-bit: Fluent-bit daemonset error [out_es] could not pack/validate JSON response

[Bulat-Gumerov]

I’m using fluent-bit as k8s daemonset from fluent/fluent-bit:latest docker image with elasticsearch 7.2.0 3-node cluster deployed in kubernetes cluster. My ElasticSearch cluster deployed on 3 servers with 8cpu/32Gb ram and I setup 28Gb Heap size for ES.
Here is my fluent-bit kubernetes configmaps:

apiVersion: v1
data:
  fluent-bit-filter.conf: "[FILTER]\n    Name                kubernetes\n    Match
    \              kube.*\n    Kube_Tag_Prefix     kube.var.log.containers.\n    Kube_URL
    \           https://kubernetes.default.svc:443\n    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\n
    \   Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token\n
    \   Merge_Log           On\n    K8S-Logging.Parser  On\n    K8S-Logging.Exclude
    On\n    "
  fluent-bit-input.conf: "[INPUT]\n    Name             tail\n    Path             /var/log/containers/*.log\n
    \   Parser           docker\n    Tag              kube.*\n    Refresh_Interval
    5\n    Mem_Buf_Limit    5MB\n    Skip_Long_Lines  On\n    "
  fluent-bit-output.conf: "\n[OUTPUT]\n    Name  es\n    Match *\n    Host  efk-cluster-master\n
    \   Port  9200\n    Logstash_Format On\n    Retry_Limit False\n    Type  flb_type\n
    \   Time_Key @timestamp\n    Replace_Dots On\n    Logstash_Prefix kubernetes_cluster\n\n\n\n
    \   "
  fluent-bit-service.conf: |-
    [SERVICE]
        Flush        1
        Daemon       Off
        Log_Level    info
        Parsers_File parsers.conf
  fluent-bit.conf: |-
    @INCLUDE fluent-bit-service.conf
    @INCLUDE fluent-bit-input.conf
    @INCLUDE fluent-bit-filter.conf
    @INCLUDE fluent-bit-output.conf
  parsers.conf: ""
kind: ConfigMap
metadata:
  creationTimestamp: "2019-07-05T14:08:37Z"
  labels:
    app: fluent-bit
    chart: fluent-bit-2.4.0
    heritage: Tiller
    release: efk
  name: efk-fluent-bit-config
  namespace: logging
  resourceVersion: "20024923"
  selfLink: /api/v1/namespaces/logging/configmaps/efk-fluent-bit-config
  uid: 61e730e5-9f2e-11e9-bf1d-0214258ba47e

Why I’m getting this error?

efk-fluent-bit-5skwk fluent-bit [2019/07/16 13:00:47] [error] [out_es] could not pack/validate JSON response
efk-fluent-bit-5skwk fluent-bit {"took":23,"errors":true,"items":[{"index":{"_index":"kubernetes_cluster-2019.07.16","_type":"flb_type","_id":"z6jf-msBHdfTnPKZAEQb","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":2649383,"_primary_term":3,"status":201}},{"index":{"_index":"kubernetes_cluster-2019.07.16","_type":"flb_type","_id":"0Kjf-msBHdfTnPKZAEQb","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":2649384,"_primary_term":3,"status":201}},{"index":{"_index":"kubernetes_cluster-2019.07.16","_type":"flb_type","_id":"0ajf-msBHdfTnPKZAEQb","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":2649385,"_primary_term":3,"status":201}},{"index":{"_index":"kubernetes_cluster-2019.07.16","_type":"flb_type","_id":"0qjf-msBHdfTnPKZAEQb","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":2649386,"_primary_term"

In ES logs I can see parsing errors:

efk-cluster-master-2 elasticsearch {"type": "server", "timestamp": "2019-07-16T13:29:12,013+0000", "level": "DEBUG", "component": "o.e.a.b.TransportShardBulkAction", "cluster.name": "efk-cluster", "node.name": "efk-cluster-master-2", "cluster.uuid": "h-fyHfLsSLGFR7rp0riMtw", "node.id": "fDdEJ6T8Rmi0CJiZh1F7Ig",  "message": "[kubernetes_cluster-2019.07.07][0] failed to execute bulk item (index) index {[kubernetes_cluster-2019.07.07][flb_type][6qT5-msBc6zN5a3zBJ5G], source[{\"@timestamp\":\"2019-07-07T22:54:09.359Z\", \"log\":\"{\\\"type\\\":\\\"response\\\",\\\"@timestamp\\\":\\\"2019-07-07T22:54:09Z\\\",\\\"tags\\\":[],\\\"pid\\\":1,\\\"method\\\":\\\"get\\\",\\\"statusCode\\\":200,\\\"req\\\":{\\\"url\\\":\\\"/app/kibana\\\",\\\"method\\\":\\\"get\\\",\\\"headers\\\":{\\\"user-agent\\\":\\\"curl/7.29.0\\\",\\\"host\\\":\\\"localhost:5601\\\",\\\"accept\\\":\\\"*/*\\\"},\\\"remoteAddress\\\":\\\"127.0.0.1\\\",\\\"userAgent\\\":\\\"127.0.0.1\\\"},\\\"res\\\":{\\\"statusCode\\\":200,\\\"responseTime\\\":23,\\\"contentLength\\\":9},\\\"message\\\":\\\"GET /app/kibana 200 23ms - 9.0B\\\"}\\n\", \"stream\":\"stdout\", \"time\":\"2019-07-07T22:54:09.359201784Z\", \"type\":\"response\", \"@timestamp\":\"2019-07-07T22:54:09Z\", \"tags\":[], \"pid\":1, \"method\":\"get\", \"statusCode\":200, \"req\":{\"url\":\"/app/kibana\", \"method\":\"get\", \"headers\":{\"user-agent\":\"curl/7.29.0\", \"host\":\"localhost:5601\", \"accept\":\"*/*\"}, \"remoteAddress\":\"127.0.0.1\", \"userAgent\":\"127.0.0.1\"}, \"res\":{\"statusCode\":200, \"responseTime\":23, \"contentLength\":9}, \"message\":\"GET /app/kibana 200 23ms - 9.0B\", \"kubernetes\":{\"pod_name\":\"efk-kibana-bb87549bc-mhkvv\", \"namespace_name\":\"logging\", \"pod_id\":\"64f4096a-9f30-11e9-bf1d-0214258ba47e\", \"labels\":{\"app\":\"kibana\", \"pod-template-hash\":\"bb87549bc\", \"release\":\"efk\"}, \"annotations\":{\"kubernetes_io/psp\":\"eks.privileged\"}, \"host\":\"ip-10-251-127-66.ap-southeast-1.compute.internal\", \"container_name\":\"kibana\", \"docker_id\":\"3038564a5c1c3eb12fd437e905cb08e46247adc950eb5f673024bc4ce670af83\"}}]}" , 
efk-cluster-master-2 elasticsearch "stacktrace": ["org.elasticsearch.index.mapper.MapperParsingException: failed to parse",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.index.mapper.DocumentParser.wrapInMapperParsingException(DocumentParser.java:191) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:74) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:764) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:741) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:713) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:256) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:159) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:191) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:116) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:77) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:927) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:108) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:398) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:316) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$14(IndexShard.java:2525) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:112) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:269) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:236) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2499) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:864) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:312) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:275) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) [x-pack-security-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) [x-pack-security-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:703) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
efk-cluster-master-2 elasticsearch "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
efk-cluster-master-2 elasticsearch "at java.lang.Thread.run(Thread.java:835) [?:?]",
efk-cluster-master-2 elasticsearch "Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field '@timestamp'",
efk-cluster-master-2 elasticsearch " at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@152f7057; line: 1, column: 591]",
efk-cluster-master-2 elasticsearch "at com.fasterxml.jackson.core.json.JsonReadContext._checkDup(JsonReadContext.java:204) ~[jackson-core-2.8.11.jar:2.8.11]",
efk-cluster-master-2 elasticsearch "at com.fasterxml.jackson.core.json.JsonReadContext.setCurrentName(JsonReadContext.java:198) ~[jackson-core-2.8.11.jar:2.8.11]",
efk-cluster-master-2 elasticsearch "at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:777) ~[jackson-core-2.8.11.jar:2.8.11]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:52) ~[elasticsearch-x-content-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:429) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:395) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:112) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:71) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-cluster-master-2 elasticsearch "... 34 more"] }
efk-cluster-master-2 elasticsearch {"type": "server", "timestamp": "2019-07-16T13:29:12,015+0000", "level": "DEBUG", "component": "o.e.a.b.TransportShardBulkAction", "cluster.name": "efk-cluster", "node.name": "efk-cluster-master-2", "cluster.uuid": "h-fyHfLsSLGFR7rp0riMtw", "node.id": "fDdEJ6T8Rmi0CJiZh1F7Ig",  "message": "[kubernetes_cluster-2019.07.07][0] failed to execute bulk item (index) index {[kubernetes_cluster-2019.07.07][flb_type][66T5-msBc6zN5a3zBJ5G], source[{\"@timestamp\":\"2019-07-07T22:54:19.376Z\", \"log\":\"{\\\"type\\\":\\\"response\\\",\\\"@timestamp\\\":\\\"2019-07-07T22:54:19Z\\\",\\\"tags\\\":[],\\\"pid\\\":1,\\\"method\\\":\\\"get\\\",\\...```
Why I’m getting this error?
```efk-fluent-bit-5skwk fluent-bit [2019/07/16 13:00:47] [error] [out_es] could not pack/validate JSON response
efk-fluent-bit-5skwk fluent-bit {"took":23,"errors":true,"items":[{"index":{"_index":"kubernetes_cluster-2019.07.16","_type":"flb_type","_id":"z6jf-msBHdfTnPKZAEQb","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":2649383,"_primary_term":3,"status":201}},{"index":{"_index":"kubernetes_cluster-2019.07.16","_type":"flb_type","_id":"0Kjf-msBHdfTnPKZAEQb","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":2649384,"_primary_term":3,"status":201}},{"index":{"_index":"kubernetes_cluster-2019.07.16","_type":"flb_type","_id":"0ajf-msBHdfTnPKZAEQb","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":2649385,"_primary_term":3,"status":201}},{"index":{"_index":"kubernetes_cluster-2019.07.16","_type":"flb_type","_id":"0qjf-msBHdfTnPKZAEQb","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":2649386,"_primary_term"

[fragtom]

I see, that log property is not parsable, seems that \"type\":\"response\" is double quoted, got this in past and with decode_field_as it did work.
Btw, you cannot use "_type":"flb_type", should be _doc .. See elasticsearch config

    [PARSER]
      Name         docker
      Format       json
      Time_Key     time
      Time_Format  %Y-%m-%dT%H:%M:%S.%L
      Time_Keep    Off
      # Command       |  Decoder  | Field | Optional Action   |
      # ==============|===========|=======|===================|
      Decode_Field_As    escaped     log try_next

[Bulat-Gumerov]

Thanks for reply, I have switched from fluent-bit into fluentd daemonset from kiwigrid instead:
https://github.com/kiwigrid/helm-charts/tree/master/charts/fluentd-elasticsearch

[edsiper]

for the record: proper docker parser is:

    [PARSER]
      Name         docker
      Format       json
      Time_Key     time
      Time_Format  %Y-%m-%dT%H:%M:%S.%L
      Time_Keep    Off

ref: https://docs.fluentbit.io/manual/installation/upgrade_notes

[wangjinh]

i also the same problem. have resloved ?