Strimzi-kafka-operator: Add Kafka, zookeeper logs in JSON format

[val1715]

Is there possible way to configure logs output as true json format for kafka and zookeeper ?
May be i just have not found it but now i don't see that it is possible for strimzi operator.

Now it looks like:

2019-02-26 13:03:52,302 [custom field] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@397fbdb (org.apache.zookeeper.ZooKeeper) [main]

But it is kubernetes, digital world. I will store logs at elasticsearch and want to configure further processing. At least i got it like:

{"level":"INFO","logger":"kafka.coordinator.group.GroupMetadataManager","message":"[GroupMetadataManager brokerId=2] Removed 0 expired offsets in 4 milliseconds.","host":"kafka-cluster-kafka-2.kafka-cluster-kafka-brokers.my_namespace.svc.cluster.local","@timestamp":"2019-02-26T13:42:47.541Z","thread":"group-metadata-manager-0","@version":"1"}

or

{"level":"INFO","logger":"org.apache.zookeeper.server.NIOServerCnxn","message":"Closed socket connection for client \/127.0.0.1:51756 (no session established for client)","mdc":{"myid":"3"},"host":"kafka-cluster-zookeeper-2.kafka-cluster-zookeeper-nodes.my_namespace.svc.cluster.local","@timestamp":"2019-02-26T09:55:10.134Z","thread":"Thread-13260","@version":"1"}

• pay attention to extra backslash for escaping to keep JSON format. It is not possible by PatternLayout.

---

Is it possible to add JSON log output for kafka/zookeeper natively by operator config?

---

Actually,
For my purpose i create custom docker image over existing with Dockerfile:

FROM strimzi/kafka:0.10.0-kafka-2.0.1
COPY files/ /opt/kafka/libs
CMD ["/opt/kafka/kafka_run.sh"]

At files folder i placed builded log4j-json-layout-szhem-1.0.0.jar from this repository > https://github.com/szhem/log4j-json-layout
I met problems and errors using official logstash repo - https://github.com/logstash/log4j-jsonevent-layout

And i added custom config to my pods:
kafka:

spec:
  kafka:
    template:
    image: 
    ...
    logging:
      type: external
      name: kafka-config-properties

configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: mynamespace
  name: kafka-config-properties
data:
  log4j.properties: |
    log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
    log4j.appender.CONSOLE.layout=com.github.szhem.logstash.log4j.LogStashJsonLayout

[scholzj]

I think that normally something like Logstash or Fluentd should convert the logs into JSON for you when collecting it. I do not think we currently support this directly in Strimzi.

[prune998]

Well you should, as :

• reading Java "stack traces" is really messy
• none of the collecting software (logstash, fluentd) really knows how to deal with multi-line java logs that are not always indented

Please, make kafka/zk dump logs as Json... it's a simple lib to add and will solve lots of people headache

[scholzj]

@stanlyDoge Is this just about adding the JAR? Or would some more changes be needed so that it can be used from the configuration file?

[sknot-rh]

Without investigation I do not know :(

[prune998]

As suggested at https://github.com/szhem/log4j-json-layout I think it's adding the JAR and changing the logging configuration

[scholzj]

I do not think we are changing the configuration to JSON by default. It would need to be configurable.

[prune998]

you can add the JAR in the image and keep the original config file... and/or make it come from a configmap so users can change it ?

[scholzj]

That is what will @stanlyDoge look into. Once we include the JAR, you should be able to configure it through config map - providing custom config files should be already supported (of course you cannot configure the JSON plugin without the JAR).

[prune998]

that's good news ! Thanks

[val1715]

Hello, friends! How is it going?
@stanlyDoge, any news about this issue ?
I am absolutely okay for solution with JAR + mounted configmap for its configuration.

[sknot-rh]

Hello. No update so far :(

[GrigorievNick]

I meet same issue, and i extend docker image with my own.
I believe add ability to add jar to classpath, is great idea for such cases.
I 'am not happy rebuild my own images only for logging.

but parsing java stack trace is also non trivial idea.

[GrigorievNick]

@stanlyDoge Is this just about adding the JAR? Or would some more changes be needed so that it can be used from the configuration file?

I'am sure for 100% that just adding json-layout lib is enough for log4 library.
It also great to give ability add log4j.properties/.xml as config map.

We use "net.logstash.log4j" % "jsonevent-layout" % "1.7"
And
log4j.appender.RollingLog.layout=net.logstash.log4j.JSONEventLayoutV1

[tombentley]

@GrigorievNick if you're able to provide a PR that would be great.

[GrigorievNick]

Ok @tombentley it pretty simple script.

mhr@mhr-HP-ZBook-15-G5:~/IdeaProjects/ISS-BIT$ export LOG4_LOGSTASH_JSON_LAYOUT_CHECKSUM="507713504f0ddb75ba512f62763519c43cf46fde jsonevent-layout-${LOG4_LOGSTASH_JSON_LAYOUT_VERSION}.jar"
mhr@mhr-HP-ZBook-15-G5:~/IdeaProjects/ISS-BIT$ export LOG4_LOGSTASH_JSON_LAYOUT_VERSION=1.7
mhr@mhr-HP-ZBook-15-G5:~/IdeaProjects/ISS-BIT$ export LOG4_LOGSTASH_JSON_LAYOUT_CHECKSUM="507713504f0ddb75ba512f62763519c43cf46fde jsonevent-layout-${LOG4_LOGSTASH_JSON_LAYOUT_VERSION}.jar"
mhr@mhr-HP-ZBook-15-G5:~/IdeaProjects/ISS-BIT$ curl -O https://repo1.maven.org/maven2/net/logstash/log4j/jsonevent-layout/${LOG4_LOGSTASH_JSON_LAYOUT_VERSION}/jsonevent-layout-${LOG4_LOGSTASH_JSON_LAYOUT_VERSION}.jar     && echo $LOG4_LOGSTASH_JSON_LAYOUT_CHECKSUM > jsonevent-layout-${LOG4_LOGSTASH_JSON_LAYOUT_VERSION}.jar.sha1     && sha1sum --check jsonevent-layout-${LOG4_LOGSTASH_JSON_LAYOUT_VERSION}.jar.sha1     && mv jsonevent-layout-${LOG4_LOGSTASH_JSON_LAYOUT_VERSION}.jar ${KAFKA_HOME}/libs/     && rm -rf jsonevent-layout--${LOG4_LOGSTASH_JSON_LAYOUT_VERSION}.*
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10292  100 10292    0     0  63530      0 --:--:-- --:--:-- --:--:-- 63530
jsonevent-layout-1.7.jar: OK
mv: cannot move 'jsonevent-layout-1.7.jar' to '/libs/': Not a directory

You can even add it with your own.
Untill i read your contributing guide.

[scholzj]

Thsi should have been addresses with #1731 and should be closed.