Graylog2-server: SSL/TLS for Kafka inputs

[ghost]

Apparently it is not possible to use SSL/TLS for Kafka inputs in latest version or there is not documentation for it.

Expected Behavior

Kafka inputs should allow SSL for accessing to server.

Current Behavior

There is no option related to SSL/TLS in Kafka inputs

Possible Solution

No Idea :)

Context

This feature is important when you want to transfer log data across the globe and don't want unwanted access from wiretappers.

Your Environment

We have multiple sites in different locations, each site stores all data in a Kafka server and one graylog server connects to each site and reads log data.

• Graylog Version: Graylog 2.2.3

[tokred]

Would be highly appreciated to fulfill regulatory and compliance requirements!

Best regards

[ghost]

@tokred sorry I didn't understand what you mean

[joschi]

@k1-hedayati I think @tokred simply wants to express some approval of this feature request.

[ghost]

So I guess this doesn't evolve me, ok :)

[madiTG]

+1

[pbr0ck3r]

@joschi I have worked with a few other devs on upgrading Kafka to 1.0.1 and have it successfully working. This allows users to enable/disable SSL/TLS. As I am looking to make a PR to master branch for this so you all can enjoy the benefits as well. Is backwards capability a requirement? Or is it more of a it would be nice to have. From version 9 to 1.01 there was a lot of deprecated API calls, so adding backwards capability may take a lot more work.

[joschi]

@pbr0ck3r Unfortunately it's not that easy since Graylog is using Kafka's journal implementation for its own message journal (see KafkaJournal) and Kafka 0.10.x and later broke backwards-compatibility with it.

This being said, keeping Kafka 0.9.x in Graylog and adding a new Kafka transport using the newer Kafka Java client libraries could work.

[pbr0ck3r]

@joschi I have kafka 1.0.1 working and have modified the KafkaJournal. If I submitted this code in a PR would it be most likely accepted for 3.0?

[joschi]

I have kafka 1.0.1 working and have modified the KafkaJournal.

@pbr0ck3r Does it still support reading journal segments written with the old version?

[pbr0ck3r]

I will make sure that older Journal versions can still be read. Assuming that I can include that functionality would that be sufficient for core graylog?

[joschi]

Assuming that I can include that functionality would that be sufficient for core graylog?

Yes, definitely.

[pbr0ck3r]

Awesome! Thanks for the feedback!

[leoluk]

You can also use stunnel to terminate the TLS:

```cert = your_cert.pem
key = your_cert.key
client = yes

foreground = yes

sslVersion = TLSv1.2
ciphers = HIGH

[kafka]
accept = 127.0.0.1:9092
connect = kafka01.example.com:9492
```

However, I ended up writing a small proxy which provides backpressure by checking Graylog's embedded Kafka queue size and throttling input from the external Kafka server, so I'm not actually using this in production. It did seem to work, but it required direct access to ZooKeeper for what appears to be an old-style consumer that stores offset in ZK.

[pbr0ck3r]

@joschi is the only backwards compatibility that you all are looking for is that the new kafka code would be able to read older kafka journals? Currently write now I have kafka on my dev environment is upgraded to 1.0.1 and you can create inputs with SSL and non-SSL and messages flow into GL fine. I am running a kafka 1.0.1 server. Would you expect that when this is merged into graylog that users would need to update their kafka instances? Just looking for what your expectations of backwards compatibility are. Thank you.

[pbr0ck3r]

Ability to use SSL is added in this PR https://github.com/Graylog2/graylog2-server/pull/4770