Envoy: zipkin tracing reports to Kafka ?

[jwfang]

currently zipkin tracing only support emit to HTTP collector AFAIK.

[adriancole]

Some mechanism to report to kafka would be a good idea. Kafka is the most robust span transport in zipkin sites.

[mattklein123]

@rshriram @fabolive ^

[adriancole]

ps thinking aloud, if image size/deps is of issue, an envoy proxy (somewhere) that forwards http messages as kafka messages would work.

[mattklein123]

We have already had multiple requests to support L7 kafka in Envoy directly. That's not necessarily required here, but if we did that, it would be very easy to send directly to kafka.

[rshriram]

Wow. This is going to pull in Kafka client complexities into Envoy. Cant we front a stateless zipkin proxy that takes the trace and dumps it into Kafka ?

From integration perspective:
Here is the most popular c lib for Kafka: https://github.com/edenhill/librdkafka

This has to be bazelfied and then integrated.

[adriancole]

most important thing here is.. it is bazelfied or bebazzled?

[mattklein123]

It's a side point, but it doesn't need to be bazelified. We have good/easy support for non-bazel thirdparty deps. The bigger question is should we do this at all, and then we can figure out how, etc. So let's sort that out first.

[rshriram]

I do like the idea, But how would you do the L7 filter for Kafka ?
serve as a proxy that talks Kafka protocol to clients while doing the actual stuff in the backend ?
Or offload specific parts of Kafka lib into Envoy ?

Essentially, are we adding async messaging support in Envoy (need to think of semantics) or are we providing something else ?

[mattklein123]

I know very little about kafka and I think we shouldn't sidetrack this issue with L7 talk of kafka. I was just pointing out that people have asked for L7 support for the same reasons they want L7 support for every other protocol.

Sending zipkin spans direct to kafka is a much simpler problem.

[rshriram]

The complexities of Kafka are many. I have been through pains of working with language libraries with varying amounts of Kafka support (in go and Python) - personally if the app would just dump the message in a generic format to Envoy and Envoy takes care of the actual Kafka dance, it would make life simple. This is where semantics come in. do we expose req response semantics or pub sub semantics? And it pubsub, it doesn't fully alleviate the application burden.

If it's a matter of just zipkin or access log transport, then we can do whatever we want.

[rshriram]

FWIW I raised this topic at Gluecon. And the discussion halted. It might be worth restarting it and seeing how a pub sub system (independent of implementation) would integrate in Envoy mesh. Sorry @adriancole for hijacking the discussion.

But I am for exploring this option and seeing where it takes us. A lot of traditional systems rely on a message bus for the application.

[adriancole]

@rshriram yeah I really kindof hoped we could setting on spelling of bazeling things, but where this is going is OK :D

[fabolive]

In general, the idea of sending tracing reports to Kafka (setting aside the discussion on how, what the implementation implications are, etc.) is really good. Kafka is widely used as a scalable entry point for all sorts of operational data in sophisticated data pipelines. Adding tracing reports to that mix greatly broadens the range of analytics one can write within the context of already-existing data pipelines. Not bad for Envoy to transparently ingest tracing data into such data pipelines without much effort from the users.

[jammerful]

+1 Lack of support for Kafka transport to Zipkin is one of the main reasons I didn't use envoy. Most of my thoughts on why it is a good idea are succinctly summarized by fabolive above.

[yancl]

Hi, any plans for support sending spans to kafka? should I implement an adaptor service receive http requests and then send to kafka? thanks.

[mattklein123]

No one is working on this. Marking help wanted.

[vijayendrabvs]

This looks like a good feature to have, and I can work on this in my spare time. @mattklein123 @rshriram I'm not able to assign this to myself - can you please assign it to me?

[sgandon]

hi Guys any status on this, cause indeed that would be nice to have this feature.
We are using kafka as a core async message broker between our apps and also using it to transport zipkin messages. This would really easy our move to Istio/Envoy.

[afalko]

FWIW we've solved this problem for our envoys by executing on the following architecture: https://docs.google.com/document/d/11_MB69A6d_9_N5ClxVcUhKJRMvVI18voV5cGv1Ji5oc/edit?usp=sharing

In summary, you can run Zipkin HTTP ingestor either as a side-loaded container or something in your isolated environment and it will push spans to Kafka that you can consume with the regular Zipkin collector.

You can try it out and give feedback on what we have thus far here: https://github.com/openzipkin/zipkin/pull/2060#issuecomment-397432922

[OneCricketeer]

Just going to put this here for anyone interested - https://expediadotcom.github.io/haystack/

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.