Skywalking: [Tracer Interop] Google OpenCensus / OpenZipkin

[wu-sheng]

This is a potential tracer interop chance, provided by Google( @bogdandrutu ), Zipkin ( @adriancole ). It provided a new way to be portable across backends, don't like OpenTracing to provide an APIs layer, but have not to tell what are really happening.

The two major things make the portable possible.

1. trace-context format, the new specification, as you known, I already take part in it.
2. An exporter mechanism, export means that the data collected by the library can be useful in zipkin or others. Here is the exporter implementations. And the data format is based on trace protobuf definition

So we always want to do some interop works with other tracers, now, we have two choices:

1. Support Zipkin v2 network format. Just need a service to receive, buffer, assemble.
2. Implement OpenCensus reporter. Do more works at agent(client) side, such as compression, quick group spans in same traces. Those will help the performance and network.

After all, this is only a discussion proposal (TBD), feel free to discuss.

[wu-sheng]

Hi @apache/skywalking-committers , since OpenCensus supports Zipkin data format, I am working on the transformation in this repo: https://github.com/OpenSkywalking/zipkin-skywalking , by taking helps from @adriancole .

Welcome to join us.

[candyleer]

I am interested in this feature am I'm glad to help to complete if I can.Does it mean when a user use zipkin in their library.the can also choose to send their trace data to skywaking by use zipkin-skywalking libraray?
If so ,maybe we can also provide jaeger-skywalking library, haha ,they are so similar and I used to build our tracing system by jaeger.

[wu-sheng]

yes, for zipkin, I and @adriancole finished the document, https://github.com/OpenSkywalking/zipkin-skywalking

[wu-sheng]

I am glad to work with you about both of these two.

[wu-sheng]

We should start our discussions about the process flow and architecture. Could provide a draft? Then I and @peng-yongsheng could join you.

[wu-sheng]

And for zipkin and jaeger, I prefer zipkin first. They have more end users and better community support.

[wu-sheng]

One thing need to correct, library is not enough. We will need a cluster to support that, if ypu compare SkyWalking and zipkin format, the major and most important different is about ref in across thread scenario. You need buffer and match spans in zipkin, that definitely requires server side process. This clustet should be between zipkin lib and SkyWalking collector.

[peng-yongsheng]

I'm planning to do this during beta2 release, welcome, discuss how to generate reference node in entry span first. The key question: the reference information are from caller trace, so callee trace must waiting caller trace, but we don't know how long should it take. In most cases, it should be take several seconds, but in extreme cases, it should be take a lot of times.
For achieve high performance, the callee trace should put into memory db such as redis, but if so many caller traces not received for a long times, the memory db will be leaked. Perhaps Mongo DB or ElasticSearch is suitable.

[wu-sheng]

Perhaps Mongo DB or ElasticSearch is suitable

I have a proposal that, use redis with traceid as key, and an expired time(lets say 2 min), the value should a redis queue, push all spans into. Then the bridge could trigger the pull by receiving the entry span.

Although this may be not the best way, need more suggestions. Building trace segment is the most important part, which zipkin didn't have.

[wu-sheng]

@adriancole I think we need some sample data about zipkin, which represents a RPC call like grpc or dubbo, and a MQ trace(maybe a batch process trace, too, if exist in zipkin.)

[candyleer]

wow, it seems that it's more complex than i first thought, cross-thread scenario is missing in my thought.I'm trying to understand zipkin tracing desgin deeply.

[peng-yongsheng]

If huge callee traces received without caller traces in 2 minutes, memory db will leak. So expired time setting js not enough

[wu-sheng]

@candyleer Yes, that is the complex part.

@peng-yongsheng Agree, but as a new part of our ecosystem, if cache implementor is plugable, that willbe good enough for first release.

[wu-sheng]

@candyleer cross-thread scenario is not so important, as a manual instrument, it could deal with it bettet than SkyWalking. Across processes in RPC and MQ are the real difficult parts.

[peng-yongsheng]

OK,just do it.

[wu-sheng]

@candyleer @peng-yongsheng Also I am considering whether we should use collector core for implementing modulization, module dependency, startup sequence, etc.. What do you think?

[peng-yongsheng]

Agree, which repository did we choose? apache/skywalking or openskywalking?

[wu-sheng]

Who would like to start the basic codes framework? @peng-yongsheng or @candyleer ? It is your decision. I am OK with either, they all will be treated as an important part of SkyWalking ecosystem.

@liuhaoyang shows me his interests too. Also @candyleer mentioned jaeger. Maybe we could consider to support it , at least in theory, or do them both by two of you.

I will wait you two finish your suggestions.

And exciting about pushing this to implement.

[codefromthecrypt]

I have limited time and a large ecosystem to help, so will only be working
on zipkin side of things. I think jaeger have a large staff so don't need
volunteer help as much.

[wu-sheng]

For Jaeger, @yurishkuro @pavolloffay @jpkrohling . Do any of you, or Jaeger team member/contributor have time to help to build this integration, or provide us some sample data of trace/span, which represents a RPC call like grpc or dubbo, and a MQ trace(maybe a batch process trace, too.

[peng-yongsheng]

@wu-sheng This is a adapter, not too complex. If @candyleer and @liuhaoyang are interest of this. 2 contributors are enough. I think I could be a supporter for help them to integrate with collector. So, let's discuss the process flow. Who can show the example trace of ZipKin or Jaeger? RPC first.

[wu-sheng]

@adriancole goes to Euro for the next tracing work shop. I assume he is busy, maybe we need to find these data.

[jcchavezs]

Here you can find the zipkin API documentation. I am up to give a hand.

>

https://zipkin.io/zipkin-api/

José Carlos

[yurishkuro]

@wu-sheng sorry, I am not sure I understand the objective here. Can you outline the architecture of this integration test?

[wu-sheng]

@yurishkuro This proposal is about provding a transform server, which transfers data from jaeger SDKs, to SkyWalking collector. Because our community has someone is using zipkin or jaeger instrument sdk, but require SkyWalking backend and visualization.

[yurishkuro]

This is the current data model https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift

[wu-sheng]

@candyleer Are you working on this? If so, could you provide some sample data from Zipkin(or Jaeger if you want to do that), I and @peng-yongsheng could provide you some suggestion about how to implement this feature.

[candyleer]

@wu-sheng yes, I am pending to start.maybe tonight I can provide some sample data

[wu-sheng]

Great to know, I suggest two types of sample data.

• A -(http)->B-(http)->C
• A -(http)->B-(mq)->C

These two type could help me to work with you.

[codefromthecrypt]

this app is 2 stage but has both patterns

https://github.com/openzipkin/sleuth-webmvc-example

On Thu, 3 May 2018, 05:11 吴晟 Wu Sheng, [e-mail] wrote:

Great to know, I suggest two types of sample data.

• A -(http)->B-(http)->C
• A -(http)->B-(mq)->C

These two type could help me to work with you.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-skywalking/issues/665#issuecomment-386181340,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAD613Qx_Rsm4KrNTT91Z9KeP2iqWQEpks5tunV4gaJpZM4Q_Ayz
.

[candyleer]

for Zipkin,this case A(http)->B(http)->C(http) sample span data:

A

{
    "traceId": "152057c40ecb8c68",
    "id": "152057c40ecb8c68",
    "kind": "SERVER",
    "name": "get /",
    "timestamp": 1525402389314992,
    "duration": 597915,
    "localEndpoint": {
        "serviceName": "test_7070",
        "ipv4": "10.12.194.200"
    },
    "remoteEndpoint": {
        "ipv4": "127.0.0.1",
        "port": 54221
    },
    "tags": {
        "http.method": "GET",
        "http.path": "/",
        "mvc.controller.class": "Frontend",
        "mvc.controller.method": "callBackend"
    }
}

B

{
    "traceId": "152057c40ecb8c68",
    "parentId": "152057c40ecb8c68",
    "id": "5225a2741bfda4d2",
    "kind": "SERVER",
    "name": "get /api",
    "timestamp": 1525402389445178,
    "duration": 386003,
    "localEndpoint": {
        "serviceName": "test_8080",
        "ipv4": "10.12.194.200"
    },
    "remoteEndpoint": {
        "ipv4": "127.0.0.1",
        "port": 54222
    },
    "tags": {
        "http.method": "GET",
        "http.path": "/api",
        "mvc.controller.class": "Backend",
        "mvc.controller.method": "printDate"
    },
    "shared": true
}

C

{
    "traceId": "152057c40ecb8c68",
    "parentId": "5225a2741bfda4d2",
    "id": "b252016ba06bbbf8",
    "kind": "SERVER",
    "name": "get /last",
    "timestamp": 1525402389670882,
    "duration": 75574,
    "localEndpoint": {
        "serviceName": "test_9000",
        "ipv4": "10.12.194.200"
    },
    "remoteEndpoint": {
        "ipv4": "127.0.0.1",
        "port": 54223
    },
    "tags": {
        "http.method": "GET",
        "http.path": "/last",
        "mvc.controller.class": "Backend",
        "mvc.controller.method": "last"
    },
    "shared": true
}

[wu-sheng]

@candyleer Thanks for providing this sample data. I suggest we could work on this first with a prototype.

1. Add a new module in collector, named zipkin-data-receiver. @peng-yongsheng Do you agree or object?
2. Provide a Jetty based HTTP service to receive these spans. I think the zipkin SDKs will send these to collector separately. Right?
3. Create a in-memory buffer to receive all spans.
4. Trigger trace finished when received the first span (in A), when find the span of trace has no parentId. In finish process, make all spans in a list, based on zipkin module.

For these steps, I suggest you submit pull request for each one, for making review easier.

For the concerns you may have:

• How to process collector cluster?
  No, we don't consider that in the prototype building stage.
• How to process MQ trace?
  I will talk with you about this, later. Don't want to make the prototype too complex.

And for @adriancole , do zipkin have provided grpc or protobuf format already(released)?

[peng-yongsheng]

1. We need to support ZipKin and Jaeger agent. Suggest those to be maven third party module's submodule, but also collector top submodule. Such as:

  apm-collector-thirdparty-receiver
     receiver-zipkin
     receiver-jaeger

1. Is Zipkin has service discovery feature? How to make ZipKin agents balance send traces?
2. Suggest use Json stream reader to parse traces.

[wu-sheng]

apm-collector-thirdparty-agent

I didn't like the agent part, even we are using this in existed collector module. It should be considered as a receiver, agent is for monitored application only. And for Zipkin and Jaeger, it is SDK libs, not an agent.

Is Zipkin has service discovery feature? How to make ZipKin agents balance send traces?

This could be discussed, but please not put this in high priority. I hope the prototype should only run in standalone mode.

Suggest use Json stream reader to parse traces.

Agree. I asked Adrian, just because I want to know more.

[wu-sheng]

@candyleer Can you provide me your zipkin demo? I will do this in these days.

[wu-sheng]

origin/zipkin/receiver-v1 branch has been created. I am going to build the skeleton for Zipkin data, started with json format.

[wu-sheng]

See SkyWalking supports Zipkin formats in https://github.com/apache/incubator-skywalking/pull/1273 . Only tested in a single case. I hope the community can start in that way. FYI @candyleer

[wu-sheng]

Move to #2044 . A more reliable and high-performance way to do this, and make them production ready.