Jaeger: about tracing and log management [not bug]

[zplzpl]

I learned that jaeger itself supports printing log content, and uses the log library for zap. Does uber's microservices use jaeger's logging method when printing logs? Or use zap print logs separately, and then use other log management schemes, such as ELK?

I want to use this question to understand, how does uber do tracing and log management, is tracing+log, or independent tracing, independent log, or depends on the scene?

thank you, look forward to getting a reply.

I summed up the above, what I actually want to know is: Should a tracing system also manage logging? What is the uber?

[yurishkuro]

We have a separate logging system.

[zplzpl]

@yurishkuro

1. what is the way the log is output, can you introduce it?
2. is the zap package used?
3. so when is the jaeger log output used?

[yurishkuro]

are you asking about logs produced by Jaeger backend or your own application logs?

[zplzpl]

1. what is the way the log is output, can you introduce it?
2. is the zap package used?
3. so when is the jaeger log output used?

1 and 2 are asking uber's log management.
3 is asking about logs produced by Jaeger backend.

[yurishkuro]

1 and 2 - there are different ways, sometimes it's written to kafka directly, sometimes to local disk and then shipped using logstash.

Not clear what you mean by 3. When do you use, say, logs from Cassandra? It's the same with Jaeger backend.

[zplzpl]

3: Jaeger's span designed the way log output, then when is it used?

[zplzpl]

spans: [
{
traceID: "69327f3939693126",
spanID: "38f273f050cade5a",
flags: 1,
operationName: "CategoryPathRequest",
references: [],
startTime: 1533004327723671,
duration: 2692,
tags: [],
logs: [], <-------------------------------- this logs
processID: "p1",
warnings: null
}
]

when is it used?

[yurishkuro]

I see - it is used to log events that are important for understanding the tracing data. It's OpenTracing concept. You can see examples in my blogs:

https://medium.com/opentracing/tracing-http-request-latency-in-go-with-opentracing-7cc1282a100a
https://medium.com/opentracing/take-opentracing-for-a-hotrod-ride-f6e3141f7941

[zplzpl]

Thank you, I learned a lot.

[zplzpl]

I looked at examples-hotrod and had a question about the spanlogger.go logToSpan method. If the level is ERROR, then the stracktrace information is not recorded to span.logs?

[yurishkuro]

Two things here. 1- Jaeger data model does not have a std structure to capture stack traces. We might add it in the future. 2- Go does not have a standard way of propagating stack traces in errors. It's possible to build special logic for pkg/errors module, but we didn't bother in this example.

These are all implementation details that can be addressed. The main point was that it is possible to integrate logs with traces, but in langs like Go it requires changes to the logging API to accept the context.

[isaachier]

I think it might be worth adding occasional/sampled stack traces at some point in the future. pprof made a Protobuf schema to represent stack traces in a language-agnostic way here. We can try using the Protobuf directly or just invent a similar solution and log it into the span.

[jpkrohling]

I'm closing this one, as I think the original question has been answered. If that's not the case, feel free to reopen.

[zplzpl]

@yurishkuro

github.com\jaegertracing\jaeger\examples\hotrod\pkg\log\spanlogger.go

func (sl spanLogger) Error(msg string, fields ...zapcore.Field) {
    sl.logToSpan("error", msg, fields...)
    sl.logger.Error(msg, fields...)
}

How does ZAP share the same stacktrace with jaeger? Do you have any good ideas?

[yurishkuro]

Please realize that HotROD is just a proof of concept example. It has this half-implemented `fieldAdapter' that maps Zap log fields into OpenTracing log fields.

https://github.com/jaegertracing/jaeger/blob/d7bce6aa19378332046b66b21e86b56074e0b53b/examples/hotrod/pkg/log/spanlogger.go#L64-L72

So in theory if someone logs the error to Zap via logger.Info("...", zap.Error(err)) this adapter will get that and can translate into Span API appropriately.

Go does not officially support stack traces, but there's a common pkg/errors that does, and I assume Zap recognizes that and extracts the stack trace. Same would need to be done in the span adapter.

[zplzpl]

@yurishkuro Stacktrace storage costs are expensive, do you agree to add stacktrace information to span.logs?

[yurishkuro]

I suggest you decide that on your own, based on your circumstances and environment.

[zplzpl]

@yurishkuro

I use pkg/errors.
When I print the error, I found that errorVerbose and stacktrace are the same. What is the purpose of errorVerbose?

I am here to ask, is the reason why I think you should understand this.

err := errors.New("hahaha")
zapLogger.Error("test", zap.Error(err))

{
"level": "error",
"ts": 1534731035.8522868,
"caller": "log/zap_test.go:35",
"msg": "test",
"error": "hahaha",
"errorVerbose": "hahaha\naz/lib/log.TestSpanLogger_Error\n\tE:/goapp/src/az/lib/log/zap_test.go:33\ntesting.tRunner\n\tE:/go1.10.2.windows-amd64/go/src/testing/testing.go:777\nruntime.goexit\n\tE:/go1.10.2.windows-amd64/go/src/runtime/asm_amd64.s:2361",
"stacktrace": "az/lib/log.TestSpanLogger_Error\n\tE:/goapp/src/az/lib/log/zap_test.go:35\ntesting.tRunner\n\tE:/go1.10.2.windows-amd64/go/src/testing/testing.go:777"
}

[zplzpl]

When using pkg/errors, stacktrace looks a bit redundant.