Jaeger: collector: Failed to insert span: failed to Exec query & agent: timeout to collector

[Dieterbe]

Hello,
I'm seeing 2 problems and I'm not sure they are related.
jaeger is deployed using the production template at https://github.com/jaegertracing/jaeger-kubernetes

problem 1 : collector errors

~ ❯❯❯ kubej logs jaeger-collector-2191353639-87k5x
{"level":"error","ts":1505932275.2450123,"caller":"metrics/table.go:54","msg":"Failed to exec query","query":"[query statement=\"\\n\\t\\tINSERT\\n\\t\\tINTO traces(trace_id, span_id, span_hash, parent_id, operation_name, flags,\\n\\t\\t\\t\\t    start_time, duration, tags, logs, refs, process)\\n\\t\\tVALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\" values=[2a07d8276378df0f 1370522953428962601 -2132593763436613675 685406953 \u0019\u000c\u0019\u000c\u0000\u0016̔\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\u0001 359177838 1820226890504781 -10 [] [] [] {ServiceName:metrictank Tags:[{Key:jaeger.version ValueType:string ValueString:Go-2.9.1dev ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:hostname ValueType:string ValueString:mt-read02-12574-medium-ops-a-563083619-cjmt5 ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:ip ValueType:string ValueString:172.20.128.8 ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]}]}] consistency=LOCAL_ONE]","error":"String didn't validate.","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:209\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:273\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:176\ngithub.com/uber/jaeger/pkg/cassandra/metrics.(*Table).Exec\n\t/home/travis/gopath/src/github.com/uber/jaeger/pkg/cassandra/metrics/table.go:54\ngithub.com/uber/jaeger/plugin/storage/cassandra/spanstore.(*SpanWriter).WriteSpan\n\t/home/travis/gopath/src/github.com/uber/jaeger/plugin/storage/cassandra/spanstore/writer.go:142\ngithub.com/uber/jaeger/cmd/collector/app.(*spanProcessor).saveSpan\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:108\ngithub.com/uber/jaeger/cmd/collector/app.(*spanProcessor).(github.com/uber/jaeger/cmd/collector/app.saveSpan)-fm\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:95\ngithub.com/uber/jaeger/cmd/collector/app.ChainedProcessSpan.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/model_consumer.go:38\ngithub.com/uber/jaeger/cmd/collector/app.(*spanProcessor).processItemFromQueue\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:132\ngithub.com/uber/jaeger/cmd/collector/app.NewSpanProcessor.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:63\ngithub.com/uber/jaeger/pkg/queue.(*BoundedQueue).StartConsumers.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/pkg/queue/bounded_queue.go:71"}
{"level":"error","ts":1505932275.248957,"caller":"spanstore/writer.go:237","msg":"Failed to insert span","trace_id":"2a07d8276378df0f","span_id":1370522953428962601,"error":"failed to Exec query '[query statement=\"\\n\\t\\tINSERT\\n\\t\\tINTO traces(trace_id, span_id, span_hash, parent_id, operation_name, flags,\\n\\t\\t\\t\\t    start_time, duration, tags, logs, refs, process)\\n\\t\\tVALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\" values=[2a07d8276378df0f 1370522953428962601 -2132593763436613675 685406953 \u0019\u000c\u0019\u000c\u0000\u0016̔\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\u0001 359177838 1820226890504781 -10 [] [] [] {ServiceName:metrictank Tags:[{Key:jaeger.version ValueType:string ValueString:Go-2.9.1dev ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:hostname ValueType:string ValueString:mt-read02-12574-medium-ops-a-563083619-cjmt5 ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:ip ValueType:string ValueString:172.20.128.8 ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]}]}] consistency=LOCAL_ONE]': String didn't validate.","errorVerbose":"String didn't validate.\nfailed to Exec query '[query statement=\"\\n\\t\\tINSERT\\n\\t\\tINTO traces(trace_id, span_id, span_hash, parent_id, operation_name, flags,\\n\\t\\t\\t\\t    start_time, duration, tags, logs, refs, process)\\n\\t\\tVALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\" values=[2a07d8276378df0f 1370522953428962601 -2132593763436613675 685406953 \u0019\u000c\u0019\u000c\u0000\u0016̔\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\u0001 359177838 1820226890504781 -10 [] [] [] {ServiceName:metrictank Tags:[{Key:jaeger.version ValueType:string ValueString:Go-2.9.1dev ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:hostname ValueType:string ValueString:mt-read02-12574-medium-ops-a-563083619-cjmt5 ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:ip ValueType:string ValueString:172.20.128.8 ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]}]}] consistency=LOCAL_ONE]'\ngithub.com/uber/jaeger/pkg/cassandra/metrics.(*Table).Exec\n\t/home/travis/gopath/src/github.com/uber/jaeger/pkg/cassandra/metrics/table.go:56\ngithub.com/uber/jaeger/plugin/storage/cassandra/spanstore.(*SpanWriter).WriteSpan\n\t/home/travis/gopath/src/github.com/uber/jaeger/plugin/storage/cassandra/spanstore/writer.go:142\ngithub.com/uber/jaeger/cmd/collector/app.(*spanProcessor).saveSpan\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:108\ngithub.com/uber/jaeger/cmd/collector/app.(*spanProcessor).(github.com/uber/jaeger/cmd/collector/app.saveSpan)-fm\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:95\ngithub.com/uber/jaeger/cmd/collector/app.ChainedProcessSpan.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/model_consumer.go:38\ngithub.com/uber/jaeger/cmd/collector/app.(*spanProcessor).processItemFromQueue\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:132\ngithub.com/uber/jaeger/cmd/collector/app.NewSpanProcessor.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:63\ngithub.com/uber/jaeger/pkg/queue.(*BoundedQueue).StartConsumers.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/pkg/queue/bounded_queue.go:71\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/asm_amd64.s:2086","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:209\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:273\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:176\ngithub.com/uber/jaeger/plugin/storage/cassandra/spanstore.(*SpanWriter).logError\n\t/home/travis/gopath/src/github.com/uber/jaeger/plugin/storage/cassandra/spanstore/writer.go:237\ngithub.com/uber/jaeger/plugin/storage/cassandra/spanstore.(*SpanWriter).WriteSpan\n\t/home/travis/gopath/src/github.com/uber/jaeger/plugin/storage/cassandra/spanstore/writer.go:143\ngithub.com/uber/jaeger/cmd/collector/app.(*spanProcessor).saveSpan\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:108\ngithub.com/uber/jaeger/cmd/collector/app.(*spanProcessor).(github.com/uber/jaeger/cmd/collector/app.saveSpan)-fm\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:95\ngithub.com/uber/jaeger/cmd/collector/app.ChainedProcessSpan.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/model_consumer.go:38\ngithub.com/uber/jaeger/cmd/collector/app.(*spanProcessor).processItemFromQueue\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:132\ngithub.com/uber/jaeger/cmd/collector/app.NewSpanProcessor.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:63\ngithub.com/uber/jaeger/pkg/queue.(*BoundedQueue).StartConsumers.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/pkg/queue/bounded_queue.go:71"}
{"level":"error","ts":1505932275.249009,"caller":"app/span_processor.go:109","msg":"Failed to save span","error":"Failed to insert span: failed to Exec query '[query statement=\"\\n\\t\\tINSERT\\n\\t\\tINTO traces(trace_id, span_id, span_hash, parent_id, operation_name, flags,\\n\\t\\t\\t\\t    start_time, duration, tags, logs, refs, process)\\n\\t\\tVALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\" values=[2a07d8276378df0f 1370522953428962601 -2132593763436613675 685406953 \u0019\u000c\u0019\u000c\u0000\u0016̔\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\u0001 359177838 1820226890504781 -10 [] [] [] {ServiceName:metrictank Tags:[{Key:jaeger.version ValueType:string ValueString:Go-2.9.1dev ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:hostname ValueType:string ValueString:mt-read02-12574-medium-ops-a-563083619-cjmt5 ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:ip ValueType:string ValueString:172.20.128.8 ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]}]}] consistency=LOCAL_ONE]': String didn't validate.","errorVerbose":"String didn't validate.\nfailed to Exec query '[query statement=\"\\n\\t\\tINSERT\\n\\t\\tINTO traces(trace_id, span_id, span_hash, parent_id, operation_name, flags,\\n\\t\\t\\t\\t    start_time, duration, tags, logs, refs, process)\\n\\t\\tVALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\" values=[2a07d8276378df0f 1370522953428962601 -2132593763436613675 685406953 \u0019\u000c\u0019\u000c\u0000\u0016̔\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\u0001 359177838 1820226890504781 -10 [] [] [] {ServiceName:metrictank Tags:[{Key:jaeger.version ValueType:string ValueString:Go-2.9.1dev ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:hostname ValueType:string ValueString:mt-read02-12574-medium-ops-a-563083619-cjmt5 ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:ip ValueType:string ValueString:172.20.128.8 ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]}]}] consistency=LOCAL_ONE]'\ngithub.com/uber/jaeger/pkg/cassandra/metrics.(*Table).Exec\n\t/home/travis/gopath/src/github.com/uber/jaeger/pkg/cassandra/metrics/table.go:56\ngithub.com/uber/jaeger/plugin/storage/cassandra/spanstore.(*SpanWriter).WriteSpan\n\t/home/travis/gopath/src/github.com/uber/jaeger/plugin/storage/cassandra/spanstore/writer.go:142\ngithub.com/uber/jaeger/cmd/collector/app.(*spanProcessor).saveSpan\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:108\ngithub.com/uber/jaeger/cmd/collector/app.(*spanProcessor).(github.com/uber/jaeger/cmd/collector/app.saveSpan)-fm\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:95\ngithub.com/uber/jaeger/cmd/collector/app.ChainedProcessSpan.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/model_consumer.go:38\ngithub.com/uber/jaeger/cmd/collector/app.(*spanProcessor).processItemFromQueue\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:132\ngithub.com/uber/jaeger/cmd/collector/app.NewSpanProcessor.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:63\ngithub.com/uber/jaeger/pkg/queue.(*BoundedQueue).StartConsumers.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/pkg/queue/bounded_queue.go:71\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/asm_amd64.s:2086\nFailed to insert span\ngithub.com/uber/jaeger/plugin/storage/cassandra/spanstore.(*SpanWriter).logError\n\t/home/travis/gopath/src/github.com/uber/jaeger/plugin/storage/cassandra/spanstore/writer.go:238\ngithub.com/uber/jaeger/plugin/storage/cassandra/spanstore.(*SpanWriter).WriteSpan\n\t/home/travis/gopath/src/github.com/uber/jaeger/plugin/storage/cassandra/spanstore/writer.go:143\ngithub.com/uber/jaeger/cmd/collector/app.(*spanProcessor).saveSpan\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:108\ngithub.com/uber/jaeger/cmd/collector/app.(*spanProcessor).(github.com/uber/jaeger/cmd/collector/app.saveSpan)-fm\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:95\ngithub.com/uber/jaeger/cmd/collector/app.ChainedProcessSpan.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/model_consumer.go:38\ngithub.com/uber/jaeger/cmd/collector/app.(*spanProcessor).processItemFromQueue\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:132\ngithub.com/uber/jaeger/cmd/collector/app.NewSpanProcessor.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:63\ngithub.com/uber/jaeger/pkg/queue.(*BoundedQueue).StartConsumers.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/pkg/queue/bounded_queue.go:71\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/asm_amd64.s:2086","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:209\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:273\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:176\ngithub.com/uber/jaeger/cmd/collector/app.(*spanProcessor).saveSpan\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:109\ngithub.com/uber/jaeger/cmd/collector/app.(*spanProcessor).(github.com/uber/jaeger/cmd/collector/app.saveSpan)-fm\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:95\ngithub.com/uber/jaeger/cmd/collector/app.ChainedProcessSpan.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/model_consumer.go:38\ngithub.com/uber/jaeger/cmd/collector/app.(*spanProcessor).processItemFromQueue\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:132\ngithub.com/uber/jaeger/cmd/collector/app.NewSpanProcessor.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/span_processor.go:63\ngithub.com/uber/jaeger/pkg/queue.(*BoundedQueue).StartConsumers.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/pkg/queue/bounded_queue.go:71"}

note that all messages were at the same time, about 6 days ago.
not sure if this provides more info, but fwiw:

~ ❯❯❯ kubej get po
NAME                                READY     STATUS    RESTARTS   AGE
cassandra-0                         1/1       Running   2          27d
cassandra-1                         1/1       Running   2          27d
cassandra-2                         1/1       Running   5          27d
jaeger-agent-1133288941-xlgp3       1/1       Running   18         8d
jaeger-collector-2191353639-87k5x   1/1       Running   2          27d
jaeger-query-241882977-19q89        1/1       Running   2          27d
~ ❯❯❯ kubej logs jaeger-collector-2191353639-87k5x -p
2017/08/30 07:04:43 gocql: unable to dial control conn 172.25.0.4: dial tcp 172.25.0.4:9042: getsockopt: connection refused
{"level":"fatal","ts":1504076683.164841,"caller":"collector/main.go:67","msg":"Unable to build span handlers","error":"gocql: unable to create session: control: unable to connect to initial hosts: dial tcp 172.25.0.4:9042: getsockopt: connection refused","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:209\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:273\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Fatal\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:208\nmain.main\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/main.go:67"}
~ ❯❯❯ date -d @1504076683
Wed Aug 30 09:04:43 CEST 2017

problem 2

my app reports:

2017/09/26 09:17:05 ERROR: error reporting span "getTargetsLocal": write udp 172.21.192.18:42191->100.90.41.92:6831: write: connection refused

i checked the k8s svc and that is setup properly. the agent logs:

~ ❯❯❯ kubej logs jaeger-agent-1133288941-xlgp3
{"level":"info","ts":1506159461.5915134,"caller":"tchannel/bulider.go:89","msg":"Enabling service discovery","service":"jaeger-collector"}
{"level":"info","ts":1506159461.5915678,"caller":"peerlistmgr/peer_list_mgr.go:111","msg":"Registering active peer","peer":"jaeger-collector.jaeger.svc:14267"}
{"level":"info","ts":1506159461.5922403,"caller":"agent/main.go:50","msg":"Starting agent"}
{"level":"info","ts":1506159462.5920308,"caller":"peerlistmgr/peer_list_mgr.go:159","msg":"Not enough connected peers","connected":0,"required":1}
{"level":"info","ts":1506159462.5920768,"caller":"peerlistmgr/peer_list_mgr.go:166","msg":"Trying to connect to peer","host:port":"jaeger-collector.jaeger.svc:14267"}
{"level":"info","ts":1506159462.599572,"caller":"peerlistmgr/peer_list_mgr.go:176","msg":"Connected to peer","host:port":"[::]:14267"}
{"level":"error","ts":1506160384.136238,"caller":"tchannel/reporter.go:131","msg":"Could not submit jaeger batch","error":"tchannel error ErrCodeTimeout: timeout","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:191\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:301\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:202\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).submitAndReport\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:131\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).EmitBatch\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:121\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*agentProcessorEmitBatch).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:137\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*AgentProcessor).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:111\ngithub.com/uber/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/processors/thrift_processor.go:110"}
{"level":"error","ts":1506160680.4919758,"caller":"tchannel/reporter.go:131","msg":"Could not submit jaeger batch","error":"tchannel error ErrCodeTimeout: timeout","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:191\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:301\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:202\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).submitAndReport\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:131\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).EmitBatch\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:121\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*agentProcessorEmitBatch).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:137\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*AgentProcessor).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:111\ngithub.com/uber/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/processors/thrift_processor.go:110"}
{"level":"error","ts":1506160682.406441,"caller":"tchannel/reporter.go:131","msg":"Could not submit jaeger batch","error":"tchannel error ErrCodeTimeout: timeout","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:191\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:301\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:202\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).submitAndReport\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:131\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).EmitBatch\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:121\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*agentProcessorEmitBatch).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:137\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*AgentProcessor).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:111\ngithub.com/uber/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/processors/thrift_processor.go:110"}
{"level":"error","ts":1506160683.9214954,"caller":"tchannel/reporter.go:131","msg":"Could not submit jaeger batch","error":"tchannel error ErrCodeTimeout: timeout","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:191\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:301\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:202\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).submitAndReport\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:131\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).EmitBatch\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:121\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*agentProcessorEmitBatch).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:137\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*AgentProcessor).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:111\ngithub.com/uber/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/processors/thrift_processor.go:110"}
{"level":"error","ts":1506160684.2446356,"caller":"tchannel/reporter.go:131","msg":"Could not submit jaeger batch","error":"tchannel error ErrCodeTimeout: timeout","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:191\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:301\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:202\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).submitAndReport\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:131\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).EmitBatch\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:121\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*agentProcessorEmitBatch).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:137\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*AgentProcessor).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:111\ngithub.com/uber/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/processors/thrift_processor.go:110"}
{"level":"error","ts":1506160684.8248649,"caller":"tchannel/reporter.go:131","msg":"Could not submit jaeger batch","error":"tchannel error ErrCodeTimeout: timeout","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:191\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:301\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:202\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).submitAndReport\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:131\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).EmitBatch\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:121\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*agentProcessorEmitBatch).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:137\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*AgentProcessor).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:111\ngithub.com/uber/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/processors/thrift_processor.go:110"}
{"level":"error","ts":1506160685.1431754,"caller":"tchannel/reporter.go:131","msg":"Could not submit jaeger batch","error":"tchannel error ErrCodeTimeout: timeout","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:191\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:301\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:202\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).submitAndReport\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:131\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).EmitBatch\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:121\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*agentProcessorEmitBatch).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:137\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*AgentProcessor).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:111\ngithub.com/uber/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/processors/thrift_processor.go:110"}
{"level":"error","ts":1506160686.8288321,"caller":"tchannel/reporter.go:131","msg":"Could not submit jaeger batch","error":"tchannel error ErrCodeTimeout: timeout","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:191\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:301\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:202\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).submitAndReport\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:131\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).EmitBatch\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:121\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*agentProcessorEmitBatch).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:137\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*AgentProcessor).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:111\ngithub.com/uber/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/processors/thrift_processor.go:110"}
{"level":"error","ts":1506160686.832627,"caller":"tchannel/reporter.go:131","msg":"Could not submit jaeger batch","error":"tchannel error ErrCodeTimeout: timeout","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:191\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:301\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:202\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).submitAndReport\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:131\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).EmitBatch\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:121\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*agentProcessorEmitBatch).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:137\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*AgentProcessor).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:111\ngithub.com/uber/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/processors/thrift_processor.go:110"}
{"level":"error","ts":1506160686.8339095,"caller":"tchannel/reporter.go:131","msg":"Could not submit jaeger batch","error":"tchannel error ErrCodeTimeout: timeout","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:191\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:301\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:202\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).submitAndReport\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:131\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).EmitBatch\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:121\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*agentProcessorEmitBatch).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:137\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*AgentProcessor).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:111\ngithub.com/uber/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/processors/thrift_processor.go:110"}
{"level":"error","ts":1506160686.8357015,"caller":"tchannel/reporter.go:131","msg":"Could not submit jaeger batch","error":"tchannel error ErrCodeTimeout: timeout","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:191\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:301\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:202\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).submitAndReport\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:131\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).EmitBatch\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:121\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*agentProcessorEmitBatch).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:137\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*AgentProcessor).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:111\ngithub.com/uber/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/processors/thrift_processor.go:110"}
{"level":"error","ts":1506160686.847109,"caller":"tchannel/reporter.go:131","msg":"Could not submit jaeger batch","error":"tchannel error ErrCodeTimeout: timeout","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:191\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:301\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:202\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).submitAndReport\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:131\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).EmitBatch\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:121\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*agentProcessorEmitBatch).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:137\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*AgentProcessor).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:111\ngithub.com/uber/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/processors/thrift_processor.go:110"}
{"level":"error","ts":1506160686.8483703,"caller":"tchannel/reporter.go:131","msg":"Could not submit jaeger batch","error":"tchannel error ErrCodeTimeout: timeout","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:191\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:301\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:202\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).submitAndReport\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:131\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).EmitBatch\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:121\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*agentProcessorEmitBatch).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:137\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*AgentProcessor).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:111\ngithub.com/uber/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/processors/thrift_processor.go:110"}
{"level":"error","ts":1506160686.8502076,"caller":"tchannel/reporter.go:131","msg":"Could not submit jaeger batch","error":"tchannel error ErrCodeTimeout: timeout","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:191\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:301\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:202\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).submitAndReport\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:131\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).EmitBatch\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:121\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*agentProcessorEmitBatch).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:137\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*AgentProcessor).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:111\ngithub.com/uber/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/processors/thrift_processor.go:110"}
{"level":"error","ts":1506160686.8767972,"caller":"tchannel/reporter.go:131","msg":"Could not submit jaeger batch","error":"tchannel error ErrCodeTimeout: timeout","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:191\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:301\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:202\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).submitAndReport\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:131\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).EmitBatch\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:121\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*agentProcessorEmitBatch).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:137\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*AgentProcessor).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:111\ngithub.com/uber/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/processors/thrift_processor.go:110"}
{"level":"error","ts":1506160686.8807807,"caller":"tchannel/reporter.go:131","msg":"Could not submit jaeger batch","error":"tchannel error ErrCodeTimeout: timeout","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:191\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:301\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:202\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).submitAndReport\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:131\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).EmitBatch\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:121\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*agentProcessorEmitBatch).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:137\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*AgentProcessor).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:111\ngithub.com/uber/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/processors/thrift_processor.go:110"}
{"level":"error","ts":1506160686.8808224,"caller":"tchannel/reporter.go:131","msg":"Could not submit jaeger batch","error":"tchannel error ErrCodeTimeout: timeout","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:191\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:301\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:202\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).submitAndReport\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:131\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).EmitBatch\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:121\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*agentProcessorEmitBatch).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:137\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*AgentProcessor).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:111\ngithub.com/uber/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/processors/thrift_processor.go:110"}
{"level":"error","ts":1506162803.0297039,"caller":"tchannel/reporter.go:131","msg":"Could not submit jaeger batch","error":"tchannel error ErrCodeTimeout: timeout","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:191\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:301\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:202\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).submitAndReport\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:131\ngithub.com/uber/jaeger/cmd/agent/app/reporter/tchannel.(*Reporter).EmitBatch\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/reporter/tchannel/reporter.go:121\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*agentProcessorEmitBatch).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:137\ngithub.com/uber/jaeger/thrift-gen/jaeger.(*AgentProcessor).Process\n\t/home/travis/gopath/src/github.com/uber/jaeger/thrift-gen/jaeger/agent.go:111\ngithub.com/uber/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/agent/app/processors/thrift_processor.go:110"}

[jpkrohling]

Was it a one-time error, or are you having troubles storing all spans? The first error has a message of String didn't validate. , which suggests an encoding issue. It would be useful if you could share the contents of the span (if you have it).

The second output (still on the first issue) is strange: it shows that the collector is in the Running state, but the logs would suggest it to be Failed (fatal error on connecting to Cassandra).

The third output could be related to the second. In both cases, I would try to scale up the pods, killing the first one right after the scale operation completes. If that works, I'd say that there was a networking issue at some time and Jaeger couldn't recover from that. In that case, it would make sense to notify the health check endpoint, so that Kubernetes can replace the problematic one.

[Dieterbe]

Was it a one-time error, or are you having troubles storing all spans? The first error has a message of String didn't validate. , which suggests an encoding issue. It would be useful if you could share the contents of the span (if you have it).

i'm not able to store any span, due to issue 2 (my app consistently logs ERROR: error reporting span ... write udp 172.21.192.18:42191->100.90.41.92:6831: write: connection refused and nothing shows up in jaeger-query when i search)
I don't have the contents of the span, I think if they are not sent they just get dropped.
FYI the app is open source and the jaeger initialisation code is at https://github.com/raintank/metrictank/blob/master/conf/tracing.go#L12

The second output (still on the first issue) is strange: it shows that the collector is in the Running state, but the logs would suggest it to be Failed (fatal error on connecting to Cassandra).

this is 99.9% surely what happened when i initially deployed things, and cassandra was not up yet. so we can ignore that.
the reason i shared this log is more to demonstrate that nothing interesting happened after startup, until the other logs, e.g. in a period of 27 days.

In both cases, I would try to scale up the pods, killing the first one right after the scale operation completes.

which, collector or agent? both?

[jpkrohling]

which, collector or agent? both?

Try the agent first. Your application should be able to send the spans to the agent without errors (you might want to enable JAEGER_REPORTER_LOG_SPANS to make sure spans are sent to the agent, see https://github.com/uber/jaeger-client-java/tree/master/jaeger-core). If the Agent is still reporting issues when connecting to the collector, then do the same with the collector (perhaps followed by the agent again, to make sure it can connect to the collector upon boot).

[Dieterbe]

ok as soon as i launched a new jaeger pod and killed the old one, the write udp 172.21.192.18:42191->100.90.41.92:6831: write: connection refused errors disappeared, traces show up in the UI, and the agents don't log errors.

i didn't touch collectors yet. but your trick successfully works around issue 2, and this seems to confirm issue 1 and 2 are unrelated.

what exactly did you mean with:

In that case, it would make sense to notify the health check endpoint, so that Kubernetes can replace the problematic one.

i presume you meant s/notify/modify/ if so modify how?

[jpkrohling]

i presume you meant s/notify/modify/ if so modify how?

Both, perhaps :) There's a health check handler[1] which is currently very simple. The idea is to notify it "somehow", setting its current state to 500 (internal server error). On the jaeger-collector deployment, a liveness check would need to be added (I thought we had one already!)[2], checking this handler's port (14269).

1: https://github.com/jaegertracing/jaeger/blob/master/pkg/healthcheck/handler.go
2: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#define-a-liveness-http-request

[jpkrohling]

new jaeger pod

Just realized you meant you did not touch the collector yet, so, I assume "new jaeger pod" meant Agent. It has no health checks yet, and I'm not quite sure if the same approach could be used. Perhaps it would be more appropriate to just exit the Agent? Kubernetes will re-schedule a new container once the one in error exits (so should systemd, monit and others).

[Dieterbe]

to me this looks quite simply like a bug in the agent. it shouldn't be necessary to restart it in this case. if it times out talking to the collector, that should just resolve itself rather than requiring a restart to resolve.
so then that also means that exiting or doing this via a healthcheck would be pointless IMHO.

(yes, i meant, agent pod)

[jpkrohling]

That's another way of viewing that, yes. I would still like to see some sort of failure/notification in case the Agent can't contact the collector for an extended period of time, though, as I believe the agent would just start dropping spans if its buffer is full.

[yurishkuro]

The agent has reconnect logic if collector drops the connection.

[jpkrohling]

The agent has reconnect logic if collector drops the connection.

That didn't seem to be enough to get his Agent back in a working state :-/

[Dieterbe]

@yurishkuro next time this (issue 2) happens (i.e. app getting conn refused to agent, and agent logging timeouts, and a restart fixes it) what I can do to provide diagnostics before i restart the process? a stacktrace? BTW maybe the agent should get a pprof endpoint (i noticed http://localhost:5778/debug/pprof doesn't work, though i can use sigquit also)

regarding issue 1, is this most likely an issue on my end? (e.g. the code that calls jaeger)

[Dieterbe]

i just had it happen again, and what happened is agent-linux process got OOM killed. but the interesting thing is for some reason the container keeps running, i'll look into how the docker container/image is set up to see if i find the problem.

edit: actually, docker got into a really weird state where the container was no longer running (e.g. the container shim process gone) but yet it thought it still was, so k8s thought the same.

this also demonstrates that the timeout errors have nothing to do with my app not being able to send spans. the actual problem for issue 2 is a docker bug + agent getting oom killed

[ffledgling]

@Dieterbe Had this exact same thing happen to our cluster with the same docker weird state symptom. Did you get get to the bottom of this? Or figure out which docker bug causes this?

[jpkrohling]

@Dieterbe, @ffledgling, did you experience this with a newer Jaeger version? The latest defaults to gRPC and I believe it retries accordingly in case of a connectivity issue.

[Dieterbe]

I haven't used jaeger in quite a while and am unaware of this still happens. If this should be fixed, feel free to close.

[jpkrohling]

I'm closing this, but feel free to reopen if this is still happening.