Jaeger: Jaeger UI hanged on "Find Traces" button

[zhufeizzz]

OS Version: SUSE Linux Enterprise Server 12 SP5 (x86_64)
Jaeger Version: 1.18.0 all-in-one
Storage Type : elasticsearch

It aways loading like this, and never response.

Server Logs:

2020/12/08 15:03:16 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
{"level":"info","ts":1607410996.354466,"caller":"flags/service.go:116","msg":"Mounting metrics handler on admin server","route":"/metrics"}
{"level":"info","ts":1607410996.3545704,"caller":"flags/admin.go:120","msg":"Mounting health check on admin server","route":"/"}
{"level":"info","ts":1607410996.3546174,"caller":"flags/admin.go:126","msg":"Starting admin HTTP server","http-addr":":14269"}
{"level":"info","ts":1607410996.3546305,"caller":"flags/admin.go:112","msg":"Admin server started","http.host-port":"[::]:14269","health-status":"unavailable"}
{"level":"info","ts":1607410996.419061,"caller":"config/config.go:173","msg":"Elasticsearch detected","version":7}
{"level":"info","ts":1607410996.587746,"caller":"static/strategy_store.go:145","msg":"No sampling strategies provided, using defaults"}
{"level":"info","ts":1607410996.6066759,"caller":"server/grpc.go:76","msg":"Starting jaeger-collector gRPC server","grpc.host-port":":14250"}
{"level":"info","ts":1607410996.6067162,"caller":"server/http.go:44","msg":"Starting jaeger-collector HTTP server","http host-port":":14268"}
{"level":"info","ts":1607410996.6068382,"caller":"server/zipkin.go:48","msg":"Listening for Zipkin HTTP traffic","zipkin host-port":":9411"}
{"level":"info","ts":1607410996.6282601,"caller":"grpc/builder.go:66","msg":"Agent requested insecure grpc connection to collector(s)"}
{"level":"info","ts":1607410996.6283166,"caller":"[email protected]/clientconn.go:106","msg":"parsed scheme: \"\"","system":"grpc","grpc_log":true}
{"level":"info","ts":1607410996.628339,"caller":"[email protected]/clientconn.go:106","msg":"scheme \"\" not registered, fallback to default scheme","system":"grpc","grpc_log":true}
{"level":"info","ts":1607410996.6283584,"caller":"passthrough/passthrough.go:48","msg":"ccResolverWrapper: sending update to cc: {[{:14250 0 }] }","system":"grpc","grpc_log":true}
{"level":"info","ts":1607410996.6283708,"caller":"[email protected]/clientconn.go:948","msg":"ClientConn switching balancer to \"round_robin\"","system":"grpc","grpc_log":true}
{"level":"info","ts":1607410996.6290116,"caller":"command-line-arguments/main.go:208","msg":"Starting agent"}
{"level":"info","ts":1607410996.6292071,"caller":"app/agent.go:69","msg":"Starting jaeger-agent HTTP server","http-port":5778}
{"level":"warn","ts":1607410996.6294172,"caller":"[email protected]/clientconn.go:1223","msg":"grpc: addrConn.createTransport failed to connect to {:14250 0 }. Err :connection error: desc = \"transport:
Error while dialing reading server HTTP response: unexpected EOF\". Reconnecting...","system":"grpc","grpc_log":true}
{"level":"info","ts":1607410996.6297212,"caller":"app/server.go:109","msg":"Query server started","port":16686}
{"level":"info","ts":1607410996.6299067,"caller":"healthcheck/handler.go:128","msg":"Health Check state change","status":"ready"}
{"level":"info","ts":1607410996.6299548,"caller":"app/server.go:146","msg":"Starting CMUX server","port":16686}
{"level":"info","ts":1607410996.6299684,"caller":"app/server.go:123","msg":"Starting HTTP server","port":16686}
{"level":"info","ts":1607410996.6299803,"caller":"app/server.go:136","msg":"Starting GRPC server","port":16686}
{"level":"error","ts":1607410997.4598336,"caller":"grpc/reporter.go:74","msg":"Could not send spans over gRPC","error":"rpc error: code = Unavailable desc = last connection error: connection error: desc = \"tra
nsport: Error while dialing reading server HTTP response: unexpected EOF\"","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/reporter/grpc.(Reporter).send\n\tgithub.com/jaegertracing/jaeger/cmd/agen
t/app/reporter/grpc/reporter.go:74\ngithub.com/jaegertracing/jaeger/cmd/agent/app/reporter/grpc.(Reporter).EmitBatch\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/reporter/grpc/reporter.go:53\ngithub.com/ja
egertracing/jaeger/cmd/agent/app/reporter.(MetricsReporter).EmitBatch\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/reporter/metrics.go:83\ngithub.com/jaegertracing/jaeger/cmd/agent/app/reporter.(ClientMet
ricsReporter).EmitBatch\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/reporter/client_metrics.go:120\ngithub.com/jaegertracing/jaeger/thrift-gen/jaeger.(agentProcessorEmitBatch).Process\n\tgithub.com/jaeger
tracing/jaeger/thrift-gen/jaeger/agent.go:138\ngithub.com/jaegertracing/jaeger/thrift-gen/jaeger.(AgentProcessor).Process\n\tgithub.com/jaegertracing/jaeger/thrift-gen/jaeger/agent.go:112\ngithub.com/jaegertra
cing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:121\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processor
s.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:86"}
{"level":"info","ts":1607410997.6296287,"caller":"base/balancer.go:196","msg":"roundrobinPicker: newPicker called with info: {map[]}","system":"grpc","grpc_log":true}
{"level":"warn","ts":1607410997.6301935,"caller":"[email protected]/clientconn.go:1223","msg":"grpc: addrConn.createTransport failed to connect to {:14250 0 }. Err :connection error: desc = \"transport:
Error while dialing reading server HTTP response: unexpected EOF\". Reconnecting...","system":"grpc","grpc_log":true}
{"level":"info","ts":1607410999.2636833,"caller":"base/balancer.go:196","msg":"roundrobinPicker: newPicker called with info: {map[]}","system":"grpc","grpc_log":true}
{"level":"warn","ts":1607410999.2644143,"caller":"[email protected]/clientconn.go:1223","msg":"grpc: addrConn.createTransport failed to connect to {:14250 0 }. Err :connection error: desc = \"transport:
Error while dialing reading server HTTP response: unexpected EOF\". Reconnecting...","system":"grpc","grpc_log":true}

[albertteoh]

@zhufeizzz can you please provide the command you used to start jaeger all-in-one?

[zhufeizzz]

@zhufeizzz can you please provide the command you used to start jaeger all-in-one?

@albertteoh

export SPAN_STORAGE_TYPE=elasticsearch
export COLLECTOR_ZIPKIN_HOST_PORT=9411
export ES_SERVER_URLS=http://xxx:10200

nohup ./jaeger-all-in-one >>${_JAEGER_RUN_LOG} 2>&1 &

[albertteoh]

@zhufeizzz I can't reproduce it locally:

$ git checkout tags/v1.18.0
$ SPAN_STORAGE_TYPE=elasticsearch COLLECTOR_ZIPKIN_HOST_PORT=9411 ES_SERVER_URLS=http://localhost:9200 ./all-in-one-darwin-amd64
2020/12/09 23:17:21 maxprocs: Leaving GOMAXPROCS=16: CPU quota undefined
{"level":"info","ts":1607516241.633766,"caller":"flags/service.go:116","msg":"Mounting metrics handler on admin server","route":"/metrics"}
{"level":"info","ts":1607516241.634084,"caller":"flags/admin.go:120","msg":"Mounting health check on admin server","route":"/"}
{"level":"info","ts":1607516241.634141,"caller":"flags/admin.go:126","msg":"Starting admin HTTP server","http-addr":":14269"}
{"level":"info","ts":1607516241.6341739,"caller":"flags/admin.go:112","msg":"Admin server started","http.host-port":"[::]:14269","health-status":"unavailable"}
{"level":"info","ts":1607516241.6495519,"caller":"config/config.go:173","msg":"Elasticsearch detected","version":7}
{"level":"info","ts":1607516241.6715221,"caller":"static/strategy_store.go:145","msg":"No sampling strategies provided, using defaults"}
{"level":"info","ts":1607516241.680831,"caller":"server/grpc.go:76","msg":"Starting jaeger-collector gRPC server","grpc.host-port":":14250"}
{"level":"info","ts":1607516241.6808782,"caller":"server/http.go:44","msg":"Starting jaeger-collector HTTP server","http host-port":":14268"}
{"level":"info","ts":1607516241.6811142,"caller":"server/zipkin.go:48","msg":"Listening for Zipkin HTTP traffic","zipkin host-port":":9411"}
{"level":"info","ts":1607516241.695253,"caller":"grpc/builder.go:66","msg":"Agent requested insecure grpc connection to collector(s)"}
{"level":"info","ts":1607516241.695319,"caller":"[email protected]/clientconn.go:106","msg":"parsed scheme: \"\"","system":"grpc","grpc_log":true}
{"level":"info","ts":1607516241.695338,"caller":"[email protected]/clientconn.go:106","msg":"scheme \"\" not registered, fallback to default scheme","system":"grpc","grpc_log":true}
{"level":"info","ts":1607516241.695362,"caller":"passthrough/passthrough.go:48","msg":"ccResolverWrapper: sending update to cc: {[{:14250  <nil> 0 <nil>}] <nil> <nil>}","system":"grpc","grpc_log":true}
{"level":"info","ts":1607516241.695408,"caller":"[email protected]/clientconn.go:948","msg":"ClientConn switching balancer to \"round_robin\"","system":"grpc","grpc_log":true}
{"level":"info","ts":1607516241.69606,"caller":"base/balancer.go:196","msg":"roundrobinPicker: newPicker called with info: {map[0xc0004889b0:{{:14250  <nil> 0 <nil>}}]}","system":"grpc","grpc_log":true}
{"level":"info","ts":1607516241.696699,"caller":"all-in-one/main.go:208","msg":"Starting agent"}
{"level":"info","ts":1607516241.696857,"caller":"app/agent.go:69","msg":"Starting jaeger-agent HTTP server","http-port":5778}
{"level":"info","ts":1607516241.697461,"caller":"app/server.go:109","msg":"Query server started","port":16686}
{"level":"info","ts":1607516241.6974988,"caller":"healthcheck/handler.go:128","msg":"Health Check state change","status":"ready"}
{"level":"info","ts":1607516241.697515,"caller":"app/server.go:146","msg":"Starting CMUX server","port":16686}
{"level":"info","ts":1607516241.697509,"caller":"app/server.go:123","msg":"Starting HTTP server","port":16686}
{"level":"info","ts":1607516241.69754,"caller":"app/server.go:136","msg":"Starting GRPC server","port":16686}

A few questions:

• Can you reliably reproduce this error?
• Did you check whether if there was an existing process listening on port 14250 prior to starting all-in-one? The error message desc = "transport: Error while dialing reading server HTTP response: unexpected EOF suggests the response was incompatible with what it expected.

[Ashmita152]

Not sure, but I think you may need to whitelist port 14250 (and maybe other jaeger related ports) in your machine's firewall. Google says SUSE has YaST firewall.

[zhufeizzz]

• desc

@albertteoh You reminded me. I found the problem.
Because I set http_proxy in /etc/profile, and foget to set no_proxy to skip local http request.

Thanks!