Spring-cloud-sleuth: Improper order of

[czujnik81]

Sleuth v Finchley.M9
While processing request between Spring Cloud Gateway and final service Sleuth improperly ordered events. Please see attached screenshot.

To reproduce:
Use: https://github.com/czujnik81/api-gateway.git and https://github.com/czujnik81/final-service.git
Define ZIPKIN_URL env variable to point working ZIPKIN server (eg. http://localhost:9411) if api-gateway and final-service are running on different servers for gateway define also HOST env variable eg. localhost:9411. To run service url to eg. http://localhost:8080/contract/abc/file
Also notice improper client address on screenshot

[codefromthecrypt]

I looked at this and what you are seeing is likely explained by where measurements occur from. So, it isn't out-of-order, rather this is literally when the frameworks thought they were done.

On the client side, it is low-level netty bubbling up through reactor. The server is being measured from the servlet api. This is why you see the client think it is done a bit before the server did.

My guess is this isn't going to happen as much in practice as usually you will have different machines (which can likely wash out the 100us you are seeing here from client server on the same host).

While there are probably some things to slightly adjust in the servlet side, I wanted to give you an early answer that this isn't going to be handled with priority. If we get a lot of reports, we might, but right now it is explainable.

Hope this helps

[czujnik81]

I understand the reason, but the thing is that in the same runtime environment sometimes it's look like mentioned and sometimes it's properly ordered.
It's not high priority, but it's kind of deficiency.

[codefromthecrypt]

anyway I will have a look, just wanted to warn you it might not resolve
today :P

[codefromthecrypt]

PS thanks for the report. you did a good job

[codefromthecrypt]

so the idea isn't lost.. one way to handle this it to move the server span lower on the stack, ex to netty for example. There would be some gotchas to handle (ex how to share http.route with netty), but it is possible (especially as brave has a netty filter already)

[codefromthecrypt]

even if we don't recommend something like this routinely, it could make sense for gateways. cc @jkschneider

[guanyalei]

I ran into similar issue in testing environment with Finchley.BUILD-SNAPSHOT. Client side requests REST service, which is deployed on the same machine with client, by RestTemplate.getForObject(). And I see CR happens before SS.

However, sequence of SS and CR of feign client(@FeignClient) looks correct in the same scenario.

[marcingrzejszczak]

Any chance for a sample? Also does is always end like this for every request?

[guanyalei]

Yes, always the same for every request. Attached service-ribbon and service-hi project in my previous screenshot for your reference. Zipkin server jar file is the latest zipkin-server-2.8.3-exec.jar.
spring-cloud-client.zip
spring-cloud-service-ribbon.zip

[marcingrzejszczak]

Works on my machine ;)

[marcingrzejszczak]

I just unpacked and ran your examples. Should I do sth else to replicate the issue?

[guanyalei]

It is an extremely simple demo to help me learn spring cloud basic. I don't have any special configuration for eureka server and zipkin server.

[guanyalei]

My host OS is a windows 10 pro, and I launched these demo services from eclipse Mars version. Does that make difference with your test?

[marcingrzejszczak]

I have a mac and I've ran the apps from Intellij. I don't really know how to help you with this one since I can't replicate it :| Maybe if someone else has the same problem it will be easier to reproduce it.

[guanyalei]

Thanks all the same for your help. Maybe I will figure it out myself after deeper understanding of spring cloud :)

[marcingrzejszczak]

Is this still a problem?

[czujnik81]

Didn’t noticed

Pozdrawiam,
Paweł Janusz
[email protected]
pawel.[email protected]
https://remote.com/paweljanusz

On 16 Jan 2020, at 09:12, Marcin Grzejszczak [e-mail] wrote:

Is this still a problem?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/spring-cloud/spring-cloud-sleuth/issues/913?email_source=notifications&email_token=ADKMYL4PYJL76JWLJT3W7YDQ6AJGFA5CNFSM4EXBFQGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJDFHMY#issuecomment-575034291, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADKMYL7XZFS3YGLCO3QK7HLQ6AJGFANCNFSM4EXBFQGA.

[marcingrzejszczak]

Ok then, let's close it. We'll reopen it if someone confirms that the issue is back.