Dd-trace-rb: Distributed tracing support through SQS using Shoryuken
The existing Shoryuken instrumentation does not support the distributed_tracing option for injecting and extracting context when sending and receiving messages from SQS.
I'm working on a prototype to have this manually instrumented by sending the conventional Datadog tracing headers as SQS Attributes and making the necessary changes in the Shoryuken workers as a middleware to inject into Datadog.
Do you know if there is anything planned around this feature? I can think about contributing a patch upstream but wanted to check if using SQS Attributes can be a general enough solution to work across different Datadog clients for different stacks. What do you think?
tjsousa
>All comments
I think connecting the traces together through SQS would be a great feature; there's a lot of value in this sort of thing.
Two thoughts:
We can make something like this happen end-to-end in Ruby no problem within the confines of this repo. Making it work for all clients and stacks would be even better (ideal), but will require some coordination on our end, as all the clients would need to consistently implement some SQS standard.
We might want to be mindful of how we connect these two ends of the traces. Distributed tracing creates a "parent-child" relationship between spans in traces, implying that the operations are part of an atomic "task"/"job"/"request." When applied to SQS, messages might sit in there for an indefinite period of time... if its processed on the other end hours later, you would see that huge gap appear within the trace, which might not be ideal. Question is, is this part of the same trace, or is it just a related trace? If the latter, this suggests we might want a different relationship/UX to link the the two halves, even if we use a similar mechanism or modification of distributed tracing.
I think this is an interesting proposal, and I'd like to take this to my team. If you have any prototype to share, especially a high level description of the mechanism by which tracing metadata is injected and extracted from SQS, then this is something that would be really useful in a pitch.
Either way, I'm supportive of this feature in general and would be very happy to see any contributions you'd like to share for it!
delner
on 17 Mar 2020
Related issues
SeanDunford
路
3Comments
mdross95
路
4Comments
KieranP
路
3Comments
truongnmt
路
3Comments
GeoffKarnov
路
3Comments
Most helpful comment
I think connecting the traces together through SQS would be a great feature; there's a lot of value in this sort of thing.
Two thoughts:
We can make something like this happen end-to-end in Ruby no problem within the confines of this repo. Making it work for all clients and stacks would be even better (ideal), but will require some coordination on our end, as all the clients would need to consistently implement some SQS standard.
We might want to be mindful of how we connect these two ends of the traces. Distributed tracing creates a "parent-child" relationship between spans in traces, implying that the operations are part of an atomic "task"/"job"/"request." When applied to SQS, messages might sit in there for an indefinite period of time... if its processed on the other end hours later, you would see that huge gap appear within the trace, which might not be ideal. Question is, is this part of the same trace, or is it just a related trace? If the latter, this suggests we might want a different relationship/UX to link the the two halves, even if we use a similar mechanism or modification of distributed tracing.
I think this is an interesting proposal, and I'd like to take this to my team. If you have any prototype to share, especially a high level description of the mechanism by which tracing metadata is injected and extracted from SQS, then this is something that would be really useful in a pitch.
Either way, I'm supportive of this feature in general and would be very happy to see any contributions you'd like to share for it!