Apm: Add transaction.context.response.body intake API field

[felixbarny]

Description of the issue

Currently, our agents offer the capture_body configuration option which allows to capture the request body in transaction.context.request.body. I got multiple requests for being able to also record the response body:

• https://github.com/elastic/apm-agent-java/issues/496
• https://github.com/elastic/apm-agent-java/issues/449#issuecomment-457572117 (this user requested the feature offline)

We should also add another configuration option (capture_response_body) for this so that users can capture request bodies but not response bodies, for example. I think the capture_content_types option can work for both.

Open questions

• Is this even possible to implement for all agents?
• If it's not, is it ok that only some of our agents support that? (I think it is)

Vote

@elastic/apm-agent-devs please vote if you are in favor of adding transaction.context.response.body to the intake API.

• [x] .NET
• [x] Go
• [x] Java
• [ ] Node.js
• [ ] Python
• [ ] Ruby
• [ ] RUM

[beniwohli]

@felixbarny correct me if I'm wrong, but https://github.com/elastic/apm-agent-java/issues/496 seems to request capturing the response of http spans, not transaction response body.

Either way, I am somewhat concerned of scope creep. I don't really see how this feature could be considered as related to APM.

[mikker]

When we get tighter log/apm integration we'd also be doubling the same info. And it's one more place we need to worry about capturing sensitive info

[felixbarny]

Either way, I am somewhat concerned of scope creep. I don't really see how this feature could be considered as related to APM.

I see what you mean but why do we support capturing the request body then?

[mikker]

Request bodies are more likely than response bodies to be the culprit for errors or needed to reproduce weird perf problems?

[beniwohli]

Either way, I am somewhat concerned of scope creep. I don't really see how this feature could be considered as related to APM.

I see what you mean but why do we support capturing the request body then?

Past scope creep doesn't justify future scope creep ;)

[Qard]

What's the actual use-case for capturing the response body? It seems complicated to build and I'm not sure if it's worth the time and effort. 🤔

[watson]

Request bodies are more likely than response bodies to be the culprit for errors or needed to reproduce weird perf problems?

To my knowledge this is why we added it to requests. I've used this several times in production apps to debug an issue that caused the server to misbehave because it got a request body I didn't expect. So having the ability to capture the request body was really convenient.

[felixbarny]

Thanks, all good points. Another one is that requests are usually much smaller than responses. For example, query criteria vs full result list or HTML file.

@felixbarny correct me if I'm wrong, but elastic/apm-agent-java#496 seems to request capturing the response of http spans, not transaction response body.

You are right, I misinterpreted that.

[felixbarny]

@skynet2 @omrishtam @nowachri-de @lsntomaszw

Since you have upvoted this issue, what is your use case for recording response bodies. Or current thinking is that it's out of scope for an APM solution. However, our agents should offer a possibility for you to manually attach custom data to transactions and spans so that you could record the response on your own. Would that work for the use cases you have in mind? Another way of approaching this would be to implement a proxy which records requests and responses and indexes them into Elasticsearch.

[omrishtam]

@felixbarny
In my case, I structure my API calls logs with the request's AND the response's body.
I Use APM for its detailed information of each API call (duration, span, user, host, etc...), it logs all of my API calls. In addition, I'm indexing theses calls to another index on my own to the same elasticsearch cluster to which APM is indexing to.

My main issue with this is that if I look for a specific call which gave an error message in the response's body, or responded with a reportedly unexpected response, then I have to look for it at my own indexed logs which do not include as much details as the APM log for the same call, and I prefer only looking at APM's logs.
This causes me to index these logs outside of APM basically just for checking abnormal responses for my API requests, and it's very difficult to find the corresponding request log of APM to see the more detailed information about this abnormal response.

It would make it very much easier for me to only use APM for these logs by not needing to deal with two sources of logs for the same thing and having all the information that I need in case there's a specific error that I need to check for only in APM.

[skynet2]

Hi @felixbarny,
My use case is pretty similar to @omrishtam , we have RPC API over AMQP protocol, and there are some critical requests like payment\tracking info, where we want to log request + response.

Here is the way which I used to implement it in the current schema.
Let's pretend that we have a json request; I have a list of "tags" which are important for me, for example, "transaction_id," when a new request arrives at API, I'm trying to parse JSON and look for "transaction_id" property. If a code can find it, I'll write a new tag to apm transaction with key "transaction_id" and value = <value>. It gives the ability to search for requests which are related to the specific transaction. And I want to see request which API received (it's already implemented Request.Body) and i also want to see the response from my server because (especially if external provider says that it`s not correct\valid) it helps to analyze information and application behaviour, especially when you have many microservices.
Currently, I'm storing a response in Custom section, but there is the ticket in which there is a proposal to remove custom (https://github.com/elastic/apm/issues/7), so we'll need to find some place where we can store that info.

I agree with that fact that the basic implementation of APM agent should not capture the response body, but it'll be great to have a possibility to do that manually.

[felixbarny]

I agree with that fact that the basic implementation of APM agent should not capture the response body, but it`ll be great to have a possibility to do that manually.

If we decide to deprecate and eventually remove custom context, we'll make sure there is a new home for that.

Thanks for your feedback!

[felixbarny]

Closing as this seems to be out-of-scope for APM and because alternatives exist (custom context/tags).