Apm: [Proposal]Add new fields in context.response for both transaction and spans

[vigneshshanmugam]

Description

With the browser API's like Navigation and Resource Timing which has been advanced to level 2 and already available in more than 1 browser platform, we are planning to add support for new fields like transferSize, encodedBodySize, decodedBodySize and serverTiming in the APM UI as part of both page load transaction and spans.

• More details about the field information can be found here

Prior Reading - TLDR version

Information that we can gather from these new fields and how we can transform it on the APM UI

• The UI can show the payload size of HTML (page load transaction) and other Resource spans (JavaScript, HTML, CSS, Images etc.)
• UI can show information about the encoded/decoded size(Transaction and Resource Spans). Super useful for debugging performance issues.
• Server timings if the resources response has that information. Some of the CDN
  providers ex- Akamai already supply this information.

There are also other possibilities like whether the resource is cached, prefetched, preloaded, served via service workers etc. I will create a separate proposal for this since its too specific to RUM agent.

Schema changes

Since this is part of the http response, I am proposing to add it as part of context.http.response object for Spans & Transactions(Might be applicable only to RUM agent).

// All of the proposed fields are optional
// (number - <bytes/octets>)
{
  "http": {
    "response": {
      "transfer_size": ["number", "null"],
      "encoded_body_size": ["number", "null"],
      "decoded_body_size": ["number", "null"],
      "headers": {
          "server-timing": ["string", "Array<string>", "null"]
       }
    }
  }
}

UI Gotchas

• This is really specific to Browser API and the UI has to make a distinction based on value being unavailable vs zero. 0 value in the above fields(transfer and compressed) indicates the response is obtained from Browsers HTTP/ServiceWorker/In memory cache
• UI have to take care of the server timing fields having a large string

Open questions

• Will this be applicable to other agents as well since there is no external resource requests.
• Can we use the response headers (Content-Length) in other agents as a value for transfer_size
  
  
  
  • We can match it with Content-Type and show appropriate transfer_size or compressed_size in UI
  
  

Vote

@elastic/apm-agent-devs please vote on the proposals if you are in favor of adding to the intake API or make a suggestion if you prefer something else

• [x] .NET
• [x] Go
• [x] Java
• [x] Node.js
• [x] Python
• [x] Ruby
• [x] RUM

Update: Fields names are adjusted with the latest comments from Sergey and Andrew.

[jahtalab]

@vigneshshanmugam , thanks for the proposal!

I suggest making a distinction between the transaction context and span context since they are defined differently. I think these field only apply to the span context since currently the resource timings are captured as spans. I'm not sure if we should add them to the transaction context.

Moreover, the response field in the transaction context refers to the response generated by the server (which the backend agents instrument), as opposed to the response received by the browser. IMO, they should be considered different.

Also, under the span context there's http field which could store the proposed fields.

[vigneshshanmugam]

I suggest making a distinction between the transaction context and span context since they are defined differently. I think these field only apply to the span context since currently the resource timings are captured as spans. I'm not sure if we should add them to the transaction context.

All of the fields apply not only to resource timings but as well as to Navigation timing which we capture as Page Load Transaction at least for RUM agent.

Moreover, the response field in the transaction context refers to the response generated by the server (which the backend agents instrument), as opposed to the response received by the browser. IMO, they should be considered different.

Hmm good point. I am not sure if such a distinction in the field level would provide any usefulness since we clearly distinguish between external resource and API calls using type field (ex - resource.script vs ext.http). I am also fine with adding them as part of context.http

[watson]

We should allow for the server_timing value to be an array in case there's more than one Server-Timing header in the response. It will be exposed in the Performance API like so:

[vigneshshanmugam]

@watson Good point. I was thinking to send them as string like how the ServerTiming header would exist on response header instead of how they are available through Resource timing API.

const server_timing = "Cache read;cache=200,app=500"

[watson]

Ah yes, that would probably work as well. We just need to make sure that the way we want to query for this is supported by how we store it (array vs. string)

[simitt]

Summarizing some offline discussion with @vigneshshanmugam:

• Values should be defined for transactions and spans on Intake API. I suggest to use transaction.context.response and span.context.http.response for it.
• None of the new fields needs to be indexed.
• Length limit 1024 for server_timing should not be defined on the Intake API as all agents then would need to cut off longer values with the same logic. Instead we should consider trimming the value to a certain length limit in the server, to reduce storage costs. (We could also check this for existing fields where large values won't necessarily give more value to the user).

In alignment with current values I suggest to map the values as following:
Transactions:
context.response.transfer_size-> http.response.transfer_size
context.response.compressed_size-> http.response.compressed_size
context.response.server_timing-> http.response.server_timing
Spans:
context.http.response.transfer_size-> span.http.response.transfer_size
context.http.response.compressed_size-> span.http.response.compressed_size
context.http.response.server_timing-> span.http.response.server_timing

[SergeyKleyman]

@vigneshshanmugam Thank you for the proposal. A couple of questions/proposals

1. Is the meaning of transfer_size the same as transferSize described at https://developer.mozilla.org/en-US/docs/Web/API/PerformanceResourceTiming/transferSize ?

... the size includes the response header fields plus the response payload body

If so then Content-Length header would not be the right header to use for transfer_size since Content-Length is size of body only.

1. Is the meaning of compressed_size the same as encodedBodySize described at https://developer.mozilla.org/en-US/docs/Web/API/PerformanceResourceTiming/encodedBodySize ?

... represents the size (in octets) received from the fetch (HTTP or cache), of the payload body, before removing any applied content-codings

If so then I would propose to use encoded_body_size instead because encoded_body_size better conveys that (a) it's only the size of the body in the payload and (b) an encoding doesn't have to be compression.
It seems that Content-Length response header and encodedBodySize should be the same although we need consider cases when Content-Length response header is not present (for example when Chunked transfer encoding is used).

[SergeyKleyman]

,NET team (@gregkalapos and I) discussed it and we are not sure if and how it's applicable to backend agents. @elastic/apm-agent-devs do you have use cases in mind when a backend agent will make of use of this proposal?

[jahtalab]

I agree with the concerns @SergeyKleyman pointed out. Generally I think if backend agents don't have a use-case for these fields or they calculate these fields differently, then we should consider RUM specific fields to store this data. One option for the transaction is to store them in context.page which is only used by the RUM agent.

[axw]

I don't intend to implement this in the Go agent any time soon. I think for outgoing spans at least, these would make sense for any agent, RUM or backend; I think having the span properties under span.http.response would be fine.

Regarding Server-Timing: I wonder if we shouldn't be parsing this? It could be useful in aggregations, or for the UI to display details for non-instrumented services.

[vigneshshanmugam]

Regarding Server-Timing: I wonder if we shouldn't be parsing this? It could be useful in aggregations, or for the UI to display details for non-instrumented services.

Definitely, The header can be parsed by the APM UI since the format would be array representing the values of the form <descriotion>;<name>=<duration>, .... I did not want to touch it since its per the spec. Happy to hear other proposals as well.

How do we want to move forward with this issue? Would like to finalize the data structure and the server spec before we merge it on the RUM side.

[axw]

@vigneshshanmugam there's some unanswered questions from Sergey above. How about the following:

• [span.]http.response.transfer_size
• [span.]http.response.encoded_body_size
• [span.]http.response.decoded_body_size

If we're not going to decode Server-Timing, I don't see any value in having a special field for it. Instead, I'd suggest introducing room for raw headers:

• span.http.response.headers

which would be an object type, under which you would have:

• span.http.response.headers.server-timing

Like headers recorded for transactions, the value could be a string or an array of strings. We could later parse the header and record it in a specific field, for aggregations and such. That could be done by the server, or perhaps an ingest node processor.

WDYT?

[vigneshshanmugam]

@axw @SergeyKleyman Sorry, Might have missed those above comments, I agree with you on the proposed values under [span].http.response and yes encoded_body_size better conveys the size than the transfer_size one as Sergey pointed out.

Sure for now we can have it raw headers unless we decide to use the value for any chart representation.

[axw]

@vigneshshanmugam Cool. Can you please update the proposal description, then we can get folks to have a second look.

[vigneshshanmugam]

@axw Sure, Updated the schema as per our discussion. @elastic/apm-agent-devs Can you please have a second look at the proposal?

[watson]

I don’t recommend to change - into _ in headers. If possible it would be best if we could just leave the headers as is. Though if we want to query for them specifically we might want to lowercase them first?

Also, I assume this headers object support arrays like our other headers do?

[vigneshshanmugam]

@watson Good point, I have updated the description to use '-' and at least for the consistency bit we have to lower them before sending to APM server. I guess most of the platforms(Node.js) does lower case HTTP headers by default?

Also, I assume this headers object support arrays like our other headers do?

Yes https://www.w3.org/TR/server-timing/#server-timing-header-field

[watson]

I guess most of the platforms(Node.js) does lower case HTTP headers by default?

For incoming headers to the HTTP server yes - so that you can easily query for them. But not for outgoing headers. There they are sent verbatim.

[watson]

We should probably update the spec in the description to also allow arrays of strings right? Not sure about the syntax - maybe "server-timing": ["string", "null", "Array<string>"]?

[vigneshshanmugam]

For incoming headers to the HTTP server yes - so that you can easily query for them. But not for outgoing headers. There they are sent verbatim.

That makes sense. Thanks for clarifying.

Updated the spec in description. I believe the format looks "server-timing": ["string", "array", "null"]
Ex - https://github.com/elastic/apm-server/blob/4fdbefc42f8b6b0a52e66d4bf85ddee463fe1ba3/docs/spec/context.json#L22-L33

[watson]

Just nitpicking - and not important - as we all know what we're talking about here, but if we wanted to make this into a proper spec we'd also need to specify which type the elements of the array could be. The example in the description looks like it's a pseudo-schema and not fully JSON Schema compliant, in which case Array<string> would do the trick instead of just array.

[simitt]

Reading up on the discussions, we would have following structure:

{ 
  "transaction": {
    "context": {
      "response": {
        "transfer_size": ["number", "null"],
        "encoded_body_size": ["number", "null"],
        "decoded_body_size": ["number", "null"],
        "headers": {
          "patternProperties": { //this is a generic definition for the headers we already have
            "[.*]*$": {
              "type": ["string", "array", "null"],
              "items": {
                "type": ["string"]
              }
            }
          }    
        }
      }
    }
  }
}

and

{ 
  "span": {
    "context": {
      "http": {
        "status_code": ["integer", "null"] // existing, would be deprecated
        "response": {
          "status_code": ["integer", "null"] // move here for alignment
          "transfer_size": ["number", "null"],
          "encoded_body_size": ["number", "null"],
          "decoded_body_size": ["number", "null"],
          "headers": {
            "patternProperties": { //this is a generic definition aligned with the one we use for transactions
              "[.*]*$": {
                "type": ["string", "array", "null"],
                "items": {
                  "type": ["string"]
                }
              }
            }
          }    
        }
      }
    }
  }
}

One thing to consider is that on the Intake API span.context.http currently doesn't have a response key, but it contains url, status_code and method. Introducing response I suggest to also move status_code to span.context.http.response for alignment, but still also support it at the current position to be backwards compatible.

On the ES schema side the fields would be mapped to http.response.* for transactions and to span.http.response.* for spans.

@vigneshshanmugam and @watson side note regarding the format of the headers: the server doesn't really care about upper-/lowercase as we use go's standard http header package to create canonicalized headers.

@vigneshshanmugam is the statement that none of the newly introduced fields needs to be indexed still valid?
We also talked about cutting off the server_timing at 1024 chars some point. The server currently doesn't do that anywhere, and especially everything sent within any response.headers is only canonicalized but otherwise unchanged to reflect the original headers. But it wouldn't be a big effort to implement this if you think it makes sense.

[vigneshshanmugam]

Had an offline discussion with @simitt

• All of the proposed fields wont be indexed until we have a use case.
• server_timing header is not limited to 1024 bytes for now. We might remove this restriction in future releases.

[jahtalab]

Just FYI, a related bug could happen in APM UI if any agents don't send the http.request but send http.response so far this has been an issue for RUM though.

[jahtalab]

2020/02/25 agents meeting:
We have implemented this in the agents that require this (currently only RUM) other agents can implement as they need, so closing this one.