Apm: Support for chained exceptions

[beniwohli]

Python 3 introduced the concept of chained exceptions. Its purpose is to handle cases like this:

try:
    something_that_breaks()
except BreakingException:
    do_something_else_that_breaks()  # raises KaboomException

Without chained exceptions, only the KaboomException is recorded, the exception information of the BreakingException is lost. Python 3 stores the original exception in the __context__ attribute of the exception value.

Python 3 also introduced an explicit way to raise a chained exception:

try:
    something_that_breaks()
except BreakingException as exc:
    raise RetryException('Retry later') from exc

In this case, the BreakingException is stored as __cause__.

In the Intake API, this could be as simple as turning the exception dictionary into a list of dictionaries. It gets a bit more difficult to put it into a sensible ES schema. One option could be to store the first exception as is, and add the additional exceptions as blob. The chained exceptions couldn't be queried upon, but they could be shown in the UI.

Chained exceptions aren't a feature exclusive to Python. Java has them, and there are libraries for Node.js, e.g. verror from Joyent.

[roncohen]

Good idea @beniwohli. It seems like something we should look at again after GA. I'll be sure to include it in our planning

[watson]

@roncohen this would be a breaking change in the schema right? If we know that we want do this after GA, it might make sense to make the exception object into an array already now and just limit it to 1 element which means to don't have to support it yet. That would mean that when we want to support it, the schema doesn't change

[watson]

But maybe I'm making schema changes sound more complicated than they are?

[ruflin]

Can the analogy be made here that this is similar to that a transaction can have multiple traces and an error can have multiple exceptions?

If we think we will going to support this anyways (and it sounds like it) +1 on the idea from watson.

[simitt]

+1 on thinking about this more closely and adding basic support before GA.
We introduced having a log and an exception which afaik should reflect an original exception and a custom exception created on top of it. Would this distinction still be necessary and valuabel or could the log object also be modeled as an exception? Maybe we could even simplify our current handling with having an array of exceptions.

[beniwohli]

@simitt I don't think the log should be seen as custom exception. A log is just a message, and an optional stacktrace that indicates where that log message has been issued. In general, we have to differentiate the concept of a stacktrace to that of a full exception. A stacktrace can be generated from anywhere (that's also why we have stacktraces in traces), an exception has to be raised.

---

The change in the JSON Schema for this would be fairly small, and even backwards compatible. We could say that error.exception is either an exception object, or a list of exception objects.

It becomes more difficult in the ES schema. One option could be to turn exceptions into nested types, but I think we've been told to stay away from that. Another option could be to store the the exception list as blob, and pull out the type, module, message etc. from the top-most exception in the list.

Whatever we decide, I don't think the ES schema change could be done in a backwards compatible way, that's why I raised this now, while we still have the chance to change the schema without too much of a hassle. (I should have raised it earlier when we were designing the schema, but my exposure to Python 3 is still pretty limited, so I wasn't super aware of chained exceptions)

[roncohen]

I see your point regarding backwards compatibility.

We could add an optional exception.cause field in the schema. cause would have the same fields as exception, so it would be a recursively defined field. The fields under cause would be the same fields as under the top level exception key, and the objects under exception.cause would also be able to contain a cause: exception.cause.cause. Compared to having a list, this is closer to how Python and Java models the exception chain. Python uses __context__ as @beniwohli described. Java has a getCause() method.

{
  "exception": {
    "message": "Couldn't send exception to Opbeat",
    "stacktrace": [],
    "cause": {
        "message": "Couldn't find thing in database",
        "stacktrace": [],
        "cause": {
          "message": "Thing wasn't available in cache",
          "stacktrace": [],
        }
    }
}

Adding optional fields to the JSON Schema is something we can do in a backwards compatible fashion.

The ES model is another issue. I don't have a good idea to solving that at the moment - but I also don't think we have to solve that right now.

[alvarolobato]

While in Java there are chained exception, if I understood correctly the Python feature, that is more like a [Java Suppressed](https://docs.oracle.com/javase/7/docs/api/java/lang/Throwable.html#addSuppressed(java.lang.Throwable) exception. The main difference is while in the chained exception there is a cause relation, there isn't in the suppressed exception. When you catch an exception you can throw another exception as a cause of the catched one and add the former as a cause, chaining them. The suppressed exception case is when the code that is trying to handle the first exception fails for some reason, not caused by the original exception, and throws a new one.

In think this is an important feature and we should support it. In Java chained exception are basic to be able to know what really happened. Imagine an error writing to disk and that exception gets captured and a business exception is thrown, if you don't have the cause you won't be able to know that the problem was an IOException writing to disk.

Something very similar happens with suppressed exception, an error in the code handling an exception can hide the error that the application had and you would have to fix the exception handling code before being able to fix the application error.

[axw]

@alvarolobato @makwarth @roncohen
Would it be possible to add this to 7.0?

In Go, errors/exceptions are often nested. For example, any HTTP client error will return net/url.Error. Inside that error will typically be a net.OpError, and inside that will be the root cause, e.g. net.DNSError.

Today, we'll just set the exception type to "Error", which isn't terribly helpful as it means these errors with differing root causes will be folded together. A user of the Go agent brought this up in https://github.com/elastic/apm-agent-go/issues/423.

If we supported nested/chained exceptions then the grouping keys could take into account the whole chain, or at least the root cause.

[axw]

A slightly more concrete proposal. I'm mostly focusing on chained exceptions here, which are more generally applicable.

API

Per https://github.com/elastic/apm-dev/issues/15#issuecomment-327107453, introduce a "cause" field to error.exception, which can hold another exception object.

Grouping key

Update the APM Server’s error grouping key calculation to combine all exceptions in the chain. e.g. consider the following Go error:

&net/url.Error{
    Err: &net.OpError{
        Err: &net.DNSError{},
    },
}

The grouping key would combine "net/url.Error", "net.OpError", and then "net.DNSError". This will enable us to better group errors with different root causes when no message format is available; see https://github.com/elastic/apm-agent-go/issues/423.

Mapping

Option 1: error.exception as an array of objects

An exception chain would be represented as an array of exception objects, starting with the outermost exception, followed by its cause, and so on. Because of how arrays of objects are flattened, queries would match against all exceptions in the chain, and it would not be possible to query a specific exception independently of others in the chain.

I propose this option. We should be able to reconstruct the exceptions in the chain, but that may require using nulls for optional fields so we can gather up the values at the same index of each multi-value field.

Option 2: error.exception as array of nested

Like option 1, but using "nested" to enable querying of individual exceptions in the chain, independently of each other. Requires nested queries, and probably more expensive due to additional documents.

Option 3: error.exception.cause as dynamically mapped object

This would require dynamic mapping to be enabled, and would lead to fields like "exception.cause.cause.cause.cause..." for a chain.

Suppressed exceptions

Assuming we want to support this, how we do it depends on the UI design. Is it important to be able to see, given an exception, what exceptions it suppressed? Or is it more important to tell if a given exception was suppressed, and then be able to navigate to the suppressing exception?

Option 1: report suppressed exceptions independently, with "suppressed_by" relation

Option 2: add a "suppressed" field to error.exception, which holds an array of exceptions

[jalvz]

This is a breaking change if i understand correctly, so decision should be made promptly
cc @roncohen @alvarolobato

not the same, but related https://github.com/elastic/apm/issues/11

[axw]

I prototyped the change:

• https://github.com/elastic/apm-server/compare/6.x...axw:6.x-exception-cause?expand=1
• https://github.com/axw/apm-agent-go/pull/new/chained-errors

If we index the exception chain as an array of objects, then no mapping change is required. However, the APM UI would have to be updated to expect an array of objects, rather than just one object. To start with the UI could just ignore everything other than the first entry, but later we might want to show the full chain as a tree, list, whatever.

In the Go agent branch I have a recursive type definition for exception: exception.cause refers back to exception. In the server this didn't work, as the inline_schemas generator choked. So in the prototype I just set exception.cause's type to object.

If we want to do this in 7.0, the minimal change would be update the UI to expect error.exception to be _either_ an object or an array of objects, and in the latter cause just take the first object. If it's achievable, it would be better if, until we accept a chain, we index the solitary exception as an array of one object, just so the UI doesn't have to cater for multiple types. We would later update the server to accept "exception.cause", include the full chain in the grouping key calculation, and index the chain as an array of objects.

[roncohen]

thanks @axw!

If it's achievable, it would be better if, until we accept a chain, we index the solitary exception as an array of one object, just so the UI doesn't have to cater for multiple types.

this is something we'd do now in 7.0 to make sure the UI doesn't have to cater for multiple types in a way that allows us to do the rest later without breaking backwards compatibility later?

we're going to ask people to reindex everything for 7.0, so we could also change the existing data from a single object to an array.

Seems like a fairly minor thing both reindex wise and UI wise, so i'd be happy to try to see if we can get it done for 7.0

[axw]

this is something we'd do now in 7.0 to make sure the UI doesn't have to cater for multiple types in a way that allows us to do the rest later without breaking backwards compatibility later?

Yes, exactly.

[beniwohli]

Awesome stuff @axw, thanks for picking this up!

[sqren]

However, the APM UI would have to be updated to expect an array of objects, rather than just one object.

👍 I've created an issue to track: https://github.com/elastic/kibana/issues/28991

@elastic/apm-server Will this only affect the exception property on error objects? And if so since it is changed from a single object to a list of objects, should it be renamed to exceptions?

[jalvz]

yes, only error.exception. Recursive definitions are usually singular, so name is fine IMO.

[sqren]

yes, only error.exception. Recursive definitions are usually singular, so name is fine IMO.

I'm not very opinionated but I think if the data structure is an array, and could contain several exceptions, it should be plural.

A contrived example: With singular naming unwrapping needs an alias:

const { exception: exceptions } = error
const exception = exceptions[0]

If the naming is plural it reads more naturally:

const { exceptions } = error
const exception = exceptions[0]

If it's a hassle to rename then nvm.

[axw]

If they were independent exceptions I would agree exceptions would be more appropriate, but in this case they describe a single exception chain. Also not too strongly opinionated though.

[sqren]

they describe a single exception chain

Fair enough, singular it is :)
Just so I get my mental model correct, it's a single exception chain, that consist of one or more exceptions?

[axw]

Just so I get my mental model correct, it's a single exception chain, that consist of one or more exceptions?

Yes. Each exception is followed by its cause.

[axw]

The initial change to index error.exception as an array is merged into apm-server@master: https://github.com/elastic/apm-server/pull/1825. The intake and error grouping key changes will come later.

[sqren]

After thinking this more through I've noticed a potential problem: arrays are not really handled in ES, so I'm seeing some problems querying specific fields of an exception.

For the error group overview I fetch the exception message, and handled property for each error group in the list. I do this to avoid fetching the full stacktrace for each error group since this can be huge, and we don't need it for this view.

I'm not sure if it's still possible to get specific fields from an object inside an array in ES - still investigating but wanted to highlight this as early as possible.

@axw

[sqren]

Okay, it seems like I can still cherrypick the beforementioned fields like exception.message, and ES will return a list of exception messages (and not the stacktraces) - so that's good.
Currently only testing this on mock data but expecting it to work on real data as well.

[alvarolobato]

@axw @sqren where did we end up here?

[sqren]

@alvarolobato This was added to APM UI in https://github.com/elastic/kibana/pull/29345

[alvarolobato]

@sqren ok, I thought we had a basic implementation ie. support an array, but where only showing the first one. If it is complete I guess it's up to the agents to implement this feature.

@axw can you confirm we have everything in place and you can implement https://github.com/elastic/apm-agent-go/issues/423

Next step would be creating an issue for the agents where this is applicable.

[sqren]

@alvarolobato Yes, we only did the basic implementation. I've created an issue for adding support for displaying multiple chained exceptions: https://github.com/elastic/kibana/issues/40375

However, this shouldn't block agents from starting implementation of reporting chained exceptions.

[alvarolobato]

Great, thanks

>

[axw]

We still need to get changes into the server before I can implement elastic/apm-agent-go#423.

Proposal

API/storage changes

Add the following optional field to the Intake API:

• Event: error
• Field name: exception.cause
• Field type: exception (that makes exception a recursive type)

exception, exception.cause, exception.cause.cause etc. would be flattened in the Elasticsearch document. For example, if the Intake API event looks like this:

{
  "exception": {
    "type": "type_1",
    "message": "message_1",
    "cause": {
        "type": "type_2",
        "message": "message_2",
        "cause": {
          "type": "type_3",
          "message": "message_3",
        }
    }
}

then the Elasticsearch doc would look like:

{
  "exception": [
    {"type": "type_1", "message": "message_1"},
    {"type": "type_2", "message": "message_2"},
    {"type": "type_3", "message": "message_3"},
  ]
}

There are known limitations with arrays of objects. Notably, you may be surprised by the result of searching for "type: 'type_1' AND message: 'message_2'". The terms are individually matched against the whole array of objects. I think this is a reasonable trade off for performance/storage overhead, vs. say using nested docs.

Error grouping keys

The server's algorithm for calculating error grouping keys should be updated to hash the fields of the entire exception chain.

Backwards compatibility

• Because the changes were made in the UI first in 7.0, we can start storing as an array of objects without breaking the UI.
• Agents should wait for a major version bump before sending chained exceptions, otherwise error aggregation keys will change.

Vote

@elastic/apm-agent-devs @elastic/apm-server sound good?

Team | Yes | No | N/A | Link to implementation issue
| --------|:----:|:---:|:----:|:-------------------:|
| Server |

• [x]

|

• [ ]

|

• [ ]

| elastic/apm-server/issues/2389, https://github.com/elastic/hey-apm/issues/120
| .NET |

• [x]

|

• [ ]

|

• [ ]

| elastic/apm-agent-dotnet#358
| Go |

• [x]

|

• [ ]

|

• [ ]

| elastic/apm-agent-go/issues/423
| Java |

• [x]

|

• [ ]

|

• [ ]

| elastic/apm-agent-java/issues/705
| Node.js |

• [ ]

|

• [ ]

|

• [ ]

| elastic/apm-agent-nodejs#1196
| Python |

• [x]

|

• [ ]

|

• [ ]

| https://github.com/elastic/apm-agent-python/issues/526
| Ruby |

• [x]

|

• [ ]

|

• [ ]

| elastic/apm-agent-ruby#469
| RUM |

• [ ]

|

• [ ]

|

• [ ]

| elastic/apm-agent-rum-js#311

[mikker]

Fine with everything but I don't think Ruby has a way to know that an exception was chained.

[axw]

@mikker I think this is relevant: https://ruby-doc.org/core-2.6/Exception.html#method-i-cause

[mikker]

You are never wrong, @axw

class One < StandardError; end
class Two < StandardError; end
class Three < StandardError; end

def one
  raise One
end

def two
  one
rescue One
  raise Two
end

def three
  two
rescue Two
  raise Three
end

def main
  three
rescue Exception => e
  pp e
  pp e.cause
  pp e.cause.cause
  pp e.cause.cause.cause
end

main
# =>
    #<Three: Three>
    #<Two: Two>
    #<One: One>
    nil

[watson]

We don't have a way to chain in exceptions in Node.js, but I was thinking we might provide an API that allows the user to record such things. E.g. it's not uncommon in Node.js to say get a low-level error back from the database driver and then throw a more high-level error. Having access to both the low and the high-level error would be useful. I'll create an issue for looking into how this can be done.

@axw your example JSON doesn't show this, but I assume you intend it to include a stack trace as well for each chained exception?

[axw]

@watson sounds good! Actually in Go we don't have native support for it either, but there are some conventions for wrapping errors. I intend to automatically handle some conventions, and also provide an API for extracting the "cause" of an error.

your example JSON doesn't show this, but I assume you intend it to include a stack trace as well for each chained exception?

Yes, any exception field could in each item in the chain. For stack traces specifically, if the agent can determine that the inner exceptions' stacktraces are a superset of the outer ones', then the outer ones' could perhaps be omitted. One caveat: because current versions of the UI will show only the first exception, we need retain the stack trace for the outer-most exception.

[SergeyKleyman]

the outer exceptions' stacktraces are a superset of the inner ones', then the inner ones' could probably be omitted

@axw Do you mean the other way around? Because usually an inner exception is thrown from deeper in the call chain. In .NET stack trace for exception only includes frames that we actually unwound between throw and catch so there won't be any overlap.

[axw]

@SergeyKleyman yes indeed, I meant the other way around. I'll amend my comment. That reminds me of an issue with omitting stack traces though... if the UI only shows the first exception (i.e. up to and including at least stack version 7.2), then if we were to omit the stack trace from the outer-most exception we wouldn't see any stack trace at all.

[SergeyKleyman]

@axw Maybe we can make exception.cause an array of exception-s instead of just one? The motivation for this change is to support use case when multiple asynchronous operations are being waited for. For example in .NET it can be done using WhenAll. Since it's possible that more than one of those operation completes with an exception WhenAll aggregates them all into one exception using AggregateException which is essentially an array of exceptions.

[axw]

@SergeyKleyman maybe. How would that be stored in Elasticsearch?

[SergeyKleyman]

@axw A tree can be encoded as array. We can even make the encoding compatible with the current case which is linear, one child per each tree node except the leaf. We can write tree nodes in BFS order and a separate array for children count of each node. We can omit this second array for the linear case.

[axw]

@SergeyKleyman it was more a question of how rather than if. That would work.

Another option: we could add an optional "parent" field on each entry in the exception array, which would be the index of another exception in the array; if unspecified, the parent is the preceding element. We used to do something like that back when we stored spans as an array inside transaction docs.

In either case we retain the ability to search over the whole tree at once, so I'm happy with that.

@elastic/apm-server @elastic/apm-ui any concerns about supporting exception _trees_ rather than just chains?

[graphaelli]

That doesn't sound like an issue server side but won't speak for the UI side for this one.

A couple questions for clarification:

• The standard linear chained exception would just have one entry per cause?
• This would change the intake to accept something like this for the .NET WhenAll case ?

{
  "exception": {
    "type": "type_1", "message": "message_1",
    "cause": [
        { "type": "type_2", "message": "message_2", "cause": [{"type": "type_3", "message": "message_3"}] },
        { "type": "type_4", "message": "message_4" }
    ]
}

What would the es document look like in that case?

• Would an incremental update to the UI be possible with this? That is, could we support linear chains first (following the first cause) and then later add support for trees, while maintaining compatibility with this?

[alvarolobato]

@SergeyKleyman do applications usually show multiple causes in this situation?
Do you have an example of how it is printed, probably would help to understand the specific situation (at least to me)

[SergeyKleyman]

@alvarolobato I'll make a simple application to reproduce this use case and see how exception cause is printed.

[alvarolobato]

@SergeyKleyman not worth it, I thought you might have had an example at hand, don't worry.

[axw]

@graphaelli yes, I think we could do this incrementally.

The standard linear chained exception would just have one entry per cause?

Yes. Also, we could make cause accept either a single exception or array of exceptions.

This would change the intake to accept something like this for the .NET WhenAll case ?
{
"exception": {
"type": "type_1", "message": "message_1",
"cause": [
{ "type": "type_2", "message": "message_2", "cause": [{"type": "type_3", "message": "message_3"}] >},
{ "type": "type_4", "message": "message_4" }
]
}

Yes, assuming you had an AggregateException with two InnerExceptions: one of type type_2 (which in turn has a cause of type type_3) and one of type type_4.

What would the es document look like in that case?

Depends on which of the above mentioned approaches we take. I'm biased towards the one I suggested, using optional parent indices. No parent field means the preceding entry is the parent, otherwise it's the 0-based index of the parent entry:

{
    "exception": [
        {"type": "type_1", "message": "message_1"},
        {"type": "type_2", "message": "message_2"},
        {"type": "type_3", "message": "message_3"},
        {"type": "type_4", "message": "message_4", "parent": 0},
    ]
}

Would an incremental update to the UI be possible with this? That is, could we support linear chains first (following the first cause) and then later add support for trees, while maintaining compatibility with this?

A couple of options come to mind:

1. Start out having the server disregard anything past the element of each cause array, so that type_4 above wouldn't even be recorded in ES
2. Have the UI stop processing the exception array as soon as it finds an entry with the parent field set. That assumes we record the exceptions are recorded depth-first, and would effectively terminate after the first chain.

[SergeyKleyman]

@alvarolobato It's small effort and I was interested myself :)
The way AggregateException is printed by built-in "to-string" conversion is relatively readable:

System.AggregateException: One or more errors occurred. (Failed to init gRPC) (Failed to init DB) ---> Sandbox.Program+GrpcInitFailedException: Failed to init gRPC
   at Sandbox.Program.InitGrpcSubSystem() in C:\Elastic\Dev\dotNET_Agent\Sandbox\Sandbox\Program.cs:line 87
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Sandbox.Program.InitSystemImpl() in C:\Elastic\Dev\dotNET_Agent\Sandbox\Sandbox\Program.cs:line 75
   at Sandbox.Program.InitSystem() in C:\Elastic\Dev\dotNET_Agent\Sandbox\Sandbox\Program.cs:line 58
---> (Inner Exception #0) Sandbox.Program+GrpcInitFailedException: Failed to init gRPC
   at Sandbox.Program.InitGrpcSubSystem() in C:\Elastic\Dev\dotNET_Agent\Sandbox\Sandbox\Program.cs:line 87<---

---> (Inner Exception #1) Sandbox.Program+DbInitFailedException: Failed to init DB
   at Sandbox.Program.InitDbSubSystem() in C:\Elastic\Dev\dotNET_Agent\Sandbox\Sandbox\Program.cs:line 81<---

But if AggregateException is the cause of some other exception (in this case SystemInitFailedException)
the output is less readable:

Sandbox.Program+SystemInitFailedException: Failed to init system because of AggregateException: One or more errors occurred. (Failed to init gRPC) (Failed to init DB) ---> System.AggregateException: One or more
errors occurred. (Failed to init gRPC) (Failed to init DB) ---> Sandbox.Program+GrpcInitFailedException: Failed to init gRPC
   at Sandbox.Program.InitGrpcSubSystem() in C:\Elastic\Dev\dotNET_Agent\Sandbox\Sandbox\Program.cs:line 87
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Sandbox.Program.InitSystemImpl() in C:\Elastic\Dev\dotNET_Agent\Sandbox\Sandbox\Program.cs:line 75
   at Sandbox.Program.InitSystem() in C:\Elastic\Dev\dotNET_Agent\Sandbox\Sandbox\Program.cs:line 58
   --- End of inner exception stack trace ---
   at Sandbox.Program.InitSystem() in C:\Elastic\Dev\dotNET_Agent\Sandbox\Sandbox\Program.cs:line 65
   at Sandbox.Program.Main(String[] args) in C:\Elastic\Dev\dotNET_Agent\Sandbox\Sandbox\Program.cs:line 35

All the messages from the exceptions contained in AggregateException are included but only the stack trace from the first thrown exception is included.

Of course those are just results of the built-in "exception-to-string" conversion - all the pieces are available via APIs so each developer can implement any "exception-to-string" conversion she likes.

[simitt]

@sqren adressing your concerns (we discussed offline) I double checked, and from 7.0 on the exceptions are already stored in an array, although there will always only be one entry max, as @axw mentioned above.

[alvarolobato]

Thanks @SergeyKleyman very helpful.

[Qard]

Exception chaining is not really a thing in JavaScript, but we can provide an API to chain exceptions manually, if the user wishes to do so. 🤔

[alvarolobato]

I see this feature as an enabler for languages that really make use of chained exceptions ie. Java where they are really useful. I don't think that we need to force all agents to support it: if this isn't usual in your language you shouldn't implement it IMO.

[sqren]

Have the UI stop processing the exception array as soon as it finds an entry with the parent field set. That assumes we record the exceptions are recorded depth-first, and would effectively terminate after the first chain.

@axw As I understand your suggestion the UI should do this for 7.4 (while 7.0-7.3 still only reads the first item) and then perhaps in the future we can look into displaying more of the tree?

So we'll have three different ways the exception chain/tree is displayed across versions:
7.0-7.3: display the first exception in the array
7.4: display the first n exceptions in the array up until an exception with parent
7.4+: display tree of exceptions

[axw]

@sqren yep that's right, assuming we can't/don't want to add support for visualising trees straight in 7.4. We should definitely sync again on the details when we come to implementation, because the way we make this non-breaking and future proof is a bit subtle.

[sqren]

Good to hear. 7.4 is already pretty packed so this could slide into 7.5 - but that should be fine and not a blocker for implementing the new behaviour in agents.

[jalvz]

apm-server support is in done: https://github.com/elastic/apm-server/pull/2541
Notes:

cause is always an array, e.g.:

{
  "exception": {
    "type": "type_1", "message": "message_1",
    "cause": [
        { "type": "type_2", "message": "message_2", "cause": [{"type": "type_3"}] },
        { "type": "type_4", "message": "message_4"}
    ]
}

And it is stored as a flat array in ES, ordered as per depth-first traversal:

{
    "exception": [
        {"type": "type_1", "message": "message_1"},
        {"type": "type_2", "message": "message_2"},
        {"type": "type_3"},
        {"type": "type_4", "message": "message_4", "parent": 0},
    ]
}

parent is not indexed. Also @elastic/apm-ui remember you need to retrieve the exception array from _source.

As mentioned above, once agents start report exception chains, grouping errors will change so please keep @bmorelli25 in the loop for documentation when you get around it.

Let me know any questions / concerns!

Update: the parent field is generated by the apm-server

[beniwohli]

@elastic/apm-agent-devs did anybody already put some thought into whether this constitutes a breaking change and requires a major version bump? I'm mostly concerned about the fact that with this feature, the grouping key of exceptions will change if they have a cause (as pointed out by @jalvz), which might lead to a lot of new-but-not-really error groups in the UI.

It's not a classic breaking change, but I think it's worth discussing.

[axw]

I'm on the fence (what's new?)
Initially I thought it should be considered a breaking change, but now I'm not so sure.

It will change the grouping key, creating an additional error group - but is that a problem for users? If it is a problem, what will they gain by deferring the upgrade to a new major agent version?

At least in the Go agent, not reporting error causes is causing bugs:

• https://github.com/elastic/apm-agent-go/issues/423
• https://github.com/elastic/apm-agent-go/issues/632

[beniwohli]

For users that have set up alerting for new errors, they might get inundated by alerts right after upgrading the agent, which might be a pretty scary experience. Releasing this in a major version bump might bring more attention to the release notes, where we can point this out.

(the high frequency with which I use the word "might" might give you an idea that I'm also on the fence ;) )

Don't get me wrong, I'm not a huge fan of bumping major versions all the time, especially because of all the necessary docs dance.

[felixbarny]

Can we introduce chained exceptions without changing the logic of how to calculate the grouping key? In the next major, we can then incorporate chained exceptions into the grouping key.

[beniwohli]

@felixbarny I think that ship has sailed, as this will be part of the imminent 7.4.

[roncohen]

Our built-in error alerting only allows you to alert on "if an error happens more than X times in Y minutes". I think it will be very rare to find someone who's set up a watcher that can do new error group style alerting. I'm OK with considering this not to be a breaking change

[beniwohli]

@roncohen ah thanks for the clarification! I'll include it in the upcoming 5.2 release then (which will be released shortly before the 7.4 stack release)

[graphaelli]

Closing this out as chained exceptions shipped in 7.6.