Julia: julep: "chain of custody" error handling

[StefanKarpinski]

A of the major issues with try/catch is that it's possible to catch the wrong error and _think_ that you are handling a mundane, expected problem, when in fact something far worse is wrong and the problem is an unexpected one that should terminate your program. Partly for this reason, Julia discourages the use of try/catch and tries to provide APIs where you can check for situations that may cause problems without actually raising an exception. That way try/catch is really left primarily as a way to trap all errors and make sure your program continues as best it can.

This is also why we have not introduced any sort of typed catch clause into the language. This proposal introduces typed catch clauses, but they only catch errors of that type under certain circumstances in a way that's designed to avoid the problem of accidentally catching errors from an unexpected location that really should terminate the program. Here's what a Java-inspired typed catch clause might look like in Julia:

try
    # nonsense
    foo(1,2,3) # throws BazError
    # shenanigans
catch err::BazError
    # deal with BazError only
end

Of course, we don't have this in Julia, but it can be simulated like this:

try
    # nonsense
    foo(1,2,3) # throws BazError
    # shenanigans
catch err
    isa(err, BazError) || rethrow(err)
    # deal with BazError only
end

Although this does limit the kinds of errors one might catch, it fundamentally doesn't solve the problem, which is that while there are some places where you may be expecting a BazError to be thrown, other code that you're calling inside the try block might also throw a BazError where you didn't expect it, in which case you shouldn't try to handle because you'll just be covering up a real, unexpected problem. This problem is especially bad in the generic programming contexts where you don't really know what kind of code a given function call might end up running.

To address this, I'm proposing adding a hypothetical throws keyword that allows you to annotate function calls with a type that they're expected to throw and which you can catch with a type catch clause (it's a little more subtle than this, but bear with me). The above example would be written like this:

try
    # nonsense
    foo(1,2,3) throws BazError
    # shenanigans
catch err::BazError
    # deal with *expected* BazError only
end

The only difference is that the throws BazError comment after the call to foo became syntax. So the question is what does this annotation do? The answer is that without that annotation indicating that you _expect_ the expression to throw an error of type BazError, you can't catch it with a typed catch. In the original version where the throws BazError was just a comment, the typed catch block would _not_ catch a BazError thrown by the call to foo – because the lack of annotation implies that such an error is unxepected.

There is a bit more, however: the foo function also has to annotate the point where the BazError might come from. So, let's say you had these two definitions:

Consider something like this:

function foo1(x,y,z)
    bar(x,2y) throws BazError
end

function foo2(x,y,z)
    bar(x,2y)
end

function bar(a,b)
    throw BazError()
end

This also introduce a hypothetical keyword form of throw – more on that below. These definitions result in the following behavior:

try
    # nonsense
    foo1(1,2,3) throws BazError
    # shenanigans
catch err::BazError
    # error from bar is caught
end

try
    # nonsense
    foo2(1,2,3) throws BazError
    # shenanigans
catch err::BazError
    # error from bar is NOT caught
end

The rule is that a typed catch only catches the an error if every single call site from the current function down to the one that actually throws the error is annotated with throws ErrorType where the actual thrown error object is an instance of ErrorType. An untyped catch still catches all errors, leaving the existing behavior unchanged:

try foo1(1,2,3)
catch
    # error is caught
end

try foo2(1,2,3)
catch
    # error is caught
end

So what is the rationale behind this proposal and why is it any better than just having typed catch clauses? The key point is that in order to do a typed catch, there has to be a "chain of custody" from the throw site all the way up to the catch site, and each step has to expect getting the kind of type that's thrown. There are two ways to catch the wrong error:

1. catch an error that was not the type of error you were expecting
2. catch an error that came from a place you weren't expecting it to come from

The first kind of mistake is prevented by giving an error type, while the second kind of mistake is prevented by the "chain of custody" between the throw site and the catch site.

Another way of thinking about this is that the throws ErrorType annotations are a way of making what exceptions a function throws part of its official type behavior. This is akin to how Java puts throws ErrorType after the signature of the function. But in Java it's a usability disaster because you are required to have the throws annotation as soon as you do something that could throw some kind of non-runtime exception. In practice, this is so annoying that it's pretty common to just raise RuntimeErrors instead of changing all the type signatures in your whole system. Instead of making runtime excpetion vs. non-runtime exception a property of the error types as Java does, this proposal makes it a property of how the error occurs. If an error is occurs in an expected way, then it's a non-runtime exception and you can catch it with a typed catch clause. If an error occurs in an unexpected way, then it's a runtime exception and you can only catch it with an untyped catch clause. You can only catch errors by type if they are part of the "official behavior" of the function you're calling, where official behavior is indicated with throws ErrorType annotations.

One detail of this proposal that I haven't mentioned yet is that throw would become a keyword instead of just a function. The reason for this is that writing

throw(BazError()) throws BazError

seems awfully verbose and redundant. For compatibility, we could allow throw() to still invoke the function throw while throw ErrorType(args...) would be translated to

throw(ErrorType(args...)) throws ErrorType

This brings up another issue – what scope should the right-hand-side of throws be evaluated in? In one sense, it really only makes sense to evaluate it in the same scope that the function signature is evalutated in. However, this is likely to be confusing since all other expressions inside of function bodies are evaluated in the local scope of the function. It would be feasible to do this too, and just rely on the fact that most of the time it will be possible to statically evaluate what type expression produces. Of course, when that isn't possible, still be necessary to emit code that will do the right thing depending on the runtime value of the expression. If the r-h-s is evaluated in local scope, then we can say that throws expr is equivalent to this:

throw(expr) throws typeof(expr)

Usually it will be possible to staticly determine what typeof(expr) is, and it is a simpler approach to explain than restricting the syntax of the expression after the throw keyword.

Note that under this proposal, these are not the same:

try
    # stuff
catch
    # catches any error
end

try
    # stuff
catch ::Exception
    # only catches "official" errors
end

Also note that this proposal is, other than syntax additions that are not likely to cause big problems, completely backwards compatible: existing untyped catch clauses continue to work the way they used to – they catch everything. It would only be _new_ typed catch clauses that only catch exceptions that have an unbroken "chain of custody".

[tknopp]

+1 for the syntax addition.

I have to admit that I am not sold on the idea of expected and unexpected exceptions. I would find it rather "unexpected" if I indicate to catch something that is then not catched ;-)

For me the real issue in such situations i that the exception hierarchy is not well designed and/or that the wrong exception is used from the exception hierarchy.

[StefanKarpinski]

The problem with typed exceptions is that they give you a sense of precision, but they're not really precise at all. Making the exception part of the type of a function is a good idea – that part Java got right. What's not so good is forcing the caller to handle every kind of exception a function it uses can throw.

Here's a motivating scenario: let's say you implement a new kind of AbstractVector – say a kind of sparse vector or something – let's call it SpArray. Internally, it stores values in normal Arrays. Someone is using this and finds that sometimes they end up making indexing errors – they use a typed catch block to handle this. But you've made a programming error in implementing the SpArray type and it's actually SpArray that's encountering the indexing error internally because of a mistake. But this is caught and treated as if was a user-level indexing error. With this proposal, that situation won't occur because at the point where the SpArray implementation is incorrect and causes an indexing error, there is no throws BoundsError annotations so the resulting exception can't be caught with a typed catch clause.

[tknopp]

I absolutely see your point that it can lead to hard to find bugs if one try/catches too generic exceptions and I have been running into this myself in C++ a lot. ("Oh this throws exceptions all the time. Put try { ... } catch(...) {}around it and problem solved").

To play around with your example maybe this should be handled by throwing SpArrayBoundsError in SpArray and ArrayBoundsError in Array. Of course both deriving from BoundsError. In this way one can catch the SpArray specific bounds errors and let all other BoundsError pass through.

[StefanKarpinski]

To play around with your example maybe this should be handled by throwing SpArrayBoundsError in SpArray and ArrayBoundsError in Array. Of course both deriving from BoundsError. In this way one can catch the SpArray specific bounds errors and let all other BoundsError pass through.

That's a common solution offered in languages with exception hierarchies and typed catch, but I don't buy it. The logical extreme of that approach is to have an error type for almost every single _location_ where an exception can be thrown. At that point, what you're really doing is labeling individual exception locations and using labels to catch things – albeit labels that are carefully arranged into a hierarchy. That need for a carefully arranged hierarchy to make the approach tenable is also fishy – and easily abused. I came up with this approach by thinking about how to compromise between a broad classification scheme with a set of standard exception types and individual labels for exception locations.

[tknopp]

Yes you are absolutely right. Exception hierarchies have the potential to get very specific up to the point that specific lines get specific error types. But as one subtypes from broader exception types I don't see the issue with that. But maybe I just have bad taste to not think this is fishy :-)
So please lets look what others think about this.

[JeffBezanson]

Talking about this as "labeling locations" gets me thinking: would it be possible just to label the locations? Every exception could consist of an exception object and a label (symbol). The SpArray library could throw BoundsErrors, labeling them as :SpBoundsError. You could catch by type and/or label.

[aviks]

You're right that using typed exceptions correctly in Java is annoying. Everyone throws RuntimeException subclases, and I've trained myself to see the Exception type as primarily informational, useful only for logging.

But my fear is, annotating exceptional (?) call sites all the way down is going to turn out to be equally tedious. Is this something that is easy enough to reason about and implement, so that it will be widely practiced?

[StefanKarpinski]

@JeffBezanson – I thought about just labeling locations, but it seems too granular. Sometimes different locations throw the same error, no? Also, how is that different from having an exception type for every location? I guess you wouldn't have to declare the types, but you also wouldn't get any type hierarchy. How would labels be scoped? By module?

[JeffBezanson]

Yes, you'd probably want some kind of scoping, which remains to be worked out. But different locations could in fact throw the same error; in my overly-simple formulation they could just use the same symbol.

The idea is that exception and origin are orthogonal: what and where-from. "What" is the exception type hierarchy, which describes problems, and "where-from" is the locations, which might follow the module hierarchy, or have no hierarchy.

[StefanKarpinski]

@aviks – I think this should actually be pretty rare. Having a "throws" annotation is essentially saying "throwing this error is part of the official API of this function". Currently we basically consider all exceptions to be runtime errors, so anything is real problem and not part of the functions official behavior. That would continue to be the default. Only when you decide that something like BoundsErrors or KeyError is part of the official interface of AbstractArray or Associative would you put throws BoundsErrors and throws KeyError annotations on function calls. And hopefully you wouldn't need very many of them. I should probably try out adding these annotations for something like that and see how it goes.

[tknopp]

The labeling idea is interesting. Still it increases the dimensionality of the dispatch mechanism from a one-dimensional to a two-dimensional thing.

I am all for putting more context into exceptions. In C# for instance one usually also gets the line numbers where the exception was thrown. Further when debugging exceptions I found it very useful to have exception chains, i.e. when an exception is rethrown (as a different exception type) the original exception is put as an "inner exception" to the outer exception.

My own experience is that in small projects and research code exceptions are mainly an error mechanism for debugging "not yet correct" code. In these situations one will hardly use try/catch and reason about what exception to catch. The intensive use of the error function in Julia base is an indicator that this is what most people use exceptions for. The importance for exceptions increased a lot for me when working on large projects and projects where others are the users (i.e. production code). In these cases one has to prevent by all means that exceptions remain uncatched and thus has to maintain a sensible exception hierarchy and use try/catch at various locations. But this is also usually nothing where the system exception hierarchy is used. Instead I want to throw TheUSBCableIsUnplugged exceptions that are carefully catches and translated to ThePrinterIsNotAvailable and so on. And for this purpose the Julia exception system is already working quite well.

[samoconnor]

I am in the process of porting my Tcl AWS library to Julia as a way to learn Julia, (and so that I can then experiment with implementing projects reliant on AWS in Julia).

In a first-pass of the Julia docs, I saw try/catch/finally and made a mental check "that is supported".
When it comes to writing actual code I am surprised that try/catch doesn't do what I expect. Most Julia features have either made me think "nice, that's how it should be done", or "that's weird" followed by some reading followed by "that's nice". The design of Julia's try/catch seems to be that it is deliberately somewhat crippled to try to wean people off using it. This smells wrong to me.

In distributed, networked systems individual nodes and connections can fail there are propagation delays, data models are eventually-consistent. Clear, robust, exception handling is a must. I want to write code that assumes everything is reliable, immediately globally updated etc and deal with the exceptional glitches in one place in high-level retry or conflict resolution loops.

I am not a fan of type-based exception handling, having had the misfortune of working with Java etc. Tcl has no types, so exceptions are trapped based by matching an error-code prefix. An error-code in Tcl is just a list.

e.g.

proc get {url} {
    ...
    throw {HTTP 404}  "Error: $url Not Found"
    ...
    throw [list HTTP 300 $location] "Error: $url has moved to $location"
    ...
}

try {

   get $url

} trap {HTTP 404} {} {
    puts "get can't find $url"
} trap {HTTP 300} {message info} {
    set location [lindex $info 2]
    get $location
}

# Ignore missing dict key...
proc lazy_get {dict key} {
    try {

        dict get $mydict "key"

    } trap {TCL LOOKUP DICT} {} {}
}

# Catch exit status of shell command...
retry count 2 {

     exec ./create_queue.tcl "foobar"

 } trap EX_TEMPFAIL {} {
    after 60000
}

Every error (exception) in a Tcl program is supposed to have a human-readable error message and a machine-readable errorcode.

The source of an error can include as much or as little machine-readable info as it likes.

Safe exception handling is achieved by trapping only sufficiently specific errorcodes. Of course you can shoot yourself in the foot by writing "trap {}" if you want to...

http://www.tcl.tk/man/tcl8.6/TclCmd/try.htm
http://www.tcl.tk/man/tcl8.6/TclCmd/throw.htm

I think using types to match exceptions isn't great. But since Julia matches methods by type as a core feature, perhaps it makes sense for Julia. I would prefer something like passing a Dict() to throw and having a convenient syntax for catching only exceptions that match a dictionary pattern.

The key thing is readability of the try/catch code.

Below are some fragments of Tcl code that I am face with porting to Julia.
I am open to the idea that there is just a better way to do these things in Julia. But on the other hand, I think these are reasonable uses of an exception mechanism...

try {

    set web_user [create_aws_iam_user $aws $region-web-user]

} trap EntityAlreadyExists {} {
    delete_aws_iam_user_credentials $aws $region-web-user
    set web_user [create_aws_iam_user_credentials $aws $region-web-user]
}

    retry count 4 {

        set res [aws_sqs $aws CreateQueue QueueName $name {*}$attributes]

    } trap QueueAlreadyExists {} {

        delete_aws_sqs_queue [aws_sqs_queue $aws $name]

    } trap AWS.SimpleQueueService.QueueDeletedRecently {} {

        puts "Waiting 1 minute to re-create SQS Queue \"$name\"..."
        after 60000
    }

proc fetch {bucket key {byte_count {}}} {

    if {$byte_count ne {}} {
        set byte_count [list Range bytes=0-$byte_count]
    }
    try {

        s3 get $::aws $bucket $key {*}$byte_count

    } trap NoSuchKey {} {
    } trap AccessDenied {} {}
}

proc aws_ec2_associate_address {ec2 elastic_ip} {

    puts "Assigning Elastic IP: $elastic_ip"
    retry count 3 {

        aws_ec2 $ec2 AssociateAddress InstanceId [get $ec2 id] \
                                      PublicIp $elastic_ip

    } trap InvalidInstanceID {} {
        after [expr {$count * $count * 1000}]
    }
}

proc create_aws_s3_bucket {aws bucket} {

    try {

        aws_rest $aws PUT $bucket Content-Type text/plain Content [subst {
                <CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
                  <LocationConstraint>[aws_bucket_region $bucket]</LocationConstraint>
                </CreateBucketConfiguration>
        }]

    } trap BucketAlreadyOwnedByYou {} {}
}

proc delete_aws_sqs_queue {queue} {

    try {

        aws_sqs $queue DeleteQueue

    } trap AWS.SimpleQueueService.NonExistentQueue {} {}
}

proc aws_s3_exists {aws bucket path} {

    try {

        aws_s3_get $aws $bucket $path Range bytes=0-0
        return 1

    } trap NoSuchKey {} {
    } trap AccessDenied {} {}
    return 0
}

    retry count 3 {

        return [$command $aws {*}$args]

    } trap {TCL LOOKUP DICT AWSAccessKeyId} {} {

        set aws [get_aws_ec2_instance_credentials $aws]

    } trap ExpiredToken {message info} {

        if {![exists ::oc_aws_ec2_instance_credentials]} {
            return -options $info $message
        }

        puts "Refreshing EC2 Instance Credentials..."
        set aws [get_aws_ec2_instance_credentials $aws -force-refresh]
    }

    try {

        aws_iam $aws DeleteInstanceProfile InstanceProfileName $name

    } trap NoSuchEntity {} {}

    set response [aws_iam $aws CreateInstanceProfile \
                               InstanceProfileName $name \
                               Path $path]

[hayd]

other code that you're calling inside the try block might also throw a BazError where you didn't expect it

This seems like an edge case, if you keep the try block small (e.g. just the line you're expecting a failure) it shouldn't be the case... if it's critically different why not subclass BazError?

+1, a syntax for typed exceptions would be great. IMO isa(err, BazError) ... is quite messy. I like:

catch err::BazError
    # deal with *expected* BazError only
catch err::FooError
    # deal with *expected* FooError only
end

[elextr]

if you keep the try block small (e.g. just the line you're expecting a failure) it shouldn't be the case...

The syntax proposed by @StefanKarpinski is really just a more compact syntactic sugar for a try block around the important line (or is it intended to apply to an expression?) which passes the exception to the outer catch.

if it's critically different why not subclass BazError?

You can't when its thrown by a library you called, not your code.

The concept of identifying several small regions where "it is understood that this may throw xxxerror, and I'm prepared to handle it in a common handler" seems a good idea, but as currently proposed this is likely to surprise those coming from other languages (who don't RTFM because they expect all languages to be the same :)

This also doesn't provide for one of the traditional use-cases where you want to catch some types of errors no matter where they occur, but let the others go:

 try
    a big fast but flakey algorithm
 catch err::matherror # oh no underflowed, overflowed or divided by zero
    alternative slow algorithm
 end

[StefanKarpinski]

@hayd: This seems like an edge case, if you keep the try block small (e.g. just the line you're expecting a failure) it shouldn't be the case... if it's critically different why not subclass BazError?

Even if the try block only calls a single function, that function can do absolutely anything and throw any kind of exception for many different reasons. In generic code this is particularly bad since you don't really know which implementation of the function you're calling is going to be invoked or how it's implemented. Of course, you shouldn't have to know this to use it correctly – and that's precisely the problem with type-based exception handling: there's massive abstraction leakage between the implementation of the function being trapped and how to trap it. Let's say you're doing try f() catch err::Foo ... end and you happen to know that f only throws the Foo error under certain circumstances and you want to handle that situation. Great. But now, let's say someone changes the implementation of f – or of any function that f calls. The change seems innocuous to them, but they happen to call a function – probably unwittingly – that can throw a Foo error under some obscure circumstances. You will now be trapping this situation even though that's not what you meant to do at all. Their seemingly harmless change has now made your code incorrect.

You could introduce new error types, but that's just completely annoying and impractical. Let's say I add a new subtype of AbstractArray. Since it's array-like, you want to throw IndexError when someone indexes into it wrong. Under the hood, however, it uses normal arrays and indexes into them, which might itself cause an IndexError. In order to distinguish the former condition from the latter, your type needs to introduce a new subtype of AbstractIndexError. This means that every standard error in Base needs to be split into an abstract type and a concrete type per implementation of that type: AbstractIndexError, ArrayIndexError, SparseIndexError, etc. And if you implement a new subtype of AbstractArray, you need your own subtype of AbstractIndexError in order to really correctly implement it. This is almost absurdly unwieldy.

[StefanKarpinski]

@samoconnor – I really don't think try/catch is a good mechanism for handling that kind of networking timeout situation. I'm not sure what a better mechanism is, but this doesn't really feel right. In particular, you probably want to be able to retry things, which try/catch does not handle.

[jakebolewski]

A networking timeout situation is usually handled by a state machine. We have gotos with labels now which makes writing state machines easier.

[samoconnor]

@StefanKarpinski - Re: "retry", note that several of my examples above use a "retry" variant of "try". Viral Shah said in a separate conversation "The short answer to your question is yes, it is easy to do retry with Julia’s macros". So I'm assuming that "retry" can be made to work as a seamless "try" variant in Julia.

Re: "network timeouts", that is not really what my examples above address. I'm dealing with timeouts like this:

    for wait_time in {0.05, 0.5, 5, 0}

        try

            response = process_response(open_stream(uri, request))

            if (response.finished
            &&  response.status >= 200 && response.status < 500)
                return response
            end

        catch e
            if wait_time == 0 || !isa(e, UVError)
                rethrow(e)
            end
        end

        sleep(wait_time)
    end

I have no choice here but to use try/catch because the underlying sockets library throws UVError.
The retry loop is safe against catching some unexpected variant of UVError, because after a few attempts it throws the error up anyway.

It might be nice to simplify the catch code a little

catch e::UVError
    wait_time > 0 || rethrow(e)

or maybe

catch e if isa(UVError, e)
    wait_time > 0 || rethrow(e)

... the "if" syntax would the allow:

catch e if isa(UVError, e) && uverrorname(e) == "EAI_NONAME"

Re: typing of exceptions, note that in my examples above there is no mention of trapping timeout exemptions. The things I'm trapping are all well-defined and precise. e.g. EntityAlreadyExists, AWS.SimpleQueueService.QueueDeletedRecently, AccessDenied, ExpiredToken, {TCL LOOKUP DICT AWSAccessKeyId}, BucketAlreadyOwnedByYou.

While it is true that there is code out there that makes a mess of exception meta-data (I've written some android code) there are plenty of APIs, e.g. AWS, that have robust exception identification.

It is essential to have an out-of-band, stack-frame-jumping, exception mechanism in complex code that sits on top of imperfect web services. The alternative is that all of the code is dominated by deciding what error information needs to be passed back up the stack and how far.

As an example of converting HTTP exceptions, In my Julia AWS library I currently do this:

        # Handle other responses as error...
        err = TaggedException({
            "verb"      => r.verb,
            "url"       => r.url,
            "code"      => string("HTTP ", response.status),
            "message"   => response.data
        })

        # Look for JSON error message...
        if response.headers["Content-Type"] == "application/x-amz-json-1.0"
            json = JSON.parse(response.data)
            if haskey(json, "__type")
                err.tags["code"] = split(json["__type"], "#")[2]
            end
        end

        # Look for XML error message...
        if response.headers["Content-Type"] in {"application/xml", "text/xml"}
            xml = parse_xml(response.data)
            if (v = get(xml, "Code")) != nothing
                err.tags["code"] = v
            end
            if (v = get(xml, "Message")) != nothing
                err.tags["message"] = v
            end
        end

and

catch e
    if isa(TaggedExeption,e) && e["code"] == "BucketAlreadyOwnedByYou"
        ...
    else
        rethrow(e)
    end
end

I think I'm too new to Julia to contribute great solutions, but I hope that my real-world examples help in some way...

[samoconnor]

Following up on: "In particular, you probably want to be able to retry things, which try/catch does not handle."

I've managed to figure out enough macro-fu to implement an exception handling retry loop.

On the first n-1 attempts, the catch block has the opportunity to "@retry".

If the catch block does not call @retry, the error is rethrown automatically.

On the nth attempt the try block is executed naked (without a catch block) so errors are passed up.

Example follows...

(please let me know if my posts on this issue are too off-topic, and/or where a better place to post would be.)

# Try up to 4 times to get a token.
# Retry when we get a "Busy" or "Throttle" exception.
# No retry or catch if we get a "Crash" exception (or no exception).

@with_retry_limit 4 try

    println("Trying to get token...")
    println("Token: " * get_token())

catch e

    if e.msg == "Busy"

        println("Got " * string(e) * ", try again...\n")
        @retry

    elseif e.msg == "Throttle"

        println("Backing off...\n")
        sleep(1 + rand())
        @retry
    end
end



# Unreliable operation simulator.
# Often busy, sometimes asks us to back off, sometimes crashes.

function get_token()

    if rand() > 0.2
        error("Busy")
    end

    if rand() > 0.8
        error("Throttle")
    end

    if rand() > 0.9
        error("Crash")
    end

    return "12345"
end


# Implementation of retry

macro with_retry_limit(max::Integer, try_expr::Expr)

    @assert string(try_expr.head) == "try"

    # Split try_expr into component parts...
    (try_block, exception, catch_block) = try_expr.args

    # Insert a rethrow() at the end of the catch_block...
    push!(catch_block.args, :(rethrow($exception)))

    # Build retry expression...
    retry_expr = quote

        # Loop one less than "max" times...
        for i in [1 : $max - 1]

            # Execute the "try_expr".
            # It can do "continue" if it wants to retry...
            $(esc(try_expr))

            # Only get to here if "try_expr" executed cleanly...
            return
        end

        # On the last of "max" attempts, execute the "try_block" naked 
        # so that exceptions get thrown up the stack...
        $(esc(try_block))
    end
end



# Conveniance "@retry" keyword...

macro retry() :(continue) end

[samoconnor]

Here is an attempt to avoid having to put rethrow(e) at the end of a catch block. This is intended both as a way to remove a bit of noise from the code, and as a safety net to avoid accidentally catching unintended exceptions.

I think the thing that made me most scared about Julia's "catch" at first was that everything is caught by default, then you have to be really careful to rethrow() the right exceptions.

#!/Applications/Julia-0.3.0-rc1-a327b47bbf.app/Contents/Resources/julia/bin/julia


# Re-write "try_expr" to provide an automatic rethrow() safety net.
# The catch block can suppress the rethrow() by doing "e = nothing".

macro safetynet(try_expr::Expr)

    @assert string(try_expr.head) == "try"

    (try_block, exception, catch_block) = try_expr.args

    push!(catch_block.args, :(isa($exception, Exception) && rethrow($exception)))

    return try_expr
end


@safetynet try

    d = {"Foo" => "Bar"}

    println("Foo" * d["Foo"])
    println("Foo" * d["Bar"])

    @assert false # Not reached.

catch e

    if isa(e, KeyError) && e.key == "Bar"
        println("ignoring: " * string(e))
        e = nothing
    end
end

println("Done!")

Question about macros, is there any way I can omit the "try" keyword and have the macro insert it for me? e.g.

@safetry
    ...
catch
    ...
end

I can't figure out if this is possible with Julia's macro system.

[MikeInnes]

@StefanKarpinski Just for clarification, I take it that the "chain of custody" works on a per-method basis? So to modify your original example:

function test(x)
  try
    # tomfoolery
    foo(x) throws BazError
    # highjinks
  catch e::BazError
    # deal with BazError
  end
end

function foo(x::Number)
  bar(x) throws BazError
end

function foo(x::String)
  bar(x)
end

function bar(a)
  throw BazError()
end

test(1) catches the error, but test("foo") doesn't?

[MikeInnes]

Also, there's at least one place this would be useful in Base, in the display code. Checking the function that threw the exception improves the situation but is still pretty brittle, and it sounds like this proposal would solve that problem.

[StefanKarpinski]

@one-more-minute, yes, that's how I think it should work. In other words even though foo has BazError as part of its overall interface, a BazError raised by foo(::String) via bar would be considered accidental since it's not annotated. I agree with the sentiment that this is "an edge case", and indeed many languages have gotten by without anything like this. However, the difference between handling edges cases correctly and not quite correctly is a fairly major one.

Also, +1 for "tomfoolery" and "hijinks" :-)

[StefanKarpinski]

@samoconnor – I think that handling this kind of complex I/O error is a really important problem but I also think it's beyond the scope of this issue, which is pretty focused: it solves just the problem of trapping the right errors and not accidentally trapping other ones.

[StefanKarpinski]

@JeffBezanson, so here's what I think is wrong with the type and/or label idea.

Let's say you have code that expects an AbstractArray. Part of the AbstractArray interface is that indexing can throw a bounds error like so:

julia> a = [1,2,3]
3-element Array{Int64,1}:
 1
 2
 3

julia> a[4]
ERROR: BoundsError()
 in anonymous at ./inference.jl:365 (repeats 2 times)

Your write some code that expects this and handles it somehow:

function frizz(a::AbstractArrary)
  try a[4]
  catch err::BoundsError
    # handle a not being long enough
  end
end

If someone calls frizz(::Array) assuming no other frizz methods, and a BoundsError is thrown bygetindex(::Array, i::Int)` it should clearly be caught here.

Now, let's say someone else comes along and implements a new kind of AbstractArray, XArray and as is quite common, they use and existing kind of AbtsractArray to do it – let's say they use an Array instance. Now further suppose that there's some kind of situation where the inner Array type can cause a BoundsError which doesn't actually correspond to BoundsError for XArray – it's a programming error, or just indicates some other kind of problem with the usage of XArray. This is an entirely plausible situation that actually happens quite a bit in the sparse array implementation.

The key observation is that you can have two situations with the exact same throw and catch sites: one where error should be caught and the other where the error shouldn't be caught. The type and/or label idea can't solve this problem since it still only depends on the throw and catch sites – one error is caught if and only if the other one is. To actually solve the problem, whether an error is caught must depend, not only on the throw and catch sites, but also on the stack between them. This is precisely what the chain of custody idea does: it makes catching an error depend on the stack between the thrower and the catcher. The exact details of how to express that are up for debate, but I think this argument makes it clear that we either decide we don't care about this problem, or we need a solution where catching an error depends on the stack that comes between thrower and the catcher.

[JeffBezanson]

Ok, well I'm not running right out to implement the label idea :)

The chain of custody is a really clever idea, but I feel it has two fatal issues: (1) It's not clear that it can be implemented efficiently. A naive implementation would probably slow down the whole language. (2) I believe it will strike most people as un-julian. Even though it is better than java's mechanism and not quite as verbose, I suspect most people will still find it fussy.

[StefanKarpinski]

That's why I left "decide we don't care about this problem" as one of our possible courses of action.

[StefanKarpinski]

Keep in mind that this is completely backwards compatible: no existing code would behave any differently with this proposal. So it's hard to argue that it's somehow more annoying that what we have currently. It is possible to argue that it's more annoying than having typed catch clauses would be, but we've been getting by without those, so that's also hard to see as really a huge problem. It's possible that more of the chain could be implicit, but I rather suspect that these chains will typically be very short – usually just one or two calls deep. Deeper errors are certainly possible, but they are usually of the "catch anything and try to recover" variety, not the "catch a very specific, expected error in this specific method call" variety – which by its very nature would tend not to be a very deeply nested error.

[JeffBezanson]

Yes, the annoyance involved is subtle, which is one of the clever parts of this. I'm more worried about the non-locality that results when somebody wants to catch an exception by type. For it to work, potentially many call sites in the system need to be annotated. When writing a function call in a library, it is hard to guess whether somebody will want to typed-catch an exception from it.

It's fine with me to annotate the call inside a try block that I expect might fail, because that's still local. Maybe one issue is that try blocks just tend to be too big --- there might be only one call in there you want to catch from, but writing a try block around a single call is awkward. Maybe extra syntax like throws just to narrow the effective range of a try block would be good. That would be kind of a lexically-scoped version of your proposal.

[StefanKarpinski]

It's fine with me to annotate the call inside a try block that I expect might fail, because that's still local. Maybe one issue is that try blocks just tend to be too big --- there might be only one call in there you want to catch from, but writing a try block around a single call is awkward. Maybe extra syntax like throws just to narrow the effective range of a try block would be good. That would be kind of a lexically-scoped version of your proposal.

I think this only addresses the most superficial part of the problem, so isn't very satisfying.

[tknopp]

Sorry but I stay at my position. When writing

catch err::BoundsError

I want to catch every occurrence of a BoundsError. I don't want to reason about the internal code and that nobody has forgotten the throws BoundsError declaration. In my opinion exceptions are complicated enough so that it is a big plus if the implementation is simple. And until now Julia has gotten this very right. I just miss to catch specific exceptions as its available in most other languages that have this.

P.S.: Greatest example of a "problematic" exception system is C++ where it is possible to throw integers around and one has no common base class

[StefanKarpinski]

Not understanding a proposal or the problem that it solves after a cursory read-through is not a particularly compelling counterargument. Exceptions themselves were a crazy new idea at some point.

[ssagaert]

On 16 Aug 2014, at 19:57, Stefan Karpinski [email protected] wrote:

Not understanding a proposal or the problem that it solves after a cursory read-through is not a particularly compelling counterargument. Exceptions themselves were a crazy new idea at some point.

—
Reply to this email directly or view it on GitHub.

Well true of course but when I first learned about exceptions in C++ a very long time ago (1996) I grasped the concept immediately and thought this was way better that error codes like in C so they weren’t that crazy ;) . From what I understood is that your are trying to distinguish between “exceptional” exceptions and “unexceptional” exceptions and I don’t know if that is a good idea.

[JeffBezanson]

I guess my issue with it comes down to it being, in effect, dynamically scoped. This makes it inefficient, except perhaps with truly heroic efforts, and harder to reason about. Imagining a fully-realized implementation of this, pick a random f(x) throws y, and I would claim you cannot figure out why the throws is there based on local reasoning. One layer's exception-to-be-caught is another layer's unexpected-error. In fact that is sort of the whole point of exceptions.

[ssagaert]

I would support the idea of adding "throws ExceptionType1" to a function definition for documentation purposes. This could also be used by a future API doc generating tool (like Javadoc does). However without the implications of checked exceptions like in Java. So when that exception is not caught at the enclosing calling scope, there is no need to put the same "throws ExceptionType1" in every function up the call stack.

[StefanKarpinski]

I guess my issue with it comes down to it being, in effect, dynamically scoped. This makes it inefficient, except perhaps with truly heroic efforts, and harder to reason about. Imagining a fully-realized implementation of this, pick a random f(x) throws y, and I would claim you cannot figure out why the throws is there based on local reasoning. One layer's exception-to-be-caught is another layer's unexpected-error. In fact that is sort of the whole point of exceptions.

Since I don't yet have an efficient implementation strategy, that's a very valid concern, but I think it's premature to nix an otherwise good idea because we haven't figured out how to do it fast. Of course, I wouldn't add it to the language without already knowing how to do it efficiently.

As to when f(x) throws y should be annotated, it's local reasoning in the exact same way that the type signature of the function is local: a function body should have a throws y annotation at points where the function throws an exception that is part of its expected behavior – i.e. part of its type signature. Throwing a BoundsError is a reasonable part of the signature of a getindex methods, whereas through a DivideError is not, even though a division error may well occur in the course of working with indices.

[elextr]

I would also add a note to the efficiency concerns of @JeffBezanson that C++11 changed exception specifications partly due to the implementation cost, the reasoning behind that could be worth closer inspection if someone has a copy of the standards committee rationale docs.

If I understand @StefanKarpinski correctly you would still be able to do the universal catch of all exceptions and then test via ifs for those you can handle and re-raise the others. This will make the cost visible, and only to those who need it.

[tknopp]

I don't think that implementation cost is the issue here. Its:

• The non-locality. If you see catch e::BazError you cannot be sure that BazError is really catched.
• The expectation. The majority of users with a C++/Java/C# background will we confused.
• People will use the isa and rethrow way if they want the standard use of exception. So code that could look much more clear is written in the "workaround" version.

For me this is kind of the switch clause in C++ where my initial expectation is not a default fallthrough. One finds workaround patterns for it (put a break everywhere) but I still think that it was not clever to design it that way.

[StefanKarpinski]

These are valid concerns, @tknopp. My only counterargument is that in the cases where catch e::BazError wouldn't catch a BazError, it shouldn't have been caught anyway – and even if that's a bit confusing to a C++/Java programmer, it makes their program (unwittingly) more correct.

[tknopp]

hehe. "Dear C++/Java programmer, we are more clever than you. You wrote ... but we think it would be better if you had written ..." :-) Just kidding.

[elextr]

To me the non-locality is really an inversion of control, the lower level functions are claiming to know what the upper level functions should be allowed to handle. But it is repeatedly shown that library (lower level code) developers often don't imagine even a part of the places where their code gets used.

IIUC the original concern of @StefanKarpinski was that you don't know the meaning of an exception thrown from some lower level library is the same as that exception thrown from a higher level function, so its not safe to handle it the same. Is this not simply that overly general exceptions are being re-used, rather than more specific ones being created by the upper level functions, allowing handlers to distinguish the exception they want to handle?

The expectation answer is more like "Dear C++ programmer (me) this is Julia, _not_ C++ ..." :-)

[ssagaert]

+1
On 17 Aug 2014, at 08:17, Tobias Knopp [email protected] wrote:

I don't think that implementation cost is the issue here. Its:

The non-locality. If you see catch e::BazError you cannot be sure that BazError is really catched.
The expectation. The majority of users with a C++/Java/C# background will we confused.
People will use the isa and rethrow way if they want the standard use of exception. So code that could look much more clear is written in the "workaround" version.
For me this is kind of the switch clause in C++ where my initial expectation is not a default fallthrough. One finds workaround patterns for it (put a break everywhere) but I still think that it was not clever to design it that way.

—
Reply to this email directly or view it on GitHub.

[samoconnor]

"... an exception thrown from some lower level library is the same as that exception thrown from a higher level function, so its not safe to handle it the same. Is this not simply that overly general exceptions are being re-used, rather than more specific ones being created by the upper level functions?"

Yes.
This is just bad API design. If you're calling a function from a library (or elsewhere) the documentation for that function should tell you exactly what results (including side-effects and exceptions) the function produces. If one of the possible results is a FooError, the documentation should tell you what that means in the context of the function. What happens inside the function, and what other functions it calls, is implementation detail and should be of no concern to the caller. Function == encapsulation.

If function a() and function b() can both throw FooError, and function a() calls function b(), then the author of function a() needs to do one of:

• catch FooError thrown by function b() and deal with it
• ensure that there is no semantic difference between directly throwing FooError and indirectly throwing b()'s FooError.

If the author of function a() calls function b() without bothering to handle all possible return states, then function a() may be ok as interesting exploratory code, but it has no place in a library. This is like not checking sys call return codes in a C program. At the very least function a() should assert that function b() has some expected return state.

try x = b(y) catch e @assert false string(e) end

[nalimilan]

I find @StefanKarpinski's proposal quite interesting provided it could be implemented efficiently. It makes sense that _by default_, people trying to catch BazError won't catch it if it comes from a lower function in the stack, from where they probably don't expect the exception to come.

But maybe there should also be a simple syntax allowing to catch BazError disregarding where it comes from? It would not be the most natural syntax, so that people don't use it without enough reflection, but it wouldn't be completely ugly either, like catchall e::BazError. It could be used in precise cases, e.g. to work around bugs in the code you call when the author forgot to add the throws BazError annotation, or do perform some debugging. If it turns out it's not really useful, then it could easily be deprecated.

[wavexx]

I was about to file a report about the 'catch' statements when I stumbled on
this (julep label?).

I find @StefanKarpinski proposal interesting, but not completely satisfying.

First, I think the main problem with exceptions is that exceptions are often
not only used as pure error conditions, but also as generic non-local returns.
This is when there is a debate whether _all_ exceptions should be handled
(because they are supposed to be errors to handle correctly!), or should be
left free to the programmer to handle (because it's just a different control
flow mechanism).

From experience, there are _many_ situations that benefit from generic
non-local returns. It turns out that most languages implement non-local returns
using exceptions (as in "error exceptions") instead of doing the opposite:
implement errors on top of a generic non-local return, which would remove
the ambiguity (errors would then be _always_ errors).

I think that I don't need to discuss the advantages/disadvantages or non-local
returns here. Coming from a CL background, there are situations which benefit
from them, and others that do the opposite. This largely depends on the
situation at hand. The only thing that I want to point out is that a language
without non-local returns ends up be _very_ verbose in several situations, and
error checking is the most common (I'm looking at you, Go). Repetition also
makes code just as error-prone.

Back to the proposal: having to declare each usage as throwing the exception
would often be very verbose:

    func throws Condition
    func throws Condition
    func throws Condition
catch ::Condition
    ... 
end

This is definitely worse than the current mechanism:

    func
    func
    func
catch e
    isa(e, Condition) || rethrow(e)
    ...
end

This compounds if you consider that forgetting a 'throws' declaration will
cause your code to silently work anyway, but fail upon receiving an exception.
The failure will be downward in the stack. Trickier to debug than a regular
catch mechanism.

You say this proposal it's all about being explicit (just to be clear: I'm
totally on-board with this assumption), so if you want this to actually help
the developer, some compilation warnings are _required_. If you declare a
function with 'throws' once, and use the same function again in the same block,
a warning would be warranted.

This leads to the second issue, which is about the left-hand side of the
'throws' statement. What about closures? What happens with macros?

Note that I also have concerns with the current situation of 'catch' in the
current way of doing things. It feels like that the default is to catch _any_
error in the block, but this is _hardly_ the case. In fact, in my experience,
the exact opposite is true: you often want to catch exactly _one_ specific
error, because you know beforehand _that's_ the case you can handle. It's
currently too easy to either trap any error or to forget a rethrow. Both
situations are just as bad. The common catch x::Type idiom is a good
compromise, which actually just avoid this kind of mistakes.

[tknopp]

@wavexx: I fully agree

[samoconnor]

FWIW I've cleaned up my @retry if... and @ignore if... exception handling macros, written some test cases and published them here https://github.com/samoconnor/Retry.jl.

The @ignore if... macro attempts to address @StefanKarpinski's initial concern about the "possibility of catching the wrong error" by making it easier to write exception filtering statements.

[wavexx]

On 24/12/15 16:29, samoconnor wrote:

FWIW I've cleaned up my |@retry if...| and |@ignore if...| exception
handling macros, written some test cases and published them here
https://github.com/samoconnor/Retry.jl.

The |@ignore if...| macro attempts to address @StefanKarpinski
https://github.com/StefanKarpinski's initial concern about the
"possibility of catching the wrong error" by making it easier to write
exception filtering statements.

[side note: I've seen an email notification from Mason Bially but cannot
find it here, and ended up forgetting about it].

The problem that I have with these macros is that they're not based on
types. I'd like the exception mechanism based on type dispatch foremost,
just like regular function calls, to make it more uniform.

If catch could have arguments with type declarations I would already be
partially satisfied. The default behavior _should_ be rethrow.

I also think proper exception handling is a bit [too much] overdue.

The amount of incorrect catch blocks I've seen in julia modules is
downright frightening, and reminds me too much of the situation there's
already in R, where the error handling has a similar interface.
People throw values and catch anything, conflating a hard I/O error with
a non-local exits. The result is that hard I/O errors go unnoticed, and
the system returns faulty values instead of failing.

A quick fix for this might just have two base types for throw to emit:
the first for a non-local exit (let's call it "exit"), the second for a
runtime error ("error").

If the type throwed is based on the 'exit' type, and is not catched on
the current frame, emit a warning. The intention of 'exit' is to have
non-local returns at hand, that we'd like to handle them locally as well
without performing nested if/else blocks. Clear intention, with
compiler-assisted warnings. This would allow to write simple macros that
help with retry/delay behavior with meaningful warning messages for the
user. Similarly, if there's no throw statement of type 'exit' in the
local frame, but catch ::exit is used, another warning is warranted (we
probably do not want to catch exits from another frame).

In my vision, a lone catch without arguments (a catch everything block)
should emit a warning that should be silenced explicitly (like an
"uninitialized variable warning in C" so to speak). If there are already
catch declarations with types instead, we assume the programmer is
actually handling the default behavior by itself.

[mason-bially]

[side note: I've seen an email notification from Mason Bially but cannot find it here, and ended up forgetting about it].

You are probably thinking of this.

For which I would submit as my thoughts on the subject, in addition to repeating yours:

I also think proper exception handling is a bit [too much] overdue.

[samoconnor]

Hi @wavexx,

The problem that I have with these macros is that they're not based on types.

You can use these macros with types, e.g.:

@repeat 3 try 
    ...
catch e
    @ignore if isa(e, EntityAlreadyExistsError) end
    @delay_retry if isa(e, NetworkTimeoutError) end
end

Fine grained exception filtering is important. If the library you are calling has fine grained exception types then all you need is type filtering. But if you're calling code that returns something with a .code field, then you need to filter on the code.

The macros attempt to make it easy to filter by type or by code.
A key point in the implementation of this is that errors thrown by the exception filtering condition expression are ignored. This allows a condition like e.code == "AccessDenied" without having to worry that the exception thrown might not have a .code field. The condition expression is true if it evaluates without error and evaluates as true.

Of course, macros are limited in what they can do, a more elegant solution would be possible if it was built into the language. In the meantime, I've found that these macros allow me to get on with writing code without worrying about making mistakes in catch code.

I'd like the exception mechanism based on type dispatch foremost,

In AWSCore.jl I had a signle AWSException type with a .code field. I cannot know in advance what all the values of .code will be because they are dynamically parsed from XML/JSON/Headers etc (e.g. .code could be EntityAlreadyExists, or AWS.SimpleQueueService.NonExistentQueue or AWS.SimpleQueueService.QueueDeletedRecently or BucketAlreadyOwnedByYou or NoSuchKey or AccessDenied or something I haven't seen yet).

I'm experimenting with making AWSException an abstract type and creating a new subtype dynamically for these error codes. What do you think about this approach:
https://github.com/samoconnor/AWSCore.jl/blob/master/src/AWSException.jl#L53

[wavexx]

On 01/02/16 00:59, samoconnor wrote:

@repeat 3 try
...
catch e
@ignore if isa(e, EntityAlreadyExistsError) end
@delay_retry if isa(e, NetworkTimeoutError) end
end

Fine grained exception filtering is important. If the library you are
calling has fine grained exception types then all you need is type
filtering. But if you're calling code that returns something with a
|.code| field, then you need to filter on the code.

Ignoring @repeat and focusing just on the catch mechanism, the above
approach is not much different compared to what you can do currently by
hand, really.

I assume you just retrow when exiting from the catch block?

Moreover, I'm not arguing to remove catch-all definitions. These will
always be necessary. I just think these should be _rare_. I won't repeat
what I wrote above.

I'm experimenting with making AWSException an abstract type and creating
a new subtype dynamically for these error codes. What do you think about
this approach:

Dynamic types for exceptions are generally not a good idea. If the code
downstream doesn't know the possible exceptions types, the type system
doesn't give you anything either.

I'd embed the underlying exception type in some record in AWSException.
Then you catch on AWSException, and you extract the relevant failure
type/code and act accordingly.

In this scenario, combined with catch-by-type, you'd need _only_ one
catch clause, where inside you can play with matching, if you wish.

It _could_ become more interesting if we had limited
multiple-inheritance, so that you could dynamically assign a known
signature for common error patterns (such as 'retry'). _Then_,
catch-by-type would fundamentally be more powerful (and yes, I did/do
this in python for exceptions for exactly this reason) and generating
types dynamically would be an advantage.

[samoconnor]

I assume you just retrow when exiting from the catch block?

yes.

... not much different compared to what you can do currently by hand, really.

That's true. I'm not proposing the Retry.jl macros for inclusion in Julia. I see them as a work-around while Julia's exception handling matures.
All these macros do is save a few character of syntax and a few lines of code. But, those small savings do make the code easier to follow. e.g. This example would be quite bit more verbose without the convenience macros.

@repeat 4 try 

    return http_attempt(request)

catch e

    @delay_retry if typeof(e) == UVError end

    @delay_retry if http_status(e) < 200 &&
                    http_status(e) >= 500 end

    @retry if http_status(e) in [301, 302, 307]
        request.uri = URI(headers(e)["Location"])
    end

end

I think its pretty important that the final exception handling design is validated against exception prone real world systems. e.g. something that deals with multiple external libraries / web services / control systems / etc. Something where exceptions are infrequent enough that the performance overhead of try/catch is not a problem, but where exception handling across multiple layers is a 1st class part of the program logic.

[samoconnor]

@wavexx wrote:

I'd like the exception mechanism based on type dispatch foremost, just like regular function calls, to make it more uniform.

I agree that something that has the feel of method matching based on type would be a good solution for Julia. I also think it is critical that value based matching is not a 2nd class citizen, otherwise you end up right back at _"must not forget to call rethrow()"_ if the library you are calling has a single error type with a .code field. e.g. you end up with every catch e::AcmeLibError clause being if e.code == xxx ... else rethrow(e).

What if the syntax was catch function.
The exception is caught if function returns true.

try
    ...
catch e::AccessDeniedError -> true
    ...
catch e::UVError -> e.info == "Timeout"
    ...
catch e -> e.reason == "com.acme.service.resource_busy_error"
    ...
catch e -> get_headers(e)["reason"] == "hardware failure"
    ...
end

_This assumes that missing field and method errors in the catch lambda are ignored._

[wavexx]

On 01/02/16 23:24, samoconnor wrote:

@wavexx https://github.com/wavexx wrote:

I'd like the exception mechanism based on type dispatch foremost,
just like regular function calls, to make it more uniform.

I agree that something that has the feel of method matching based on
type would be a good solution for Julia. I also think it is critical
that value based matching is not a 2nd class citizen, otherwise you end
up right back at /"must not forget to call |rethrow()|"/ if the library

Note that this is already happening.

During the Xmas vacations I cloned all the packages as listed in
pkg.julialang.org for 0.4 and 0.5 and concocted a few greps to see
roughly how exceptions are handled in the wild. Notably, I wanted to know if

1) exceptions are as narrow as possible (ie: handle the ONE failure case
they're after)
2) exceptions are rethrown correctly

with some grep -C | grep -v you can get #2 reasonably. For #1, I
inspected some cases by hand, but it's harder, so I couldn't really come
up with something general.

But the situation for #2 is bleak.
And it doesn't surprise me really.

As I wrote before, the current catch mechanism is very similar (in terms
of laxity) to R, and in R exceptions are currently a _major_ issue,
because _nobody_ (especially statisticians) seem to understand how
exceptions should be handled.

It's very common to run R jobs in a server, and see OOM/IO exceptions be
completely _ignored_ as part of try() block which just catches
everything, probably because the author intended to handle some fitting
exception only. It's so bad I _have_ to save the full output of the job
and grep into it to see problems like these. I have _zero_ confidence in
R exceptions due to that.

I remember I've seen a couple of such cases in julia's base itself when
I was working on the Polyglot package and inspecting the
multiprocess/spawn calls as a base.

you are calling has a single error type with a |.code| field. e.g. you
end up with every |catch e::AcmeLibError| clause being |if e.code == xxx
... else rethrow(e)|.

What if the syntax was |catch function|.
The exception is caught if |function| returns true.

It's an interesting suggestion. I wouldn't strictly substitute it to a
type signature, but it's interesting nonetheless. Difficult to say,
especially without trying, to see the advantages of such an approach.

[hayd]

At the moment, it seems that the speculative "chain of custody" issue is holding back the (more standard) exception handling approach (i.e. some kind of case matching). IMO We should strive for a solution that works now but can be extended later if "chain of custody" can be fleshed out.

The syntax (allow both type and boolean function) seems pretty powerful to me:

  ...
catch err::FooException
  ...
catch err::BazException -> contains(err.msg, "not found")
   ...
end

I think this solves the majority of use cases, and it's more than python's exception handling.

---

Later we could, for example, have a method that can assert more specific information e.g. the label/callsite of the exception (if this were accessible):

  ...
catch err::FooException -> thrownby(err, foo)
   ...
end

I agree with @wavexx that right now it's very easy to forget to rethrow in the default case.

[samoconnor]

I have not yet fully digested this post on Joe Duffy's Blog about the Midori Error Model, but it seems like a fairly good review of known error handling mechanisms...

http://joeduffyblog.com/2016/02/07/the-error-model/

[nalimilan]

The examples provided above look terribly similar to pattern matching à la Match.jl (see https://matchjl.readthedocs.org/en/latest/#match-types). We don't need to reinvent the wheel just for exceptions, when we already have great package for this kind of situation.

I wonder whether it would be possible to combine a general catch statement with Match.jl without even including that package into Base, and keeping catch very general as it is now. The following already works fine:

using Match
try
    throw(ArgumentError("test"))
catch err
    @match err begin
        err::ArgumentError, if contains(err.msg, "false") end => println("a")
        err::ArgumentError                                    => println("b")
        _                                                     => rethrow(err)
    end
end

That said, I don't know how to get rid of the need for the final rethrow clause without some syntax support.

[tknopp]

I fully agree with @hayd on this topic. @nalimilan: This is absolutely not reinventing the wheel. Its just implementing what is standard in other languages (Java, C#) and has proven to be effective.

[nalimilan]

@tknopp I'm not saying this isn't useful (I actually strongly believe it's a good feature), but that it could be implemented by re-using the powerful syntax from Match.jl, so that Julia is more consistent. I suspect it will be easier to get something merged if it only requires a limited amount of special syntax.

[tknopp]

Yes I know, and my point is that if the syntax is very natural and has already proven to be useful in other languages, why not simply add it instead of requiring the user to understand a more complex (though powerful) macro like @match.

The catch err::FooException is in my point of view very Julian as it tells us (similarly as in function signatures): This code will only be entered if the type is matched. Using @match feels like that function should be written in the form

function myFunction(arg)
    @match arg begin
        arg::Int             => println("a")
        arg::Float64       => println("b")
    end
end

I know this cannot be directly compared but it should illustrate why I think that language minimization is not always good.

[mauro3]

@samoconnor: thanks for the link, that was a good read. I guess Julia encourages "abandonment", by discouraging try-catch blocks. By the sounds of that post, that is excellent. I also liked the look of their try syntax, translated to Julia-like syntax:

y = try foo(x, xx) else
   .... # do something with the returned error
end

I don't have much baring on this topic, so not sure if this could work.

[hayd]

Using match is ugly above, we need some syntactic-sugar! (Or are you just saying the implementation could be similar.)

Using else would be a bad idea:

1. we already have catch!
2. else has the _opposite_ meaning in python.

[samoconnor]

@mauro3, the main thing I gleaned from my first read of the Midori Error Model is the decision to separate handling of bugs from recoverable errors.

The other stand-out (very similar to @StefanKarpinski's original "chain of custody" idea) is the requirement for a try keyword before every call to a function that could throw an exception.

Some relevant quotes...

_Quote from Retrospective and Conclusions:_

• Using contracts, assertions, and, in general, abandonment for all bugs.
• Using a slimmed down checked exceptions model for recoverable errors, with a rich type system and language syntax.

Abandonment, and the degree to which we used it, was in my opinion our biggest and most successful bet with the Error Model. We found bugs early and often, where they are easiest to diagnose and fix. Abandonment-based errors outnumbered recoverable errors by a ratio approaching 10:1, making checked exceptions rare and tolerable to the developer.

_Quote from Bugs: Abandonment, Assertions, and Contracts:_

A number of kinds of bugs in Midori might trigger abandonment:

• An incorrect cast.
• An attempt to dereference a null pointer.
• An attempt to access an array outside of its bounds.
• Divide-by-zero.
• An unintended mathematical over/underflow.
• Out-of-memory.
• Stack overflow.
• Explicit abandonment.
• Contract failures.
• Assertion failures.

_Quote from Recoverable Errors: Type-Directed Exceptions:_

Examples include:

• File I/O.
• Network I/O.
• Parsing data (e.g., a compiler parser).
• Validating user data (e.g., a web form submission).

_Quote from Language and Type System:_

void Foo() throws {
    ...
}

We stuck with just this “single failure mode” for 2-3 years. Eventually I made the controversial decision to support multiple failure modes. It wasn’t common, but the request popped up reasonably often from teammates, and the scenarios seemed legitimate and useful.

int Foo() throws FooException, BarException {
    ...
}

In a sense, then, the single throws was a shortcut for throws Exception.

_Quote from Easily Auditable Callsites:_

A callsite needs to say try: int value = try Foo();
This invokes the function Foo, propagates its error if one occurs, and assigns the return value to value otherwise.

_Quote from Syntactic Sugar:_

try {
    v = try Foo();
    // Maybe some more stuff...
}
catch (Exception e) {
    Log(e);
    rethrow;
}

Result<int> value = try Foo() else catch;
if (value.IsFailure) {
    Log(value.Exception);
    throw value.Exception;
}

int value1 = try Foo() else 42;
int value2 = try Foo() else Release.Fail(); // Abandonment

[samoconnor]

See related: #15514 RFC: Uncatchable FatalException. Separating bugs from recoverable errors

[simonbyrne]

Another point of reference: the not-yet-released Genus language has a paper on how they handle exceptions, which seems similar to Stefan's proposal:
http://www.cs.cornell.edu/andru/papers/exceptions/exceptions-pldi16.pdf

[oscardssmith]

What if we used stack-traces to allow this feature on the catch side. Specifically, you could catch AbstractArray.IndexError e which would look at the error, and only catch it if it came from a specific place. This way, if you only want to catch your errors, you could just use catch myModule.Error but if you wanted to catch any errors, you could catch the specific error type?

[StefanKarpinski]

At the very least, having a throws annotation on expressions would help narrow the scope of where and what kind of error can be caught in a block of code where some expressions are expected to throw while others aren't. E.g., I found myself writing the following today:

x = try f() catch
    warn("mumble")
    nothing
end
if x != nothing
    g(x)
end

This is a weird control flow, and it would have been much more natural to write this:

try
    g(f())
catch
    warn("mumble")
end

However, g might throw, which I do not want to catch. I could filter on the error thrown by f, but in this case it was ArgumentError which is extremely common and could quite possibly be thrown by g due to some unexpected condition. With a throws annotation, this would be the following instead:

try
    g(f() throws ArgumentError)
catch e::ArgumentError
    warn("mumble")
end

This syntax could be mechanically transformed into code that catches an argument error thrown by f but which passes through any error thrown by g as well as any other kind of error thrown by f. This doesn't address the problem that f might throw an ArgumentError in some unexpected way, but at least it would allow safely broadening the scope of a try block when convenient, allowing the error handling code to be moved further out of the way of code doing "real work".

[StefanKarpinski]

Implementation idea for the full feature. When this code occurs:

function f(x)
    check(x) || throw ArgumentError("$x is very, very bad")
end

there would be a hidden bit-mask for each exception location in the body of the function (so, I guess we would have a limit of 64 exception locations in a given function, which seems acceptable), indicating whether each location is expected in the current context or not. The lowering would then be something like this:

function f(x)
    check(x) || begin
        e = ArgumentError("$x is very, very bad")
        throw(catchable[1] ? e : RuntimeError(e))
    end
end

Here RuntimeError is an error type that wraps another error type, preventing it from being caught by catch e::ArgumentError. Of course, if one wants to catch any RuntimeError then one could catch those. Old-style "catchall" catch clauses would automatically catch and unwrap RuntimeErrors, which would leave legacy code working as it did previously. Aside from passing the catchable mask around, I don't think any other changes would be required. The implicit catchable argument would only be needed when calling functions that actually throw errors – when calling a method that has no throw statements, it's not necessary.

When chaining functions catchable functions, for efficiency, one needs to translate shared slots from the caller's mask to the callee's mask. One probably wants to arrange for their callable masks to line up as much as possible for efficiency, so that this can be done with an integer &. Example:

function f()
    c1() && throw FooError()
    g() throws BarError
    c2() && throw BarError()
end

function g()
    c3() && throw BarError()
    c4() && throw BazError()
end

If one calls f() throws BarError, BazError then the callable mask for f would be set by in the caller with both BarError and BazError on. Since g throws both BarError and BazError, the callsite of g in f would want to & f's incoming mask with a static mask with only BarError and BazError set on. That way g gets a mask value with BarError on if and only if f got a mask with it on (and BazError always off since g doesn't expect that exception from f). The tricky part here would be making sure that potential caller and callee methods have compatible notions of what the mask slots mean. This problem feels somewhat similar to register coloring. Note that generating code to permute compatibility masks between caller and callee is always an option if strict compatibility can't be accomplished.

[PallHaraldsson]

@StefanKarpinski links here from HN with: "it can be done in the 1.x timeframe instead of needing to wait until 2.0."

[c42f]

The implicit catchable argument would only be needed when calling functions that actually throw errors – when calling a method that has no throw statements, it's not necessary.

So does this imply that bar( ) throws Bar implicitly calls bar with a different calling convention rather than simply calling bar()? That is, a lowering such as

# On the throw side
function bar(x)
    throw Bar(x)
end

# lowering to
function bar_catchable(__catchable__, x)
    e = Bar(x)
    throw(__catchable__[1] ? e : RuntimeError(e))
end
bar(x) = bar_catchable(0, x)

# On the catch side
bar(1) throws Bar
# lowering to
bar_catchable(__somehow_make_magic_catchable_bitmask(Bar), 1)

Hmm. Though my sketch here isn't coherent when bar has many methods, not all of which may throw. Having a different calling convention would suggest that "may throw" is a property of the function but that seems weird in a dynamic language like Julia. Having written that, it occurs to me that the chain of custody smells rather like contextual dispatch; maybe it could be prototyped with Cassette.

Overall I think the chain of custody idea is a piece of the puzzle but I do feel it would be incomplete to implement as-is without thinking about how it interacts with task abandonment (fatal errors / panics) in the spirit of Rust / Swift / Midori (#15514 etc), and also to learn about how Go 2.0 is planning to do error handling. For context, I'm re-reading this thread while thinking about concurrency and cancellation (#33248).

More generally, there's already a strong convention in Julia that "expected errors" should be returned as values which can be checked, but we don't have a uniform API convention for how that should be done. I wonder whether we should have such a thing, and whether it interacts with what's been discussed here.

[tkf]

More generally, there's already a strong convention in Julia that "expected errors" should be returned as values which can be checked, but we don't have a uniform API convention for how that should be done.

There are already a few implementations:

• https://github.com/iamed2/ResultTypes.jl
• https://github.com/KristofferC/Expect.jl

The disadvantage is that we loose stack trace by using this approach (and the flip side is that run-time cost is lower). But maybe there can be a slow exception-tracing mode of execution (for debugging)? This also sounds like implementable using Cassette.

[c42f]

What if a key goal of "chain of custody" was that it sidesteps the usual unwinding mechanism completely, effectively turning exceptions into error objects?

This seems somewhat possible if you view the chain of custody as changing the calling convention. It's kind of like the colored functions idea... but the colors are more the compilers business and don't prevent a user from calling the function either way.

[c42f]

With that idea, you'd probably want some nice syntactic sugar to cover a few cases similar to what Swift does:

• Catch a given error type and return nothing / substitute with the result of some code (Swift try?)
• Catch a given error type and convert to a fatal exception, if we want that to be a thing (Swift's try!)

[tkf]

What if a key goal of "chain of custody" was that it sidesteps the usual unwinding mechanism completely, effectively turning exceptions into error objects?

Sounds like a great feature to have. I guess it fits well with how other things work Julia; start with somewhat slow but correct and simple implementation and then add annotations to make things robust and fast (e.g., @inbounds/@propagate_inbounds).

[c42f]

Yes I think it would fit nicely. But it's not exactly about making the slow->fast / safe->unsafe tradeoff. With this approach we may get both correctness and speed:

• Correctness because only a named set of errors would be caught this way (the others propagate as normal)
• Speed, because exception matching happens at the throw site and can thus avoid some of the costly and non-optional runtime work like collecting backtraces.

[tkf]

Right, it's more like speed-usability tradeoff rather than speed-safety tradeoff. I still think this is a tradeoff since we loose exactly where the exception is thrown with this throw-by-return approach.

[c42f]

Fair enough. But I think it's a great tradeoff to make because in the case where you have the chain of custody in place you already know exactly where to expect the error from and you've got some cleanup code ready to go — there's very little reason to want a backtrace in that case. If you do want the backtrace you can use a normal try-catch without the chain of custody.

In this scheme it becomes clearer that certain errors really shouldn't be catchable. For example StackOverflowError is triggered internally in a signal handler and cannot take part in this mechanism. But that's ok! Nobody should be expecting a stack overflow; that's a sure sign of a bug and not something that should be papered-over.

[tkf]

in the case where you have the chain of custody in place you already know exactly where to expect the error from

I thought the chain of custody is a tree, rather than a linear chain. For example, consider:

foo1() = rand() > 0.5 ? (throw BazError) : 1
foo2() = rand() > 0.5 ? (throw BazError) : 2

function bar()
    x = foo1() throws BazError
    y = foo2() throws BazError
    return x + y
end

bar()

Then I suppose it is impossible to know if it is foo1 or foo2 that threw the error?

[c42f]

Agreed, it's a tree, and you won't get perfect information about how the error was thrown internally within bar(). But this is very similar to return codes and I wouldn't say it's considered a disadvantage in that case. Arguably it's an advantage which makes the scheme more composable because the callers of bar() don't get to peer inside the implementation.

Another thought about that: if callers do want to distinguish between a failure in foo1 vs foo2, there must be something semantically different about those failures, in which case a different exception type may be warranted. Alternatively we could possibly consider matching on exception values rather than types. I do feel like pattern matching of values (eg Haskell or Elixir — like) lends itself quite naturally to handling exceptions; in many ways more naturally than a simple type check.

[tkf]

I probably should have said debuggability rather than usability. I don't think the expressiveness is the problem. I was trying to point out that stack trace would contain less information and so impeding fast debug. Something like post-mortem debugger would also be impossible with this approach.

[c42f]

Oh sorry, I think I see what you're getting at. The point is that the chain of custody is broken "missing a link" because at the top level bar() is called without being annotated?

Interesting. In that case I guess there's a design decision to make about where the rethrow happens but the most obvious would be that the error is rethrown and appear to come from line 1 or 2 of bar(). That seems somewhat sufficient, as the rest of the chain of custody is intact and manifest in the source code?

[tkf]

On no, I don't think it's broken. I just think throw-by-return is harder to debug. I still think it's a nice feature to have. But I don't think it's entirely free (i.e., there is a speed-debuggability tradeoff).

That seems somewhat sufficient, as the rest of the chain of custody is intact and manifest in the source code?

My first example was the simplest case where the chains were short. But I don't think it is possible to recover the full trace in general. Consider

foo1() = rand() > 0.5 ? (throw BazError) : 1
foo2() = rand() > 0.5 ? (throw BazError) : 2
foo3() = rand() > 0.5 ? (foo1() throws BazError) : (foo2() throws BazError)
foo4() = rand() > 0.5 ? (foo1() throws BazError) : (foo2() throws BazError)

function bar2()
    x = foo3() throws BazError
    y = foo4() throws BazError
    return x + y
end

I don't think the line number of bar2 would tell you the origin of the error accurately.

[tkf]

@c42f I think another relation of the chain of custody to the structured concurrency is the multi-exception handling. Should throws support CompositeException?

function baz()
    (@sync begin
        @spawn foo() throws FooError
        @spawn bar() throws BarError
    end) throws FooError BarError  # ???
end

qux() = baz() throws FooError BarError  # ???

My guess is that the answer is "no" and the function using @sync should manually re-throw well-typed exceptions (and it's better to discourage using throws CompositeException).

[c42f]

the function using @sync should manually re-throw well-typed exceptions

Agreed, I think CompositeException should be viewed more as a transport mechanism than something you ever want to match on. That goes for most of the exceptions which wrap exceptions; they don't have much meaning in their own right and you typically want to unwrap to figure out what really went wrong. An ideal system would have a way to automatically unwrap some of these things.

But this observation about "unwrapping to get at the real error" is not even limited to these cases. Consider SystemError; it comes with an error code and it might be this which you want to match on to determine the difference between an "expected error" and something which should continue propagation.

I feel like the way to deal with this is some kind of pattern matching which constructs a predicate from an expression like SystemError(_, FILE_NOT_FOUND, _) (similar to Match.jl or other such systems). If we could pass such a predicate up the call stack, this can communicate precisely which errors are "expected" to the throw site and intercept them before they've been thrown. With that approach the compiler even has a chance of optimizing away parts of the construction of the exception itself if only the matched parts of the structure are returned. For example, in the pattern above we discard the first argument to SystemError which is an error message string which might be expensive to construct.

To expand this suggestion in rough made up syntax

try
    foo(filename) throws IOError
catch @match SystemError(_, FILE_NOT_FOUND, @var(extrainfo))
    # Only get here if a `SystemError` with FILE_NOT_FOUND error code
    # *would have* been thrown inside `foo`.
    # The variable `extrainfo` now contains the value it would have had if
    # the exception actually been thrown.
    println("File not found (extra info $extrainfo)")
end

I'll admit there's a tension here in matching internal structure of types which is often considered an implementation detail in Julia. A rather speculative solution to that could be to make the suggested throw keyword take key value pairs a bit like @info, with pattern matching somehow based on those rather than the internal structure of the exception object.

[tkf]

In terms of compilation, I guess it is not beneficial to have value-based matching in catch? If so, maybe just allow arbitrary Julia expression as the predicate? For example, how about

try
    ...
catch err::ExceptionType if PREDICATE
    ...
end

The reason to prefer this over if-else blocks inside catch is that the if-else approach requires users to not forget to put else rethrow() at the end. This is a boilerplate and can easily introduce bugs.

This also lets you do the match based on information other than exception

try
    foo(filename) throws SystemError
catch err::SystemError if endswith(filename, ".jl") && err.errnum == FILE_NOT_FOUND
    println("Julia file not found (extra info $(err.extrainfo))")
end

Going to this direction, it would be nice to have some kind of uniform public accessor/query API. One simple API may be dueto(err, reason) :: Bool which can be used as

dueto(err::CompositeException, FooError)
dueto(err::SystemError, FILE_NOT_FOUND)

But defining a nice full API for CompositeException sounds very hard (just reading https://github.com/python-trio/trio/issues/611)....

[c42f]

In terms of compilation, I guess it is not beneficial to have value-based matching in catch?

From the point of view of the compiler the matching function is just a closure and this doesn't seem very different from the many other places where we currently pass closures around and specialize the called function based on the type of the closure. reduce for example :-)

So I think the general idea is harmonious with the other features we already use in the language and may not be super hard on the compiler, provided that the chain of custody for errors isn't deep on average (cf. https://github.com/JuliaLang/julia/issues/7026#issuecomment-52342673). (The amount of specialization can also be tuned using compiler heuristics, as usual.)

The reason I like this idea is that it unlocks some optimization opportunities which are more or less impossible in normal exception systems. Usually it's a choice between:

• Be slow but explanatory: Create a nice formatted error message with lots of metadata so users can figure out what broke.
• Be fast but opaque: Return some integer error code or equivalent.

The decision about which to use should belong to the caller, but typically it's the callee which makes this decision. (In rare circumstances the callee provides an option to select one or the other. For example the raise option to Meta.parse.)

What I'm suggesting here is a general mechanism which allows every throwing function to benefit from the efficiencies of having a flag like raise. But without any extra effort from the programmer.

[tkf]

From the point of view of the compiler the matching function is just a closure

What I meant to ask was if it makes sense to have "structured" match expression (e.g., @match ...) rather than allowing any expression in order to help the compiler. But your comment answered my question.

a general mechanism which allows every throwing function to benefit from the efficiencies of having a flag like raise

This would be great :+1:

[c42f]

Right, from the perspective of the compiler the use of @match (or whatever) is mostly irrelevant; that's just one possible front end for generating a closure to destructure the exception object and figure out whether to throw-by-return vs throw by the usual mechanism (currently longjmp).

[c42f]

I tried implementing a prototype of the exception-matcher-as-closure idea and it seems very promising. However a full prototype seems to require language support because we need the closure to not participate in dispatch (though we very much do want it to participate in specialization). The issues are almost the same as for keyword argument dispatch and Jeff's proposed solution at https://github.com/JuliaLang/julia/issues/9498#issuecomment-316144694 applies in a very similar way.

That makes me think it would be useful to have generic runtime support for arguments which are ignored during dispatch, which have some rule for "filling them in" when they're not specified, and some way to pass them implicitly.

Another case where something like this might be useful is macro calls and the magic __module__ argument. Currently this is implemented with special lowering, but that produces confusing method errors like no method matching @foo(::LineNumberNode, ::Module) where the implicit arguments are exposed to the user. Of course this could be patched up, but the implicit arguments don't need to participate in dispatch and could go via a custom calling convention instead.

[non-Jedi]

As a potential source of inspiration, one piece of prior art I don't see mentioned here is Pony's error handling. On the surface it smells very similar to this proposal.

Basically, any time a function can throw an error it becomes a "partial function" which must be marked with a ? both at the function declaration and at the call-site. Partial functions can only be called within other partial functions or within a try block. From my brief experience with Pony I found this a pleasant enough way of handling exceptions as it forces each function that can throw to explicitly document all conditions under which it can throw.

Of course, Pony without multi-methods that can be extended by users occupies a very different space in language-design than Julia, but I still thought the similarities with this proposal were striking.

[bcasselsND]

I haven't read through all of the discussion here. But I would encourage people to look at the Common Lisp condition system for inspiration. Dan Weinreb designed a system that covers a lot of useful cases.

Here's a good discussion:
http://www.gigamonkeys.com/book/beyond-exception-handling-conditions-and-restarts.html

[bcasselsND]

Of course, the Common Lisp ideas were carried over to Dylan's exception handling. But I would look at Lunar. It looks like Dave Moon (of Common Lisp and Dylan worlds, among many other accomplishments) has distilled this down and moved it firmly to the generic function world, in his proposed Lunar language:
http://users.rcn.com/david-moon/Lunar/exception_handling.html

[c42f]

Beautiful, thanks for these links. I agree the common lisp condition system is quite interesting. And it's particularly interesting to have the modern take on it in that Lunar writeup.

There was some very related discussion of algebraic effect systems over at https://github.com/JuliaLang/julia/issues/33248#issuecomment-570092961 (see also the following discussion). Though for a dynamic language like Julia the common lisp condition system seems a better analogy than the algebraic effects systems in strongly typed functional languages.

I had a quick look at the Lunar writeup (will need an in-depth read later). I think it's exactly the kind of thing which would fit in well in Julia; in fact it's almost exactly the design I had in mind after reading about algebraic effects, including the behavior of the chain of catch handlers. In particular, the following paragraph about Lunar was very close to what I had in mind:

The throw function (when called with one actual parameter) executes a method that was made dynamically available by catch. The method is not selected by the usual rules; instead throw invokes the first applicable method in the sequence of available catcher methods, with no consideration of method specificity or ambiguity.

(For people following along, note that Lunar's throw is very different from Julia's throw. In the language of algebraic effects, Lunar's throw would be the perform keyword mentioned at https://overreacted.io/algebraic-effects-for-the-rest-of-us/)

I'd kind of like to bundle all these things which provide restarts under the label "effects systems" and I have a half-written summary about this and how it could be applied in Julia. Actually I feel all this is possible to prototype already — indeed @MikeInnes has prototyped https://github.com/MikeInnes/Effects.jl which is super cool and very much relevant.

The really difficult thing is figuring out how to fit this stuff into the runtime so that it becomes a reliable language feature which doesn't stress the compiler too much and which also generates really fast code when it matters. I feel like we'll only truly succeed with an effects system if the runtime can generate code which

1. Would be as fast as returning an integer status code by hand and testing that for shallow call stacks where "failure is expected" and dealt with immediately.
2. Can support unwinding from deeply nested function calls without specializing the intermediate call stack on the effects handlers.

If we don't succeed at (1), people will continue to need return codes; if we don't succeed at (2) the compilation time will (presumably) be unacceptable. But if we can do both things, we may get one consistent error handling strategy everywhere.