Language: Add checked exceptions.

[bsutton]

It has to be said...

The one thing I miss moving from Java to Dart is the lack of checked exceptions.

It makes it really hard to put complete error handling mechanisms in place if you don't know the full set of exceptions that a method can throw.

I find it rather ironic that dart doesn't have checked exceptions whilst 'Effective Dart' lints generate an error essentially saying 'check your exceptions'.

try {
      Directory(path).createSync(recursive: recursive);
    } 
    catch (e) {
      throw CreateDirException(
          'Unable to create the directory ${absolute(path)}. Error: ${e}');
    }

generates the following lint:
Avoid catches without on clauses.

To fix this lint I need to read source code to discover the exceptions that are going to be thrown.

Exceptions were created to stop people ignoring error.
Unchecked exceptions encourages people to ignore errors.

Have we learnt nothing?

I know this probably won't get anywhere due to the current religious movement against checked exceptions and the disruption to the dart eco system but this implementation decision was simply a bad idea.

[julemand101]

I do agree but you should properly move this issue to the language project: https://github.com/dart-lang/language/issues since the language itself does not support checked exceptions.

[lrhn]

Most languages created after Java has learned from Java and not introduced checked exceptions.
They sound great in theory, but in practice they are (rightfully or not) vilified by the people having to maintain the throws clauses. I think the general opinion is that it's technically a good feature, but it doesn't carry its own weight.

I don't see Dart going that way at the current time.

To fix this lint I need to read source code to discover the exceptions that are going to be thrown.

You can write on Object catch (e) if you just want to satisfy the lint. Or you can drop the lint.
If you actually want to catch the exception being thrown (and you should when it's an exception, not an Error), then the method documentation should document it clearly.

The standard way to document exceptions is a paragraph starting with "Throws ...". I admit that not all code follows that standard, and dart:io suffers from being written before most standards were formed.

[bsutton]

I admit that not all code follows that standard

And this is exactly the problem.

I would argue that checked exceptions more than carry their weight.
With modern IDE's the overhead of managing checked exceptions is largely automated.

The lesson that doesn't seem to have been learned is that developers are lazy and inconsistent (myself included) and if we don't force them to directly address errors then they simply ignore them and the quality of software suffers as a result.

I use a fair amount of flutter plugins and largely there is simply no documentation on what errors can be generated.
Dart actually makes it harder to document errors as a developer now actually has to write documentation.
Checked errors force devs to document their errors and actually make it easier as the IDE inserts them.

The end result is that for a large chunk of the code base we use on a daily basis, errors are simply not documented and we have to do additional testing and investigation to determine what errors need to be dealt with.

In the business world they have the concept of 'opportunity cost' which is essentially if I invest in 'A', I can't invest in 'B'. What is the cost not doing 'B'? That is the opportunity cost.

With unchecked exceptions we wear the cost of maintaining the checked exceptions but we don't have to wear the cost of investigating what errors can be generated nor the debugging time spent because we didn't handle an exception in the first place.

If a library maintainer has to spend time to declare the checked exceptions that time is more than offset by developers that use that library not having to investigate/test for what errors will be generated.

I believe the backlash against checked exception is because most developers prefer to ignore errors and checked exceptions require them to manage them.

[munificent]

I think if a function's failure values are so important to be handled that you want static checking for them, then they should part of the function's return type and not an exception. If you use sum types or some other mechanism to plumb both success and failure values through the normal return mechanism of the function, then you get all of the nice static checking you want from checked exceptions.

More pragmatically, it's not clear to me how checked exceptions interact with higher-order functions. If I pass a callback to List.map() that can throw an exception, how does that fact get propagated through the type of map() to the surrounding caller?

You describe the movement away from checked exceptions as "religious", but I think that's an easy adjective to grab when you disagree with the majority. Is there a "religious movement" towards hand washing right now, or is it actually that the majority is right and hand washing is objectively a good idea? If almost everyone doesn't like checked exceptions, that seems like good data that it's probably not a good feature.

most developers prefer to ignore errors and checked exceptions require them to manage them.

You're probably right. But if you assume most developers are reasonable people, then that implies that it should be relatively easy to ignore errors. If it harmed them to do it, they wouldn't do it. Obviously, individuals make dumb short decisions all the time, but at scale I think you have to assume the software industry is smart enough to not adopt practices that cause themselves massive suffering.

[bsutton]

"I think if a function's failure values are so important to be handled that you want static checking for them, then they should part of the function's return type and not an exception."

You must be younger than me :)

Exceptions were introduced to solve the failings of return types.
1) having to pass return types up the call stack.
2) using the return to indicate error conditions means that you can't use
the return type to pass the 'good path' values.
3) poor documentation
4) devs choosing to ignore error codes.

Before checked exceptions, ignoring error returns was the norm not the
exception (pun possibly intended).

I'm involved in the usage and maintenance of a number of open source dart
packages and I'm seeing the same problem again.
Code that ignores errors is almost the norm rather than the exception.
A complete lack of documentation on the errors that methods return.

I've been adding effective dart lints to a number of packages lately and
one of the lints is to use an 'on' clause in the catch block.
I've mostly had to suppress the lint as I would have literally had to read
through thousands of lines of code to find out what exceptions can be
thrown.
I never have this problem with java. I can always work out exactly what
exceptions I need to deal with and if you read my code you can easily
determine what error conditions you have to handle.

You identify a religion by a pattern of behaviour that defies logic.
The lack of checked exceptions has resulted in a proliferation of packages
that poorly handle errors and applications that do the same.

Do a survey of pub.dev and the evidence is all over the place.

Google seems to take a fairly statistical approach to language changes.

I would advocate that Google do an analysis on the level of documentation
for errors in pub.dev.

I suspect the stats will be damning.

Brett

On Fri, 29 May 2020 at 09:47, Bob Nystrom [e-mail] wrote:

I think if a function's failure values are so important to be handled that
you want static checking for them, then they should part of the function's return
type and not an exception. If you use sum types or some other mechanism
to plumb both success and failure values through the normal return
mechanism of the function, then you get all of the nice static checking you
want from checked exceptions.

More pragmatically, it's not clear to me how checked exceptions interact
with higher-order functions. If I pass a callback to List.map() that can
throw an exception, how does that fact get propagated through the type of
map() to the surrounding caller?

You describe the movement away from checked exceptions as "religious", but
I think that's an easy adjective to grab when you disagree with the
majority. Is there a "religious movement" towards hand washing right now,
or is it actually that the majority is right and hand washing is
objectively a good idea? If almost everyone doesn't like checked
exceptions, that seems like good data that it's probably not a good feature.

most developers prefer to ignore errors and checked exceptions require
them to manage them.

You're probably right. But if you assume most developers are reasonable
people, then that implies that it should be relatively easy to ignore
errors. If it harmed them to do it, they wouldn't do it. Obviously,
individuals make dumb short decisions all the time, but at scale I think
you have to assume the software industry is smart enough to not adopt
practices that cause themselves massive suffering.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dart-lang/language/issues/984#issuecomment-635675501,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAG32OGKTAU2HOQKVSMIPBLRT3ZZVANCNFSM4NKANDJQ
.

[bsutton]

Just a final thought.

Why would you move from a system (checked exceptions) which automates the documentation of code to one that requires developers to document their code.

The empirical evidence (again look at pub.dev) is that developers do a terrible job of documenting and view documentation as a burden.
Managing checked exceptions is a far smaller burden than documenting code.

The lack of checked exception really has to be the best example of developers shooting themselves in the foot :)

[leafpetersen]

Why would you move from a system (checked exceptions) which automates the documentation of code to one that requires developers to document their code.

My general take on this is that effect type systems (of which checked exceptions is one) tend to interact badly with higher-order code, and modern languages including Dart have leaned in heavily to using first class functions in libraries etc. Indeed, the first search result I got when I just looked to see what Java does with this these days was this, which isn't inspiring. There may be better technology for making this non-clunky now, but last I looked at it, it took some pretty hairy technology to be able to express higher-order functions that were parametric in the set of exceptions that could be thrown. Inference can help with that, but as the saying goes, now you have two problems... :)

The empirical evidence (again look at pub.dev) is that developers do a terrible job of documenting and view documentation as a burden.

I will admit that it personally bothers me a lot that even for well-documented Dart team owned code, I often have to go poke around in the implementation to figure out whether an exception can be thrown, and in what circumstances.

[rrousselGit]

Exceptions were introduced to solve the failings of return types.
1) having to pass return types up the call stack.
2) using the return to indicate error conditions means that you can't use
the return type to pass the 'good path' values.
3) poor documentation
4) devs choosing to ignore error codes.

Exceptions are not the only way to solve these.
There are multiple languages out there with no exception mechanism at all and that do just fine.

The alternative is usually a combination of union-types and tuples and destructuring. This leads to self-documenting code, where it's impossible to ignore errors, while still being able to return valid values

For example using unions, instead of:

int divide(int value, int by) {
  if (by == 0) {
    throw IntegerDivisionByZeroException();
  }
  return value / by;
}

void main() {
  try {
    print(divide(42, 2));
  } on IntegerDivisionByZeroException catch (err) {
    print('oops $err');
  }
}

we'd have:

IntegerDivisionByZeroException | int divide(int value, int by) {
  if (by == 0) {
    return IntegerDivisionByZeroException();
  }
  return value / by;
}

void main() {
  switch (divide(42, 2)) {
    case IntegerDivisionByZeroException (error) => print('oops $error'),
    case int (value) => print(value);
  }
}

[bsutton]

I'm not certain unions result in more readable code than catch blocks.
If I'm reading the code correctly it does provide a level of documentation but it appears that it will still allow the caller to ignore the error and fail to pass it back up.
So once again we are dependant on the developer to do the correct thing and we are stuck with undocumented code.

Tuples will have the same issues.

[bsutton]

My general take on this is that effect type systems (of which checked exceptions is one) tend to interact badly with higher-order code, and modern languages including Dart have leaned in heavily to using first class functions in libraries etc

This appears to be a broader problem with how do you handle errors in lambda etc.
Whether its a checked/unchecked exception, error, union or tuple you still need to handle the errors in the top level function.
Error returns simply allow you to (incorrectly) ignore the error.
Your code looks nice, but does it actually behaviour correctly?

[rrousselGit]

Error returns simply allow you to (incorrectly) ignore the error.

That is not the case.
Unions forces you to check all possible cases, or it is otherwise a compilation error.

Continuing with the code I gave previously, it would be impossible to write:

int value = divide(42, 2); // IntegerDivisionByZeroException is not assignable to int

You would have to either cast the value or check the IntegerDivisionByZeroException case.

This is in a way non-nullable types, but broadened to apply to more use cases

[bsutton]

OK, sure.

There however seems little point to introducing a new mechanism when we already have exceptions in the language.
We could possible even start by requiring checked exceptions via a lint.

We need some language experts to comment on this.

[leafpetersen]

We need some language experts to comment on this.

Touché. I'm afraid we're the best you're going to get though - Google can't afford to hire better.

[bsutton]

Sorry, no insult was intended:)

On Sat, 30 May 2020, 11:13 am Leaf Petersen, [e-mail]
wrote:

We need some language experts to comment on this.

Touché. I'm afraid we're the best you're going to get though - Google
can't afford to hire better.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dart-lang/language/issues/984#issuecomment-636252762,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAG32OE6BNZ7C5ZWAU4DZ5TRUBMUNANCNFSM4NKANDJQ
.

[bsutton]

So I come back to my evidenced based approach for this feature:

I would advocate that Google do an analysis on the level of documentation
for errors in pub.dev.

If it turns out that errors are well documented then I will withdraw my feature request.

Let's now find out if decisions on checked exceptions are driven by religion or science.

[munificent]

So I come back to my evidenced based approach for this feature:

I think the clearest, easiest-to-acquire evidence is that almost no users are requesting this feature despite ample evidence of its existence thanks to Java. Given that, I think the burden of proof is on those requesting it to show that it's a good idea. If the language team had to actively disprove every feature request, we'd never make any progress. :)

[bsutton]

So what would be an acceptable metric?

[munificent]

Pragmatically, I can't see what metric would be persuasive here. Even if we were all 100% sold on checked exceptions, adding them would be a massively breaking change to the language. We have too many users and too many lines of code in the wild. In order to justify a migration of that scale, we'd want to see hordes of users beating down the door demanding it (which is what we do see, for example, for non-nullable types). I just don't see the user demand for this, at all. And, ultimately, we are obliged to make a thing that users want.

[bsutton]

@munificent yes I do see that as a problem.

The 'somewhat vague' thought was this could be introduce via a lint so like types it is optional.

Sometimes you need to give people what they need rather than what the want :)

[leafpetersen]

Pragmatically, I can't see what metric would be persuasive here. Even if we were all 100% sold on checked exceptions, adding them would be a massively breaking change to the language

In addition, I'd also add that it's not enough to have a metric that says there is a problem. You also need to be able to argue that a proposed solution is a good one. My objection (and I think most of the objections above) is not to the characterization of the problem, but to the solution. I think that checked exceptions are not a good solution to the problem, for reasons that I touched on above, and that is one of the reasons they have not become more widespread outside of Java. These kind of effect systems don't easily compose with existing language features (like generics and higher order functions). While encoding errors into return types can be a bit heavy, it has the great advantage that it is an approach which composes well with already existing (and otherwise useful) language features.

[bsutton]

Encoding errors into return types has already been proven not to work which is why C++ had exceptions to fix the problem of C.

Encoding errors into return types simply results in users ignoring the errors which is the problem that dart has now due to the lack of checked exceptions.

I would argue that check exceptions is the best solution we have to the problem. Its just that developers are inherently lazy and don't like being forced to deal with errors.

The pub.dev code base demonstrates this in spades.

Whilst we do need to listen to the user base, we also need to ensure the health of the eco system and sometimes that means administering some medicine, even if it tastes awful.

[leafpetersen]

Encoding errors into return types has already been proven not to work which is why C++ had exceptions to fix the problem of C.

There are many more modern languages which use this approach, either in it's monadic formulation, or directly, than which use checked exceptions, so I'm going to have to disagree with your characterization of the relative success of these approaches.

That said, I don't claim it's a wonderful solution. My take on the space is as follows:

• There is a problem (though not clear how large of a problem)
• There is a bad, and very expensive solution (Checked Exceptions)
• There is a not great, but almost entirely free solution (encoding your errors in abstract return values)

I understand that the "not great" solution doesn't help enforce hygiene for the ecosystem, which is why I say it's not great.

I would argue that check exceptions is the best solution we have to the problem.

That may be, but it is still a bad solution. As far as I can tell, the best advice out there for working with higher order functions in Java is to take every single lambda, wrap its body in a try/catch which catches the checked exceptions that may be thrown in it, and rethrow an unchecked exception. Am I wrong? That helps nobody, and makes for a completely unusable language.

[bsutton]

As far as I can tell, the best advice out there for working with higher order functions in Java is to take every single lambda, wrap its body in a try/catch which catches the checked exceptions that may be thrown in it, and rethrow an unchecked exception. Am I wrong? That helps nobody, and makes for a completely unusable language.

This is the crux of the problem.

You are essentially arguing that error handling makes the code look ugly so you would prefer to ignore the errors.

Whether the errors are a checked exceptions or a return type, if an error is going to be returned the lambda MUST handle the error.

Ignoring errors to make the code look nice is not an acceptable argument.

I too would prefer a more elegant solution to handling errors in lambda's but you are blaming the messenger (checked exceptions) for the problem.

The issues is really with lambdas not with checked exceptions.

Perhaps we need to look at how futures handle exceptions and whether some similar concepts could be applied to lambdas.

[leafpetersen]

You are essentially arguing that error handling makes the code look ugly so you would prefer to ignore the errors.

Whether the errors are a checked exceptions or a return type, if an error is going to be returned the lambda MUST handle the error.

No! You are deeply misunderstanding the problem, and this gets at the crux of it! :)

The lambda is not the correct place to handle the problem. The correct place to handle the problem is higher up the stack. This can be clearly seen in the advice I quote above (which really is all over the internet - look for it)! The advice is not catch the error and deal with it the error is catch the error and throw an unchecked exception because the lambda is the wrong place to deal with the error. So the result is you get neither safety (your lambdas just catch and discard errors) nor brevity (you must explicitly catch and discard errors).

To make this actually work properly, you implement a full effect system, which lets your higher-order functions be parametric over the implied effects of their arguments. So your .map method is parametric over the set of exceptions possibly thrown by its argument. This works out ok, sort of, but it's terrible to work with, and an extremely heavyweight feature.

So you instead you get the mess that is Java checked exceptions, which, as I say, doesn't compose with higher order functions.

If you encode errors in the return type (e.g. using monads), then it's entirely up to the programmer where to handle the error. The lambda simply gets type int -> OrError(int) instead of int -> int, mapping gives you an Iterable<OrError(int)> and now to get at the contents, you need to explicitly iterate through either discarding the errors (nothing stops you from hitting yourself if you really want to), accumulating them (e.g. with a monadic fold), or handling them. But the point is that the programmer can choose at what level of the stack to handle the error.

I'm not saying it's ideal (it's not!) but it is strictly better than checked exceptions as implemented in Java, and it is entirely expressible in Dart as it exists today.

Perhaps we need to look at how futures handle exceptions and whether some similar concepts could be applied to lambdas.

Yes! We do need to! And ... it's... a ... wait ... for ... it.... monad! Futures are a monad, and they reify exceptions into the monadic value, so that the exception can be handled (or not handled) by the end user by binding the error and handling, rather than throwing an asynchronous exception from somewhere deep in the bowels of the runtime system (usually).

[simophin]

Checked exception:

1. Is infectious, from bottom to top;
2. Because its infectiousness, developers will be encouraged to either:
   a. Swallow the error.
   b. Convert it to RuntimeException.

Both of which defeat the purpose of a checked
exception. These actions taken by developers are not rare, you see it everywhere! What ironic it is: that a feature designed to help write better code, will encourage you to ignore it?

If you talk about documentation purpose, surely a union type should be sufficient?

[OlegAlexander]

An excellent discussion so far. I agree with @leafpetersen that adding checked exceptions to the language may not be the best solution to this problem because as @simophin pointed out, checked exceptions will not magically turn bad programmers into good ones. However, there _is_ a problem here, especially when it comes to the documentation of exceptions. As you will see, the problem could be at least partially solved with some changes to dartdoc.

I've recently run into this issue with the glob package. Before I go on, I should say that glob is an excellent package! I'm only using it as an example to discuss potential improvements to the documentation of exceptions in Dart.

This line will throw a StringScannerException defined in the string_scanner package, which is a dependency of glob.

final invalidPattern = '';
final glob = Glob(invalidPattern); // Throws StringScannerException 

The 1.2.0 Glob constructor documentation doesn't mention the StringScannerException. Okay. But what if I wanted to document that _my_ function, which uses Glob may throw a StringScannerException? Simply saying /// Throws [StringScannerException] at the top of my function doesn't work, because it seems dartdoc can only link to identifiers that are visible in the current file. So I have to first add string_scanner as a dependency in my pubspec.yaml and then import 'package:string_scanner/string_scanner.dart'; into the current file. Now dartdoc will recognize the link to [StringScannerException].

Having to directly depend on a transitive dependency for documentation purposes is not ideal. Also, a package you depend on may add/remove exceptions without warning, which may make your own documentation (and code!) out of date.

It would be better if dartdoc could somehow automatically enumerate all the possible exceptions a function may throw (except for the ubiquitous ones, like OutOfMemoryError.) In other words, the exceptions should be a part of the function signature, but this doesn't necessarily need to happen in the language. It just needs to happen...somehow 🙂

Thoughts?

[bsutton]

I've raised an issue suggesting that we add a lint rule as a middle ground.

The lint rule would give most of the benefits in that exceptions would be documented without requiring a fairly significant changed to the language.

https://github.com/dart-lang/linter/issues/2246

[lrhn]

For what it's worth, that's actually design smell from the Glob package that its public API exposes a class from a different package, one which it doesn't export itself. I'd prefer if it caught the string scanner's exception and threw a FormatException instead.
If the analyzer had warned about the undocumented exception, then it's probably more likely that the package would have noticed and handled it.
(Edit: And they apparently do throw FormatException since StringScannerException implements it. It would still be better if that was documented in the public API.)

[OlegAlexander]

Thank you, @lrhn, for your insight. Upon digging into the string_scanner code further, I discovered that StringScannerException ultimately implements FormatException! So thankfully, I don't have to directly depend on string_scanner anymore just for the sake of a dartdoc link.

I'll admit that I'm a relative newcomer to Dart and that maybe I could've found this solution sooner. But I think the main point is still valid because I only found out about it after considering your (original unedited) response and then reading the code.

I like your idea about the analyzer warning about undocumented exceptions. I think it would have a major impact on the quality of documentation on pub.dev. Is anyone else in favor of this idea?

[bsutton]

@OlegAlexander

Add a vote to this issue will help to move the discussion forward.

https://github.com/dart-lang/linter/issues/2246