Chai: Better Error Reporting

[AnAppAMonth]

An assertion library like _Chai_ knows the tests. It knows that objects A and B are supposed to be deeply equal, or that array X should be a superset of array Y. But it doesn't currently use this knowledge when reporting an error, to help the user see at a glance which part in a complex object or long string violates the expectation. The current solution is to print both operands in their entirety, regardless of the type of the assertion (the knowledge you have).

There are the expected and actual values, but they are only for equality tests, and are only used by supported testing frameworks. Additionally, these frameworks may not use them in an optimal way. For example, Mocha doesn't actually diff the objects, instead it uses the jsdiff library to diff the JSONs of the objects. The problem is, jsdiff isn't very good at diffing JSONs.

Ultimately, this is a problem that should be solved in assertion libraries, not the testing framework above it, because only the assertion library _knows_ the tests. Errors should be reported according to their nature. For example, if an assertion that array X is a superset of array Y fails, instead of printing out both X and Y, we should only display a list of elements in Y that are not in X: these are the root cause of the failure.

My proposal of implementing this functionality is to create a middleware-like system for _error message producers_. Each assertion, including core assertions and assertions defined in plugins, should have a stack of error message producers. Users can add producer functions to a stack, and a producer function can handle either all situations, all just some situations. When an assertion fails, the producer functions in its stack are called one by one, until an error message is produced for it. If all producer functions are called and the error message is still not produced, then the default error message is used.

For example, the user can add an error message producer function to the contain assertion. When the assertion fails and the function is called, it checks whether the two operands are strings, if so, it checks whether it can find a fuzzy match of the substring in the superstring. If a good enough fuzzy match is found, the difference between the substring and its fuzzy match in the superstring is displayed in the error message because it's likely the cause of the failure. If no such match is found, or if the two operands are not strings, the function returns a falsy value, and the default error message is displayed (or the next producer function is called).

The assertion library should also provide utilities to assist the user compare and contrast values in various ways. I'm writing a MIT-licensed library for diffing objects, strings, and arrays, I can adapt it to be used by _Chai_ if needed.

I want to know whether you feel this belongs to the core. If so, I can also contribute some code to implement it; if not, I will probably write a separate "assertion library enhancer" that extends existing assertion libraries to add this functionality.

[diversario]

Even a simple error from assertion like

assert(arr.length === 100)

prints

Uncaught AssertionError: Unspecified AssertionError

Built-in assert reports

AssertionError: false == true

which at least gives me an idea where to look.

[AnAppAMonth]

You always know where to look, my friend, always.
You need no assertion libraries, you need no testing.
Just look at your program, it always contains bugs, so that is where to look.

OK, sorry, I just couldn't resist...

[diversario]

I guess I'll have to stick with bundled assert :/