Visidata: On saving sheets with cell errors

[anjakefala]

The problem

In VisiData, a red exclamation mark in a cell signifies an error in processing the contents of the cell. z^E then allows the user to view the traceback for that cell's error.

• the sheets are not cluttered with error tracebacks that are meaningless to a subset of users
• data is not just dropped without users knowing about it

However, the question then becomes what the behaviour should be when a sheet with cell errors is saved.

pre v1.3 behaviour

Whatever was visibly displayed in the sheet (getDisplayValue) was what was saved. This has since been changed to something more performant, but not equivalent to getDisplayValue.

Possible approaches for v1.4.1

1) Have an option save_errors. When set to True saves some error string associated with the error. When set to False, cell is saved as blank. Con: saved error cell value is not consistent or searchable.
2) Have an option safe_error which saves the same configurable error string for all errors (e.g. '#ERR'). Can be set to '' to save the cell as blank. Con: errors are not differentiable; rollups differentiate but this will be completely lost upon saving.
3) abort the save with an exception (on status line) when error is encountered. Con: Inability to complete a save if any errors exist. May be very frustrating.

These are our main tactics so far, and we can set the defaults either way. The main concern with
having them off by default is that data may be lost (via errors) during save and the user may not know until it is way too late. The main concern with having them on by default is that they are messy and nothing else does it like this.

[anjakefala]

An example comparing save_errors when set to True vs False

[anjakefala]

This issue was made to welcome @jsvine, @deinspanjer, and anyone who is using VisiData to regularly work with messy data to provide input. =)

[jsvine]

Thanks for raising this interesting question, @anjakefala. My preference would be to default to Approach 2, with the user-settable option of choosing Approach 1.

[anjakefala]

Thanks for replying. :D I like approach 2 best as well.

For now, integrating both is sadly fairly involved and we think is out of scope for 1.4.1...

But! If we do settle on approach 2, the question becomes to what the default should be. ^^;

We are leaning towards leaving the default as '' and having a motd that signifies the option's configurability. Other considerations are something like '#ERR' but we do not really have any solid ideas that are jumping out.

Thoughts?

[jsvine]

I have a slight preference toward #ERR or #ERROR, so that errors are easy to spot and distinguishable from null/blank values (which might be plausible in certain computed columns).

[anjakefala]

Option has been added here: https://github.com/saulpw/visidata/commit/ed335394ad9446975919e4c766e08d9c2f60b4c1

=)