Visidata: Design a loader compatibility chart, and write tests for core loaders

[anjakefala]

It would be nice to come up with criteria for vdcore loaders.

E.g.:

1. A user should be able to add-row.
2. A user should be able to use clipboard commands.

This way:

• we know which functionality to write tests for the current loaders
• have criteria to distinguish plugins from core loaders
• will be able to provide folks who are making PRs for new loaders guidance
• will be able to set expectations for users

This is an open discussion for the community. =)

• [x] does the saver preserve formatting if a "text file" (e.g. html); does the saver save types for a format that has basic types (e.g. npy)

[ajkerrigan]

Some that come to mind:

• Select/unselect/toggle rows
• Add/delete rows (individually or in bulk)
• Edit cells
• Sort

[frosencrantz]

This is a great idea. When implemented it should help increase the quality and stability of the code. I've encountered a few breakages that I think might have been caught by basic and standard tests like you are suggesting.

Some that come to mind:

• Safe round trip of data, load and save.
  
  
  
  • Do a simple round trip
  
  
  • Make multiple and different types of edits, then undo all, and save.
  
  
• All options/flags for a loader should be tested.
  
  
  
  • with non-default values.
  
  
  • in combination with non-default values
  
  
• If a sheet has additional commands they should be tested
  
  
  
  • run the commands
  
  
  • make sure they show up in help.
  
  
• Reload a table
• Check all columns including hidden ones. Unhide them all.
  
  
  
  • Some loaders have hidden columns like xml or the file directories
  
  
• Check for extra unexpected columns
• Expand data that can be expanded.
  
  
  
  • check expanding multiple levels deep
  
  
• For a loader that relies on an external module, make sure it works with supported versions of the module.
• For loaders that have multiple tables/sheets (e.g. sqlite, html) check operations on the top level
  
  
  
  • that all tables can be entered.
  
  
  • check top level edits
  
  
• Can you go to the meta sheets, Columns, Describe, Frequency, aggregators work.

One idea is a have a standardized generic test that can be applied to a fixed dataset. The standardized test would do regular visidata operations, and it could look for expected results. With a standard dataset, the expected results will already be known. This would not take advantage of any specialized features of a given loader or file type, but would make sure the loaded data works with the core of Visidata. This would be a test you could use as part of loader development and might not take much to convert to the new datatype. I'm not sure about the fine details that would make this difficult.

[anjakefala]

:sleepy:

[anjakefala]

Thanks @frosencrantz and @ajkerrigan for helping me put together an attribute list!

[anjakefala]

@frosencrantz My initial instinct was :sleepy: because that feels like a major infrastructure addition and a lot of work.

I do think these lists are useful, and I do agree having automation would be useful. Here are the various levels this can go:

• A manual testing checklist, based on our curation of which traits are essential for core loaders
• A selection of core loaders
• A series of .vds that test these attributes for core loaders
• A large testing infrastructure that comprehensively reports which loaders support which behaviours

We can see which level we end up going with, based on who works on this, and how much energy they allocate! =)

[frosencrantz]

@anjakefala, I'm glad you are feeling better about this. I like having the spectrum of manual to complete automation. It allows your idea of compatibility chart and testing to move forward over time as effort can be applied.

[anjakefala]

I added a baby file on dev/loader-compatibility.tsv (which we will definitely possibly rename, I just wanted to get this work started).

Anyone who has the energy can feel free to populate it!

[saulpw]

I added all the filetypes and indicated whether they had a loader or a saver and what the requirements are. We should find a place to post this as part of the docs. (visidata.org/formats?) Next we should collect all the current docs about loader internals and combine/review/update them for 2.0. (visidata.org/loaders?)

[anjakefala]

visidata.org/formats is the perfect place for this :+1:

Next we should collect all the current docs about loader internals and combine/review/update them for 2.0. (visidata.org/loaders?)

I started that work, and have not finished it, yet. we should definitely not forget to finish that.

https://github.com/saulpw/visidata-book/pull/6/files#diff-204f5a19390e8bce4c1fa6c641d1b0a4

[anjakefala]

We created a basic formats.jsonl where we track information about the loaders we support. This data is also displayed at https://visidata.org/formats.

We decided to not make a checklist of minimal expected behaviour from a loader, because it is pretty hard to define a precise list for everything. I want to encourage loader creation to develop organically alongside usage of the loader, based on how people want to interact with the data.

:bee: