Visidata: [batch] AttributeError in batch replay

[Geoffrey42]

Small description

Hi, I want to automate some CSV files edition using vd replays. To be more specific:
From an existing CSV, I add a new column and save the sheet as a new file. That's it.

I believe it's probably some simple task I can't wrap my mind on it.

Expected result

If I replay the log file using the interface (--play option) everything works fine and the newly edited file is created. If I try to do the same without interface using ---batch mode I don't have the new file.

Actual result with screenshot

I get no error.

Steps to reproduce with sample data and a .vd

Here are the samples:

original csv: addresses.csv

|First name |Last name|street |state |ZIP|
|---------------------|---------|--------------------------------|-----------|---|
|John |Doe |120 jefferson st. |Riverside | NJ|
|Jack |McGinnis |220 hobo Av. |Phila | PA|
|John "Da Man" |Repici |120 Jefferson St. |Riverside | NJ|
|Stephen |Tyler |7452 Terrace "At the Plaza" road|SomeTown |SD |
| |Blankman | |SomeTown | SD|
|Joan "the bone", Anne|Jet |9th, at Terrace plc |Desert City|CO |

resulting csv: addresses-edited.csv

|Team |First name|Last name |street |state|ZIP |
|---------------------|----------|--------------------------------|-----------|-----|------|
|IT |John |Doe |120 jefferson st.|Riverside|NJ |
|IT |Jack |McGinnis |220 hobo Av.|Phila|PA |
|IT |John "Da Man"|Repici |120 Jefferson St.|Riverside|NJ |
|IT |Stephen |Tyler |7452 Terrace "At the Plaza" road|SomeTown|SD |
|IT | |Blankman | |SomeTown|SD |
|IT |Joan "the bone", Anne|Jet |9th, at Terrace plc|Desert City|CO |

commands log file: cmdlog-addresses.vd

|sheet |col |row |longname |input|keystrokes|comment|
|---------------------|--------|--------------------------------|-----------|-----|----------|-------|
| | | |open-file |addresses.csv|o | |
|addresses |First name| |addcol-new |Team |za | |
|addresses |Team |0 |edit-cell |IT |e | |
|addresses |Team | |fill-nulls | |f | |
|addresses |Team | |slide-left | |H | |
|addresses | | |save-sheet |addresses-edited.csv|^S | |

steps to reproduce:
1) Check the directory with:

$ ls
addresses.csv  cmdlog-addresses.vd
$

2) Play the record in batch mode:

$ vd -b cmdlog-addresses.vd
opening cmdlog-addresses as vd
$

3) Check if addresses-edited.csv is created:

$ ls
addresses.csv  cmdlog-addresses.vd
$

4) Alternatively, play the record using -p option:

$ vd -p cmdlog-addresses.vd

5) Watch the record and quit the sheet

6) Check again if addresses-edited.csv is created:

$ ls
addresses-edited.csv  addresses.csv  cmdlog-addresses.vd

Additional context
Please include the version of VisiData.

saul.pw/VisiData v1.5.2

Thanks for the help, vd really is a cool product.

[anjakefala]

Hi @Geoffrey42! I am sorry that the docs were not clear about this. I will make a note to ensure they are more clear. I really appreciate how thorough you were with your report.

There are two ways batch mode is currently used:

1. To convert between file formats (vd -b input.tsv -o output.tsv).
2. To replay .vds in non-interactive mode (vd -b -p replay.vd). You were missing the -p flag. It is needed to specify replay, even in batch-mode.

The clue was in the:

opening cmdlog-addresses as vd

Batch mode was not actually replaying the .vd. It was loading it, had no additional commands to do, and exiting. The -p flag specifies, replay this .vd.

Does it work for you now?

[Geoffrey42]

Oh I was sure I was missing some point !

Unfortunately I still get this error, but it's a real good sign !

$ vd -b -p cmdlog-addresses.vd 
opening cmdlog-addresses as vd
"addresses.csv"
opening addresses as csv
AttributeError: 'NoneType' object has no attribute 'erase'
replay aborted
$

Thanks for the relabeling, bug wasn't appropriate.

[anjakefala]

I have been able to reproduce the above! Now, we have a batch bug. ^^;

The replay is fine in live mode, but throws this Exception on batch.

I can take a look at this, soon! @saulpw, is there a way to get full stack traces in batch mode?

[Geoffrey42]

@anjakefala Thanks for the quick reply, I was already looking at the csv loader to understand what could be wrong.

Well if it's a bug I will try some workaround to get what I want but I'll stick to vd since it's so dope.

[saulpw]

I think maybe in batch mode we should make --debug abort with stack trace. However, aborting on every single exceptionCaught would make this useless; the trick is to only re-raise internal errors like this, and not exceptions which are 'natural' and just part of working with the data.

[anjakefala]

@Geoffrey42, I changed this issue back to a bug report!

I am ++ for that, @saulpw. That would make debugging stuff like this a lot simpler.

[anjakefala]

addresses.zip

Contains addresses.csv and addresses_cmdlog.vd

[saulpw]

Thanks for the excellent bug report, @Geoffrey42! Should be fixed now in develop.