Visidata: Timing issue (?) replaying select-col-regex followed by delete-selected

[ajkerrigan]

Small description
When replaying a cmdlog that does a selection followed by a bulk delete, the delete intermittently begins before the selection is finished.

Expected result
A bulk delete (delete-selected, gD) should not disrupt an in-progress selection thread.

Actual result with screenshot
Intermittent index errors and inconsistent deletion results, with a traceback like:

Traceback (most recent call last):
  File "/Users/akerrigan/.local/pipx/venvs/visidata/lib/python3.8/site-packages/visidata/threads.py", line 201, in _toplevelTryFunc
    t.status = func(*args, **kwargs)
  File "/Users/akerrigan/.local/pipx/venvs/visidata/lib/python3.8/site-packages/visidata/selection.py", line 48, in select
    for r in (Progress(rows, 'selecting') if progress else rows):
  File "/Users/akerrigan/.local/pipx/venvs/visidata/lib/python3.8/site-packages/visidata/selection.py", line 71, in <genexpr>
    self.select((self.rows[i] for i in rowIdxs), progress=False)
  File "/Users/akerrigan/.local/pipx/venvs/visidata/lib/python3.8/site-packages/visidata/movement.py", line 134, in searchRegex
    c = findMatchingColumn(sheet, sheet.rows[rowidx], columns, regex.search)
IndexError: list index out of range

Steps to reproduce with sample data and a .vd
Sample data, reproduction steps, cmdlog, and example success/error logs up here

Additional context
Please include the version of VisiData.

v2.-4dev (commit 3446661caa8b68f2ce5ab7b90fbe9b1b1a3b5e46)

Notes

Tweaks that seem to avoid this issue:

• Working with a CSV file rather than JSON
• Replaying in batch mode (-b)

[anjakefala]

Thanks for the wonderful bug report @ajkerrigan! I loved the sample data and .vd.

I think that replay mode, outside of batch, did not complete a task before starting the next one.

Replay mode should block similarly to batch, per @saulpw, so I am going to look into making that change, and see how we feel about it.

[anjakefala]

I have more information!

Replay does block (runs one command directly after another) the actual cmdlog execution. The main difference between it + batch mode, is that replay mode has a draw cycle that runs asynchronously along with the cmdlog replay.

This stacktrace references status(), which makes me think that something went wrong when it was trying to display the progress() meter, and possibly the replay had already bull-dozed ahead (or something like that). The selected rows had already been deleted while this Progress meter draw-er was figuring out what to display (maybe?).

I was also able to reproduce this with the quantum-sum test. The difference is that that one fails when it tries to draw a column header while in some in-between state.

This might be really difficult to fix. =( Or, at least, it is outside of my comfort zone. This will be @saulpw's call for fixability.

[ajkerrigan]

Thanks for digging into this @anjakefala! It does seem like a tricky one, but reproducible helps 😁.

I'm going to want to come back and dig into this again, but it's not at the top of my list so I don't expect it to be at the top of yours either!

[anjakefala]

Kondo'd for now!