Visidata: Display errors with emoji of various widths

[lxcode]

Small description

I apologize for even wading into the waters of unicode and terminal character width, but I haven't found a workaround that will sanitize text appropriately. When looking at data that contains smatterings of emoji and other unicode oddities, the glitches in character width cause the column separators to jump around and get out of alignment. This is particularly apparent with various flag emoji (particularly severe when many of them are used in sequence). Is there anything that visidata could do to mitigate this, perhaps using jquast/wcwidth?

Expected result

Everything displays cleanly and columns line up.

Actual result with screenshot

Additional context
v 2.0.1
Tested with both kitty and iTerm2 on macOS, local machine (no tmux or anything).

[saulpw]

Try setting disp_ambig_width to 2 (it defaults to 1) and let me know if that helps.

[lxcode]

Unfortunately, that doesn't appear to make a difference. :-/ I've attached a tsv (had to rename it to txt to upload) that should work for repro.

chartest.txt

[saulpw]

Thanks for the test data. I use urxvt on Linux, and I also tried it with lxterminal:

My font doesn't include country flags apparently, but you can see that a) the non-emoji full-width characters display properly, and b) the alignment of the columns is correct. I'd love to get mac terminals to do the right thing, especially (b), but I'm not sure it's possible. I'm guessing this has to do with combining characters (a flag is actually two characters which result in the country flag icon when combined), which VisiData is allocating 2 full-width spaces for, but only takes up one space when drawn.

[ajkerrigan]

This is an interesting issue - I've had similar trouble but oddly _not_ on my Mac. When I use VisiData locally (generally inside Alacritty+tmux) everything is fine. If I record and play back an asciicast everything still looks fine. But when I upload a cast to asciinema.org, the column separators go out of alignment for rows with unicode values (sample).

I suspected I'd be able to work around this with some combination of font/encoding/terminal tweaks, but haven't found the right magic mix yet. I also tried messing with the disp_ambig_width option thanks to this thread. I'm not sure if this is the same core issue or a subtly different one, but it seems worth noting in case it's a useful data point.

[lxcode]

I just tested on Linux in kitty, and I do get misalignments when I expand rows to full width (row 3). gnome-terminal keeps the alignment right, but mangles the emoji pretty badly.

My guess is this is primarily going to be an issue with flags and things like 👪 that are composite emoji of skin tone/gender.

[lxcode]

Digging a bit further into this: it looks like dispwidth() can be replaced with calls to wcwidth/wcswidth (in cliptext.py and column.py), and this is probably a more future-proof and comprehensive way to deal with unicode string lengths. General calls to len() for strings like trunch and sepchars may as well use wcwidth as well.

Unfortunately, replacing those doesn't seem to fix the issue in kitty, iTerm or Terminal. I'm assuming the answer is somewhere in drawRow, but I'm not seeing it.

[saulpw]

Maybe there's a way to work around it by directly placing the column separators instead of appending them to the value string. I thought it was already doing this, but apparently not. The wcwidth library looks great (and I have been frustrated myself by the same problem, as evidenced by dispwidth), but for the time being I'd rather not take on another dependency if it doesn't fix a problem.