Bat: Garbage unicode chars on wide screen display

[saumyajyoti]

What version of bat are you using?
bat 0.15.0

I see some garbage characters and thus misaligning the bat output on screen. If I use 4K display, I see this issue when column size grows beyond 338. The below 3 chars are coming: ���

bat .\.gitignore
───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────���───────────────────────────────────────────
       │ File: .\.gitignore
───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────���───────────────────────────────────────────
...

I expect these 3 chars to be deleted from output, these are extra causing the rule line to carry to next line
...

How did you install bat?
Chocolatey

---

[paste the output of info.sh here]

[saumyajyoti]

Issue persists on bat 0.15.1 also.

[eth-p]

I don't have access to a Windows environment to confirm this, but it might be Windows-specific. I can't reproduce this on MacOS Mojave or Ubuntu Linux (via SSH) using either iTerm or the MacOS Terminal.

Just to rule out a couple things:

• What terminal are you using?
• Are you using bat through Cygwin/MSYS2, or natively through the Windows command prompt or PowerShell?

[sharkdp]

Thank you for reporting this.

Another few more things which might be worth testing:

Redirect the output to a file and see if those invalid UTF8 characters are present in the output:

bat .\.gitignore --decorations=always > output.txt

You might need to set the terminal width to a large value manually

bat .\.gitignore --decorations=always --terminal-size=350 > output.txt

It could be interesting to see what those characters actually are. You could redirect to bat -A to see them:

bat .\.gitignore --decorations=always --terminal-size=350 | bat -A

Another thing that you could test would be to disable the pager:

bat .\.gitignore --paging=never

[sharkdp]

If I use $K display

what is that?

[be5invis]

@saumyajyoti What is the terminal you are using? You _may_ need to file an issue on https://github.com/microsoft/terminal.
@sharkdp Should mean "4K". Shift + 4 → $

[sharkdp]

@sharkdp Should mean "4K". Shift + 4 → $

ah :-)

You _may_ need to file an issue on https://github.com/microsoft/terminal.

It actually looks a bit like https://github.com/microsoft/terminal/issues/386 (see also this duplicate: https://github.com/microsoft/terminal/issues/5233).

It could be triggered by the large amount of ─ characters that bat would print for this large terminal width. Note that ─ is a multi-byte UTF-8 sequence: \xE2\x94\x80

[saumyajyoti]

Thanks for looking into this. Sorry for late reply.
I am using a 4k display, sorry for the typo., The char come at column numbers 339,340,341.
I used Powershell and CMD with Windows terminal preview and the default windows console as well. I used the default less installed via chocolatey,
* Today I again checked with latest Windows terminal and console. The issue still persists in console but the Windows terminal does not show garbage anymore. Ran both with powershell .
* I used bat w/o pager on powershell default console again and it seems w/o pager issue is not there in default console also.
So, I assume it is coming with pager+console combination. As you have already mentioned the chars after copying to editor looks like \xE2\x94\x80.
Does not come when I redirect the output to file.
So, I guess we can close the issue on bat database. I think less pager issue.

[sharkdp]

You could try to use a very recent version of less from https://github.com/jftuga/less-Windows/releases

[saumyajyoti]

Thanks for quick response.
The issue is there with this version of less also. I tried this version few weeks back as well. Now I tried again with their latest release 561.1 also.
Actually I tried 4 different less versions:

1. Coming with git for windows
2. Chocolatey one
3. The one you mentioned : https://github.com/jftuga/less-Windows/releases
4. The one compatible with delta: https://github.com/lzybkr/less/releases/tag/fix_windows_vt

The delta diff does not work well with other 3 versions than the 4th one above. Maybe they should integrate the Win10 color fix to upstream.
Some time back I observed the git less version was better handling unicode chars in git log etc. compared to the default 551 (chocolatey) or any other releases. But now I don't see this problem.
To summarize: the latest LESS for windows is not perfect.

I am big fan of your RUST utilities. Maybe time to write a pager in rust which works well in all Scenarios: git, bat, delta etc.

[sharkdp]

If we want to further debug this, could you please perform the steps suggested in https://github.com/sharkdp/bat/issues/1000#issuecomment-629448983 and tell us what the outcome was?

I am big fan of your RUST utilities.

Thank you for the feedback!

Maybe time to write a pager in rust which works well in all Scenarios: git, bat, delta etc.

I have actually thought about that a few times :smile: But more for the fun of it. Don't be fooled. Rewriting a pager from scratch will certainly not result in something that can compete with less. All of the edge cases and pitfalls that less has already figured out since 1984 (!) need to be handled again.

The biggest hurdle is cross-platform or rather cross-terminal support, I believe. One thing I thought about was to write a pager for ANSI terminals only. That would make things much easier but there wouldn't be any support for older Windows versions, for example.

[saumyajyoti]

Thanks for detailed reply.
I had already run the above steps, I summarized few of them in previous answer i.e. coming only with paging involved the chars are \xE2\x94\x80. Giving detailed info now.

Redirect the output to a file and see if those invalid UTF8 characters are present in the output:
bat ..gitignore --decorations=always > output.txt

I think this bypasses pager so the issue is not seen.

You might need to set the terminal width to a large value manually
bat ..gitignore --decorations=always --terminal-size=350 > output.txt

Same as above. No issues w/o pager I guess.

It could be interesting to see what those characters actually are. You could redirect to bat -A to see them:
bat ..gitignore --decorations=always --terminal-size=350 | bat -A

Could not see the unknown chars this way, but I copied the terminal putput w/o bat -A and then found those are below unicode chars after hex decoding. Each of these 3 chars correspond to ef bf bd.

Another thing that you could test would be to disable the pager:
bat ..gitignore --paging=never

Issue not seen.

[sharkdp]

Ok. One last thing that could be checked would be to run with --paging=always and pipe to a file:

bat .\.gitignore --decorations=always --terminal-size=350 --paging=always > output.txt

Otherwise, I would assume that this is a bug in either less or the Windows terminal and would like to close this issue.

[saumyajyoti]

Hi,
I don't see the unknown char dumped to text file with above experiment.
Terminal shows it: As I mentioned earlier new Windows terminal has fixed it. But Windows console (old one) still has this issue. So, I assume we can close as terminal issue.
Thanks for your time.