fish-shell 🚀 - Unicode support broken (was This ⸻ unicode ⸻ character ⸻ makes ⸻ weird ⸻ things ⸻ happen)

This seems similar to #2199, no?

faho on 4 Jan 2016

Yes, and I think also #750.

zanchey on 4 Jan 2016

I don't experience the same thing, but instead the color of the dash gets all weird, guess it is due to the overlapping. I don't see any repetition of other characters.

I am using termite on arch linux.

terlar on 4 Jan 2016

This failure mode is when fish's idea of where the cursor is gets out of sync with where it actually is. This can come about if fish thinks a character is single width but it's actually double width, or vice versa.

My font doesn't seem to have this character :(

ridiculousfish on 6 Jan 2016

There are two things you can do to contain the effect of the disparity between wcwidth and the actual terminal: 1) turn off autowrap whenever the editor is in control (echo '\033[?7l' to turn off, '\033[?7h' to turn on) and insert returns explicitly; 2) always re-render whole lines, from left to right, when the buffer changes. This way the line number is always correct and you minimize your dependency on the column number.

I hope the advice is useful. :)

xiaq on 25 Jan 2016

@xiaq Hi, I didn't know you are back.

@terlar, what font are you using? @ridiculousfish and I too actually doesn't have that character.

pickfire on 26 Jan 2016

@pickfire You can find it in Symbola and Duolos SIL.

jakwings on 26 Jan 2016

@jakwings I didn't know that those two fonts isn't in my package manager. Trying to install it manually.

pickfire on 26 Jan 2016

Yes, I think this glyph comes from the symbola package. Not sure which one it fallbacks to. But I know I have symbola to cover these kind of things.

terlar on 26 Jan 2016

I can't reproduce the problem using iTerm2 on OS X without a font installed that supports that character. So I see the single width undefined char glyph. Is it possible a recent change (such as the one I made to support the C locale better) has "fixed" this? Can anyone reproduce this problem using a fish built from git head? If so can you provide better directions for reproducing the problem than was in the original comment?

krader1961 on 25 Mar 2016

After installing Symbola (from http://www.fonts2u.com/symbola.font) I can reproduce this. It's pretty clear that fish is using the wrong char width. From https://en.wikipedia.org/wiki/Dash:

Less common are the two-em dash (⸺) and three-em dash (⸻), both added to Unicode with version 6.1 as U+2E3A and U+2E3B.

So our fish_wcwidth() implementation is out of date.

P.S., It isn't just that character which renders weirdly. The Symbola font is ugly --- at least as handled by OS X Terminal.app. The inter-char spacing is just awful; even for plain ASCII chars.

krader1961 on 19 Jun 2016

It shouldn't be necessary to actually select Symbola as your font. Just install it and it will be used for odds and ends (like ⸻) with a regular terminal font selected.

For me this font+character does the same thing as the double-width emojis in the just-closed #750 -- just a bigger difference in width with this example.

floam on 19 Jun 2016

FWIW, fish_wcwidth() returns 1 and the OS X El Capitan (10.11.6) wcwidth() returns -1.

For me this font+character does the same thing as the double-width emojis in the just-closed #750

That's weird because on my system that emoji renders as a single-width char in both Terminal.app (where I explicitly made Symbola the primary font) and iTerm2 using Consolas. However, in both of those multiple instances of that char overlap each other. So it's pretty clear it should be treated as double-width but fish and my terminals are treating it as single-width.

krader1961 on 19 Jun 2016

Right, and for me the same thing happens with this character - both fish and my terminal seem to render it as width-is-1 (leaving the glyph overhanging a few characters to the right). I'm not sure it's something fish could fix (at least here) - beyond the fact that it should probably just not allow it, like zsh or like it gets filtered out when using mosh to ssh into a remote fish instance.

floam on 19 Jun 2016

I'm not sure it's something fish could fix...

Clearly it's impossible for fish to know whether the fonts and terminal will treat this as a character with a width of one or two. The question is whether the Unicode standard mandates either width or is silent. If the former we should ensure we conform to the standard. If the latter then my recommendation is to treat it as single width as we currently do. I'm leery of filtering such characters because that means we have to distinguish their presence in interactive commands versus scripts.

krader1961 on 19 Jun 2016

BTW, the latest relevant standard that defines how chars should be treated can be found here:

ftp://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt

It says that the two chars in question have a width property of "Na" which means "narrow" (i.e., single cell when displayed). See the section "5.11.1 Enumerated and Binary Properties" in

http://www.unicode.org/reports/tr44/

for what the symbols "N", "Na", "A", etc. mean.

So simply updating our fish_wcwidth() implementation to the latest Unicode standard would not fix this issue. Nonetheless, I recommend modifying our implementation to dynamically construct its width table from that file at build time. We could then layer any customizations on top of it we felt were necessary.

I think there needs to be a discussion whether fish should continue to have its own implementation of wcwidth(). Unicode is now a mature standard that still has warts. Is it really the best use of our time to track updates to that standard rather than relying on the maintainers of the platforms we run on to do so?

krader1961 on 23 Jul 2016

From e7273e1d81e1ef7c615:

// Big hack to use our versions of wcswidth where we know them to be broken, which is
// EVERYWHERE (https://github.com/fish-shell/fish-shell/issues/2199)
#ifndef HAVE_BROKEN_WCWIDTH
#define HAVE_BROKEN_WCWIDTH 1
#endif

The problem was (is?) that the maintainers of the platforms we run on weren't tracking updates to the standard either!

zanchey on 25 Jul 2016

On Linux, it looks like glibc is pretty good? Or at least not bad (latest)?

floam on 25 Jul 2016

The problem was (is?) that the maintainers of the platforms we run on weren't tracking updates to the standard either!

True, but the Gnu library and distro maintainers have more incentive to do so than we do. Simply because changes they make to the core locale subsystem affect a lot of programs. Any changes we make only affect fish and make fish behave differently than every other program on the system. If we are going to continue maintaining our own implementation then it should be semi-automated in the manner I outlined above.

krader1961 on 25 Jul 2016

👍2

This also happened to me when I accidentally pasted a tab character.

adriaanzon on 10 Jan 2017

@krader1961 @floam I don't think this is glibc issue, I tested this on void linux (musl), same thing happens. The 3-em dash seems to merge with other character.

I have also tested this with dash and bash, seems all of them have this problem too.

pickfire on 11 Jan 2017

This happened to me after an upgrade from Fedora 24 to 25.
I had this character in my prompt and didn't have any problem before the upgrade

sebastiencs on 30 Mar 2017

👍1

I don't know anything about fish or Unicode and I've maybe written ten lines of C++ in my life but I just compiled with HAVE_BROKEN_WCWIDTH 0 and my Ubuntu 17.04/Terminator/symbola has been handling emoji great.

🤷

iameli on 13 Jul 2017

👍1

I've got the same problem as @sebastiencs . Arch linux, and the problem appeared between 2.5.0 and 2.6.0.

jaredwindover on 26 Sep 2017

I changed to a different character, and realized that my terminal seems to be displaying the original character as double width (I thought I had spaces around it). This may be a problem with urxvt.

jaredwindover on 26 Sep 2017

EastAsianWidth.txt suggests that the double width for the high-voltage-sign (26A1) is correct. This header file seems to be a good alternative to wcwidth. Maybe worth a look.

jaredwindover on 26 Sep 2017

@jaredwindover are you sure the problem is the fish upgrade? i tested with fish 2.5.0 and got the same problem. Perhaps it is due to the glibc 2.25 -> 2.26 update?

bennofs on 29 Sep 2017

@bennofs That's totally plausbile. fish 2.5 vs 2.6 could be a red herring. I just looked at my pacman log and assumed based on the timing. The glibc update fits the timing as well.

jaredwindover on 29 Sep 2017

From #archlinux:

16:16 <bennofs> Tom^: unfortunately, i haven't exactly measured it
16:16 <Tom^> bennofs: 2.26 did bring in unicode 9,10 which changed the wcwidth on tons of unicode chars that were in the past reporting as width 1 but were in fact 2. and it seems fish has some ugly hack around that.
16:16 <Tom^> so it is very possible glibc broke it by in fact fixing it.

so I think it is the glibc update.

bennofs on 29 Sep 2017

👍2

Having the same issue again and again xD Would be cool to merge it.

pgolebiowski on 7 Dec 2017

👍1

Same issue here. Can't remember fish not being able to handle emojis before, how can I install a working version?

albinekb on 19 Feb 2018

5282d3e7110f40fc9cb51f4ae952f65000bbf0ae rationalizes a lot of how fish measures characters including emoji. In particular U+2E3B should be properly measured as width 1 (though it overdraws into other characters). Please comment / reopen if you still see this - thanks!

ridiculousfish on 26 Feb 2018

🎉4 😕1

I can confirm that:

– Using stable or master release of fish, emoji spacing is broken
– Using master release with this edit works great, as suggested up there.

ariasuni on 12 Mar 2018

👍1

@ariasuni what is your terminal emulator and OS? Does set -g fish_emoji_width 1 (or 2) fix it?

ridiculousfish on 21 Mar 2018

Changing fish_emoji_width does not seem to change anything (Konsole on Arch Linux).

ariasuni on 21 Mar 2018

Just wanted to say as soon as that commit landed 24 days ago I installed the master fish on my Arch (with Konsole) and added set -g fish_emoji_width 2 to my config.fish and it has worked great since without any emoji issues.

YaLTeR on 22 Mar 2018

Most emoji work fine for me with master, fish_emoji_width 2 and Konsole. But I found ☺️ which seems to not handled correctly. And also all others that have skin color or gender as a "second attribute" (see https://unicode.org/reports/tr51/#Emoji_ZWJ_Sequences) don't get handled correctly.

z3ntu on 22 Mar 2018

I tested several emojis, both with or without ZWJ, on Konsole and Kitty. It works better with system wcwidth, except for a Konsole bug.

@z3ntu what problem did you encounter? Displaying color and interpreting ZWJ sequences is not fish job but terminal emulator’s.

ariasuni on 22 Mar 2018

Displaying color and interpreting ZWJ sequences is not fish job but terminal emulator’s.

Not quite. Fish does a bunch of cursor movement (for suggestions, the right prompt and such), and for that it needs to know the width, and it needs to agree with the terminal on it.

faho on 24 Mar 2018

👍1

Using fish 2.6.0, changing the fish_emoji_width doesn't seem to make a difference. The problems I have are small but still annoying. Is there a plan how to properly fix this?

spacekookie on 23 Apr 2018

👍1

I have fish version 2.7.1-808-g9444c65e

with set -g fish_emoji_width 2 in my fish.config

kapture 2018-04-23 at 14 49 56

Still seeing issues like this

albinekb on 23 Apr 2018

👍1

Using fish 2.6.0, changing the fish_emoji_width doesn't seem to make a difference

@spacekookie: Not too surprising, 2.6.0 does not have fish_emoji_width.

@albinekb: Please try building fish with --disable-internal-wcwidth (if using autotools, -DINTERNAL_WCWIDTH=OFF if using cmake) and try again.

faho on 23 Apr 2018

👍1

@faho Ah woops! I thought it was added then; I'll go do that then! Thanks.

spacekookie on 23 Apr 2018

I'm building with brew, do you know how to pass that option to cmake in brew? 🤔
https://github.com/Homebrew/homebrew-core/blob/master/Formula/fish.rb#L31
I don't know ruby, looks like it should be passing all extra args to cmake?

Edit: nevermind, after rebuilding, i got version 2.7.1-1096-ga9e9af5c, and now emojis work like expected, got some other issues with my prompt but I assume thats becuase it's using deprecated stuff

fatal: ambiguous argument '^/dev/null': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: Refusing to point HEAD outside of refs/
head: ^/dev/null: No such file or directory

albinekb on 23 Apr 2018

@faho Unfortunately that (and setting fish_emoji_width) didn't change anything for me :disappointed:

spacekookie on 23 Apr 2018

@spacekookie: Which fish version did you build? Which terminal on which OS (including version) are you using?

faho on 23 Apr 2018

Fish version: fish, version 2.7.1-1096-ga9e9af5c

Os: Fedora 27 Workstation

Terminal:

Tilix version: 1.7.5
VTE version: 0.50
GTK Version: 3.22.26

spacekookie on 23 Apr 2018

Oh, and also which character is causing the problem?

Can you try with another terminal?

faho on 23 Apr 2018

I use emoji in my prompts to differentiate between machines.

My current one is :point_right:. The problem occurs with other characters (for example my servers are: :bullettrain_front:, :house: and :cookie:) and the issue occurs just the same with gnome terminal and Terminator (same backend though). I also tried konsole (KDE Terminal) where the problem is actually a lot worse...

spacekookie on 23 Apr 2018

My current one is point_right.

So that's 👉 - U+1F449 ("WHITE RIGHT POINTING BACKHAND INDEX")

Just to be clear: You _have_ built fish with cmake -DINTERNAL_WIDTH=OFF or ./configure --disable-internal-wcwidth? That fixes it for me, though my system is a teensy bit more up-to-date (tilix is 1.7.7 instead of 1.7.5, gtk3 is 3.22.30, vte is 0.52.1).

faho on 23 Apr 2018

👍1

@faho I don't know what to tell you, I wish it was fixed :disappointed:

I built the master branch of fish-shell via cmake (because fuck autotools) and the -DINTERNAL_WIDTH=OFF flag. I installed fish to /usr/local/bin and set fish_emoji_width to 2.

The errors I get are more subtle then what is displayed here in the repo but there is some definite corruption, especially when trying to use tab-completion. Also, I had to essentially disable my right prompt because it would always overflow my cursor onto the next line.

Now, if you're telling me that's a different bug then maybe that is so. But it's still a bug caused by emoji with fish which I wish could be fixed. I hope this helps :crossed_fingers:

spacekookie on 24 Apr 2018

@spacekookie:

and the -DINTERNAL_WIDTH=OFF flag.

Sorry, I apparently made a mistake.

It's -DINTERNAL_WCWIDTH=OFF. It refers to the "wcwidth" function.

Please double-check!

and set fish_emoji_width to 2

(Without internal wcwidth, that variable won't have any effect - it's a different solution to the same problem)

faho on 26 Apr 2018

@faho Ah, yea. I made a mistake when writing my reply because I copied the flag from your comment. But in actuality I compiled it with the correct setting. If you provide a flag that doesn't exist cmake throws an error!

I just tried to capture the issues I'm getting on a screen-recorder but Wayland is preventing this. When I'm not at work I will re-login with Xorg and record what's going on.

But yea, I'm still having issues, using the newest fish master branch, using the right flag... :disappointed:
I'll edit this comment with a link to a video later.

spacekookie on 2 May 2018

If you provide a flag that doesn't exist cmake throws an error!

Mine doesn't - instead it prints a warning: "Manually-specified variables were not used by the project".

Anyway, I have now managed to reproduce this with my GNOME Terminal. With the offending character U+2588 (█ - "FULL BLOCK"), this happens:

screenshot_20180503_201654

The "solution" is to go into GNOME Terminal's settings and setting the width for ambiguous/unknown-width characters to "narrow" in the "Compatibility" tab (I don't know what it's actually called in english because I can't get it to start in anything but german).

Which is quite annoying, because glibc's wcwidth just says it has a width of 1, as does our copy (those correspond to INTERNAL_WCWIDTH=OFF/ON, respectively)! So GNOME Terminal uses its own system that differs from it, which is impossible for us to figure out.

faho on 3 May 2018

😄1 👍1

I've seen a few solutions to the problem and I've tried some of them... what's the definitive solution?

I use Ubuntu 18.04, GNOME Terminal, with fish 2.7.1... Whenever an emoji appears in the prompt, it gets put on the next line (and the beginning of each new line has an "m" for some reason)

It seems like a solution was found but I'm finding it hard to get a step-by-step procedure for this... anyone have anything like that?

deanveloper on 1 Jun 2018

I've seen a few solutions to the problem and I've tried some of them... what's the definitive solution?

@deanveloper: There's no silver bullet. Fundamentally, what needs to happen is that your terminal and fish agree on the width of characters.

We have two approaches to make that more likely, neither of which is available in fish 2.7.1 - they'll be included in 3.0.

The first is @ridiculousfish's $fish_emoji_width. The idea there is that there are some characters that have "ambiguous" width, so they could be interpreted as having a width of 1 or 2, and some terminals (like GNOME Terminal) have a setting to change that. So you set $fish_emoji_width to 1 or 2 and your terminal to the same thing.

The other approach is my -DINTERNAL_WCWIDTH=OFF compile option. The idea there is that the terminal will most likely just use your system's "wcwidth()" function to determine character width, so fish should just use the same copy. This mostly helps with systems that have a wcwidth that returns different results than what fish ships with. If you use this, you need to set GNOME Terminal to see ambiguous characters as "narrow" (i.e. width of 1), but then it should work pretty much flawlessly.

As you can see, neither of the two approaches is bullet-proof, and they are currently mutually exclusive (they could be reconciled, though).

If you are using an up-to-date linux distribution with glibc >= 2.26 (which I believe Ubuntu 18.04 is), you probably want -DINTERNAL_WCWIDTH=OFF, which the Ubuntu packager should probably also pick once 3.0 is released, which would make it "just work" for Ubuntu specifically.

If you are building for a system that is supposed to be used via ssh or similar, and that system uses a standard C library that is different from the system your sshing _from_ - either because the server is RHEL 5 with an old and ~~crusty~~ tested glibc, or because it's Void or Alpine with Musl - you want -DINTERNAL_WCWIDTH=ON. That would cause issues if you also ran a desktop with a terminal emulator on those systems locally, but there's nothing we can do there.

Also if your terminal emulator is _wrong_ on some character width - we've had an example where one displayed a zero-width space with a width of 1 - there's nothing we can do. Which is more annoying, because the _font_ can also influence the width - the typical case is when it displays a replacement glyph with a different width from the actual char, or when it's not monospace.

faho on 3 Jun 2018

👍2

I've switched to a different Terminal and testing it now on Arch Linux, I still get some isues and I'm wondering if the compilation flag is being considered or if I'm doing something else wrong. I would really like to figure this out.

The terminal emulator is kitty which has a use_system_wcwidth option which I enabled. When using a different shell (like bash) emoji support work flawlessly.

I cloned the fish-shell master branch and built it with

mkdir build ; cd build/
cmake -DINTERNAL_WCWIDTH=OFF ..
make
sudo make install

Which means my current fish version is: fish, version 2.7.1-1269-gf025607c. Yet, when I paste an emoji into, I get some weird issues. Here is a small screen recording of what it looks like. Note that the only thing I'm pressing here is the space bar! When using bash I'm purposefully inserting new emoji

spacekookie on 23 Jun 2018

@spacekookie thanks for making that gif. It looks like the red heart is unicode U+2764 followed by a variation selector U+FE0F. fish doesn't make any effort to handle these sort of composed character sequences yet. What you're encountering is probably a different issue; I'll take a look.