Kitty: Extra spacing after emoji variants
I ran across this with the "rainbow flag" emoji, which is in hex => \u1f3f3\ufe0f\u200d\u1f308, or "waving white flag", "variant selector", "zero-width joiner" and "rainbow". However I input it, either by copy/pasting it or using the ctrl-shift-u kitty unicode input, it always renders with extra space afterwards:
If I try to print several of them in a row, it prints only a few of them, with really wide spacing. If I add an ascii space between each, then it prints all of them, and closer together.
- kitty 0.14.4
- noto color emoji font
paul
All 22 comments
First of, dont use them in a shell. Shells are full of unicode handling bugs. Instead run cat and see if the issue reproduces there, if so let me know and I will take a look.
kovidgoyal
on 19 Sep 2019
I copied the unicode sequence from here: https://emojipedia.org/rainbow-flag/
First cat is just that emoji in a text file ten times.
Second is the emoji + space ten times
Third is me pressing Ctrl-Shift-V with that emoji 10 times
(Forth is me attempting to Ctrl-Shift-V then Space 10 times, getting an extra space, hitting backspace and pasting again, but that seemed to screw everything up, so I aborted and tried again)
Fifth is is pressing Ctrl-Shift-V then Space in sequence 10 times.
nospace.txt:
π³οΈβππ³οΈβππ³οΈβππ³οΈβππ³οΈβππ³οΈβππ³οΈβππ³οΈβππ³οΈβππ³οΈβπ
with-space.txt:
π³οΈβπ π³οΈβπ π³οΈβπ π³οΈβπ π³οΈβπ π³οΈβπ π³οΈβπ π³οΈβπ π³οΈβπ π³οΈβπ
$ xxd nospace.txt
00000000: f09f 8fb3 efb8 8fe2 808d f09f 8c88 f09f ................
00000010: 8fb3 efb8 8fe2 808d f09f 8c88 f09f 8fb3 ................
00000020: efb8 8fe2 808d f09f 8c88 f09f 8fb3 efb8 ................
00000030: 8fe2 808d f09f 8c88 f09f 8fb3 efb8 8fe2 ................
00000040: 808d f09f 8c88 f09f 8fb3 efb8 8fe2 808d ................
00000050: f09f 8c88 f09f 8fb3 efb8 8fe2 808d f09f ................
00000060: 8c88 f09f 8fb3 efb8 8fe2 808d f09f 8c88 ................
00000070: f09f 8fb3 efb8 8fe2 808d f09f 8c88 f09f ................
00000080: 8fb3 efb8 8fe2 808d f09f 8c88 ............
$ xxd with-space.txt
00000000: f09f 8fb3 efb8 8fe2 808d f09f 8c88 20f0 .............. .
00000010: 9f8f b3ef b88f e280 8df0 9f8c 8820 f09f ............. ..
00000020: 8fb3 efb8 8fe2 808d f09f 8c88 20f0 9f8f ............ ...
00000030: b3ef b88f e280 8df0 9f8c 8820 f09f 8fb3 ........... ....
00000040: efb8 8fe2 808d f09f 8c88 20f0 9f8f b3ef .......... .....
00000050: b88f e280 8df0 9f8c 8820 f09f 8fb3 efb8 ......... ......
00000060: 8fe2 808d f09f 8c88 20f0 9f8f b3ef b88f ........ .......
00000070: e280 8df0 9f8c 8820 f09f 8fb3 efb8 8fe2 ....... ........
00000080: 808d f09f 8c88 20f0 9f8f b3ef b88f e280 ...... .........
00000090: 8df0 9f8c 8820 .....
$ cat nospace.txt| ruby -e 'STDIN.read.split(//).map {|c| printf("U+%04x ", c.ord) }'
U+1f3f3 U+fe0f U+200d U+1f308 U+1f3f3 U+fe0f U+200d U+1f308 U+1f3f3 U+fe0f U+200d U+1f308 U+1f3f3 U+fe0f U+200d U+1f308 U+1f3f3 U+fe0f U+200d U+1f308 U+1f3f3 U+fe0f U+200d U+1f308 U+1f3f3 U+fe0f U+200d U+1f308 U+1f3f3 U+fe0f U+200d U+1f308 U+1f3f3 U+fe0f U+200d U+1f308 U+1f3f3 U+fe0f U+200d U+1f308
$ cat with-space.txt| ruby -e 'STDIN.read.split(//).map {|c| printf("U+%04x ", c.ord) }'
U+1f3f3 U+fe0f U+200d U+1f308 U+0020 U+1f3f3 U+fe0f U+200d U+1f308 U+0020 U+1f3f3 U+fe0f U+200d U+1f308 U+0020 U+1f3f3 U+fe0f U+200d U+1f308 U+0020 U+1f3f3 U+fe0f U+200d U+1f308 U+0020 U+1f3f3 U+fe0fU+200d U+1f308 U+0020 U+1f3f3 U+fe0f U+200d U+1f308 U+0020 U+1f3f3 U+fe0f U+200d U+1f308 U+0020 U+1f3f3 U+fe0f U+200d U+1f308 U+0020 U+1f3f3 U+fe0f U+200d U+1f308 U+0020
I think that the emoji are making this problem seem harder than it is.
It is at its core an error in ligature handling. Consider Nimbus Mono PS, or my font TT2020 Style B (which is how I first knew about this problem):
(Note: This will only work for Nimbus Mono PS post–d250555, because I removed the specific processing for it now that we have font_features to do it.)
Kitty renders ligatures across multiple cells according to the wcswidth of their components and dnot the wcswidth of their completed form, that's the problem. No rainbow flags necessary.
ctrlcctrlv
on 7 Jan 2020
There is no wcswidth of the completed form, since in general the
completed form is not a unicode codepoint but an arbitrary glyph. Also
if a terminal application print fl to the screen it expects it to take
two cells, if it does not it will break many applications. So I dont
really see how this can be fixed. fl and ligatures in general must
always take the number of cells indicated by the wcswidth() of the
underlying string.
kovidgoyal
on 7 Jan 2020
Indeed, I agree fully. My "fix" would be just better spacing—the extra space wouldn't look so bad if it was just factored into the whole word.
We already have ability to disable_ligatures when hover (cursor). So we could repurpose that for spacing. If the width of the ligature _X_ is != the width of wcswidth(X) cells, break the ligature on hover.
Thought?
ctrlcctrlv
on 7 Jan 2020
I dont think that's possible. In kitty character images per cell (or
multipe cells for a ligature) are stored on the GPU and rendered
directly for each cell. You can distribute spacing around a word, unless
you make the entire word a ligature and you cant do that in the general
case because kitty has a max ligature size and words could be longer,
not to mention that long ligatures are not good for performance.
kovidgoyal
on 7 Jan 2020
Could not an extra step be added right at the end, to look for badly spaced ligatures, and fix them?
ctrlcctrlv
on 7 Jan 2020
There is no end. Rendering works by mapping each cell to a number. That
number acts as an index into a sprite map which is uploaded to the GPU.
The list of numbers is sent to the GPU and the GPU renders them in a
single pass. In the case of ligatures, the entire ligature is rendered,
then split into cells and the cells added to the sprite map.
kovidgoyal
on 7 Jan 2020
Idea for you @kovidgoyal, and I'm sorry, it might be a stupid one. Maybe so stupid you don't even want to reply, which I will understand and not hold against you.
But in case you haven't thought of this already:
The OSWindow contains one or more Windows, which are made of a Screen, which contains cells. Each cell is literally a quad built out of an array of OpenGL vertices, known internally as vao, accessed by vao_idx.
Let's say we have the word affiliate, and the following sprites: «a f_f_i i l t e».
Laid out it would look something like:
aο¬ liate mark
We know that when we rendered ο¬ we got back a bitmap of a length less than the number of cells of its components. OK, we're not allowed to render the word all in one pass as a big sprite, due to performance cost. (I'm of the view this, along with making the max ligature length unlimited, could be made optional, contained inside a single option, but really it's a big hack and I won't be proposing a PR to do it, even as a non-default option recommended against.)
But why we can't bump the vertices around to make things look a little nicer, when cursor not over text? (In formal terms, assuming a quad of four vertices Q0…Q3, add a bump factor _b_ to the x axis of all four vertices.)
As I see it there are two possible ways to bump the vertices: either you bump them to remove the space where it is (“bump style A”), and the bump is always expressible in the form _b_ × cell_width, or you bump cells according to a formula (“bump style B”).
Assuming bump style A, the layout appears as:
aο¬liate mark
Cells 0 and 1 go unbumped, cells 2 and 3 are bumped to the end, every other cell until the end of the word is bumped backwards one cell.
Bump style B is not expressable in monospace terms, because it works by breaking the grid; cells are bumped in fractions of a cell. It assumes bump style A has already been applied.
So, the absolute value of strlen("aο¬liate") (_X_ = 7) - strlen("affiliate") (_Y_ = 9) is 2 (_S_). The bump factor, in number of cells, is easily expressed as S÷2=1.
So all cells which were not bumped to the end need to be bumped one cell:
aο¬liate mark
We can, and ought to I think, also include cell 9 in our calculation, after all, it is visible space. So, really, S=3 and we need to bump each cell one and one half cells.
Is this overkill and am I overthinking it? Could very well be. But if cells can be nudged around, don't see why it can't work.
ctrlcctrlv
on 16 Jan 2020
Rendering happens per cell, there is no vertex data for a cell. The
vertex locations on the screen are calculated in the shader from
instance id. See the section set cell vertex positions in the
cell_vertex.glsl shader.
You could of course have a separate array to displace cells that you
send to the GPU on every render, but that is a performance/code
complexity cost that will affect all rendering for a use case that is
not really that important. kitty is designed to render in character
grids. If I really wanted to support complex text shaping I would not
have made that design in the first place. The vast majority of monospace
fonts used in terminals implement ligatures that are the same width as
their constituent characters. Therefore, adding extra per cell data is
not worth it.
kovidgoyal
on 16 Jan 2020
I think this should be closed then. Improved spacing is really the best solution to this problem.
ctrlcctrlv
on 17 Jan 2020
The general problem of ligatures certainly. This particular issue however, might be solved by improving wcswidth calculations for flags. I have to look into it someday when i have time.
kovidgoyal
on 17 Jan 2020
No @kovidgoyal, not without making the flag itself have a widechar width of 0, but 🏳 on its own has a widechar width of 2.
Using the word "affiliate" is just an easier way to think about the general problem because all its components have a widechar width of 1, instead of 2-0-0-2 as in the rainbow flag ZWJ sequence.
ctrlcctrlv
on 17 Jan 2020
yes the idea would be for wcswidth to know about flags sequences. I dont know how feasible that is, will have to see.
kovidgoyal
on 17 Jan 2020
It's not feasible if wcswidth(S) must always equal sum(wcwidth(C) for C in S); changing that will certainly break applications...wcwidth itself was always a hack :-)
ctrlcctrlv
on 17 Jan 2020
wcswidth is defintely not always equal to sum wcwidth() if it were then emoji variation selectors would not work. See the kitty implementation of wcswidth() in screen.c. The only reason for wcswidth to exist at all is that it is not in general equal to sum(wcwidth())
kovidgoyal
on 17 Jan 2020
Sorry to say, that if your internal implementation does not match that of glibc's, (which works how I explain,) the mismatch will lead to subtle rendering errors (unless Kitty somehow forces client applications to use its internal wcswidth? But anyway, most programmers familiar with the glibc implementation consider them always equal and wcswidth to be a mere convenience function, so I don't understand how this can work)
ctrlcctrlv
on 17 Jan 2020
And yet it does. Pretty much all advanced terminal applications use their own wcswidth() implementations, precisely because glibc's is a broken umm POS. glibc is not the canonical source for how to calculate widths, the unicode standard is. And kitty's wc(s)width is autogenerated from the unicode standard. Indeed using the system libc's wcwidth() is fundamentally a bad idea because it can be arbitrarily old and broken. Not to mention it can vary between systems when you ssh. Any serious terminal application needs to use a standards based implementation.
This has all been discussed before, ad nauseum, search this issue tracker of wcwidth
kovidgoyal
on 17 Jan 2020
Interesting :-)
Well in that case, yes, to my knowledge all emoji ZWJ sequences have a visual wcswidth of 2, but a glibc wcswidth of 4 or more.
I'm sorry to waste your time with repeated discussions
ctrlcctrlv
on 17 Jan 2020
No worries, and yes looking into modifying wcswidth for ZWJ/flags is why this issue remains open.
kovidgoyal
on 17 Jan 2020
Hi - is this the same issue I'm seeing when I do curl https://en.wttr.in/format=v2 ?
For some reason extra spaces are added after the sun emojis. I've tried this in a few other terminals (gnome-terminal, alacritty, st) and they all render this as expected. Also tried multiple different emoji fonts with the same result. This only seems to happen in kitty.
jsravn
on 14 Apr 2020
@jsravn: No, there is no zero width joiner or flags in that output. The reason you're seing extra spaces after the sun emojis is that the output actually contains extra spaces. This is probably done because most other terminals wrongly consider the sun emoji to be only one character wide, so that service has added spaces until it aligns in those terminals.
You can verify this by running this command:
echo -e "\u2600\ufe0f Sunny"
The sun emoji is the two escaped characters, and as you see, I've added a space between the emoji and the text. In kitty this renders as sun-emoji, space, text, which is the correct rendering. In all other terminals I've tried, except qterminal, you don't see the space.
If you remove the space and run `echo -e "u2600ufe0fSunny", you can see that the S appears on top of the sun emoji in the terminals you mention.
trygveaa
on 17 Apr 2020
Related issues
RedBeard0531
Β·
4Comments
jasminabasurita
Β·
3Comments
reed-jones
Β·
4Comments
JJGO
Β·
3Comments
skosch
Β·
3Comments
Most helpful comment
@jsravn: No, there is no zero width joiner or flags in that output. The reason you're seing extra spaces after the sun emojis is that the output actually contains extra spaces. This is probably done because most other terminals wrongly consider the sun emoji to be only one character wide, so that service has added spaces until it aligns in those terminals.
You can verify this by running this command:
The sun emoji is the two escaped characters, and as you see, I've added a space between the emoji and the text. In kitty this renders as sun-emoji, space, text, which is the correct rendering. In all other terminals I've tried, except qterminal, you don't see the space.
If you remove the space and run `echo -e "u2600ufe0fSunny", you can see that the S appears on top of the sun emoji in the terminals you mention.