Epsilon: Accents and Diacritical Marks

[vardai]

We really need to address accents and diacritical marks in Epsilon, the sooner, the better. Why? _Because it is the right thing to do,_ and there really is no alternative if we want this project to succeed in the long term. The important question how to do it properly, if the current structure of translations is adequate or how it has to be improved.

Please comment or react if you are in favor.

[boricj]

While I would not say that the particular issue of accents and diacritics is a life-or-death issue for epsilon, the wider topic of i18n does warrant a discussion.

Epsilon's charset is currently 8-bit ASCII+proprietary set of ~24 symbols for codepoints above 0x7F. There's not enough flash for non-latin languages and all their glyphs, but even when reducing scope to western languages there are several things to consider before extending the charset:

• Keep symbols on proprietary 8 bits charset or move to "sparse" UTF-8,
• Use combined or combining characters,
• Add mechanism to input non-ASCII characters or keep them for UI only.

Personally I'd go for a sparse UTF-8 with combining characters for UI translations only and 7-bit ASCII+proprietary glyphs moved into ASCII control characters for everything else. It gives a unified system with the needed breathing room for translations while not forcing the entire system into variable byte-length encodings. This scheme should keep things under control so that compressing fonts would not be necessary in the short term to preserve Flash, but Python scripts from older versions might require a one-time conversion to the new scheme.

On the matter of translations, they are only in-tree. Third-party languages are condemned to bitrot in forks as shown time and time again, so a mechanism for out-of-tree translations like third-party apps is needed. Since out-of-tree translations are bound to lag behind, a mechanism for optional sloppiness (missing translations being substituted by the first language) to not error out is probably needed too.

[Ecco]

You definitely have a good point @vardai. Here the limiting factor is indeed the size in memory of rasterized glyphs, just like @boricj said, and he also raises good questions.

In the short term, translation files should use full-blown UTF-8, and the conversion script should do the job of finding what is the best transliteration available given the current alphabet useable by Epsilon.

This allows to decorelate the technical limitations of Epsilon with the localizers' contributions. Localizer can say what the best output shall be, and then Epsilon does whatever it can to match the localizer's text.

For example, at the moment, since we only support ASCII characters, this script removes diacritics (é becomes e). I think a good first step would be to tweak this script to handle special characters for German.

Then of course a second step would be to actually support all characters, but that's a lot more difficult 😄

[vardai]

Yeah, @Ecco, another problem is that eg. in German, umlauts are substituted differently than in any other language I know of.

• ß to ss
• ä to ae, and Ä to AE, except where at the beginning of a word Ae
• ö to oe, and Ö to OE, except where at the beginning of a word Oe
• ü to ue, and Ü to UE, except where at the beginning of a word Ue

Another point is, as calculators are traditionally mostly used by school children, correct translation is important if you want to stay relevant and sell devices.

[Ecco]

Correct localization is extremely important indeed.

Some of these rules you laid out are indeed German-specific, but that very well could (should, actually) be handled by the script (i.e. "if lang == de then"). Are there any other rules for German? What is the best job one could do to display German text using only ASCII letters ?

[vardai]

For German, it would be easy, but in reality you have to know if the word is in uppercase or not, because there is a difference.

• ä to ae
• ö to oe
• ü to ue
• Ä to AE or Ae
• Ö to OE or Oe
• Ü to UE or Ue
• ß to ss
• ẞ to SS or Ss (this is a new letter actually introduced last year, good luck in dealing with it)

Additionally there are foreign letters handled just like in French (á to a, é to e, etc.).

[mhaupt]

I agree with @vardai's comments - this is a good way to represent special characters in German texts if such characters aren't available.

@vardai - I assume with the final "SS" you mean the upper-case "ß", which is indeed a new one. :-)

The test script looks like a good place to address the transcription.

In the long run, I'd still prefer "correct" display of umlauts. While I appreciate @boricj's points about memory consumption, is it feasible to think about storing all special glyphs in compressed form and only extracting the required ones (a set used for the language at hand) when a language is selected? (I may well be nerding off here. Please tell me. I'm new to this project, and don't yet understand the hardware limitations.)

[boricj]

@mhaupt The NumWorks calculator has 1 MiB of NOR Flash, 256 KiB of SRAM and a dedicated external framebuffer just big enough to hold a 320x240x16 screen. That's not enough for full-blown TTF Unicode fonts like the HP Prime, but with some tricks we can accommodate Latin scripts with diacritics at very little cost. The main idea is not to store all hundreds of characters in the Latin Unicode blocks but to use combining diacritical marks to draw accented characters by composing them on-the-fly. We can keep the 8-bit proprietary charset while doing that, but the next logical step would be to introduce limited Unicode and UTF-8 support.

Anyway, in the 8-bit version it's something that should be quite easy to prototype. There's really no reason to mess around with substitutions rather than doing the right thing in the first place.

[boricj]

I've made a pull request demonstrating a (quick and dirty) way to achieve diacriticals in epsilon (#689). It appears nearly all relevant translations lack accents in the translation files, but those do present did work out of the box.

Check the pull request for details but please, switch to Unicode and UTF-8 before somebody gets hurt.

[vardai]

Commenting on @boricj and @mhaupt, it is clear that we need to move, in one direction or the other:

• concentrating on the latin alphabet and combining characters on the fly or
• compressing and only decompressing certain subset of characters that are needed for selected language.

There is another thing no one has mentioned so far: what happens if the hardware is to be sold and used in other parts of the world, e.g. Japan or China? They have a far greater variety of characters that need to be incorporated, and cannot be put together by combination. We also should look at the problem by work needed in the future. Those with deep knowledge of the current and planned hardware should decide how to move forward.

In my opionion it is better to go the long-term route and do it as @mhaupt suggested. Parse the language files and uncompress only those needed for the selected language.

I've put together a list of characters needed for each language. Maybe others can come forward with additional languages.

• French:
  ÀàÂâÆæÇçÉéÈèÊêËëÎîÏïÔôŒœÙùÛûÜüŸÿ
• German:
  ÄäÖöÜüẞß
• Hungarian:
  ÁáÉéÍíÓóÖöŐőÚúÜüŰű

[boricj]

The current NumWorks hardware clearly doesn't have the flash capacity to store tens of thousands of han glyphs. There's a placeholder on the PCB for an external NOR chip and the community already successfully experimented with it, so an upgraded model can have the hardware guts to support East Asian scripts.

The other thing to remember is that epsilon uses a proprietary 8-bit charset based on ASCII and there's about a hundred codepoints left. Even just adding Cyrillic or Greek letters would quickly eat up the remaining slots, which is why eventually switching to Unicode and UTF-8 is important beyond "it's not the 80s anymore".

While compressing images and glyphs will free a significant amount of Flash to invest in more glyphs, we will hit the 256 codepoints limit before running out of Flash. Each codepoint costs 400 bytes right now (12^2+16^2), but even basic compression techniques would drop the average codepoint cost by so much that supporting hundreds (if not a thousand) of codepoints with the existing hardware would not be an issue.

[Ecco]

We've been thinking about this, and we're most likely going to go with a custom charset based on Latin1 as a first step. We fully understand why UTF-8 would be great in theory, but in practice we believe it might not be the optimal solution at the moment.

String encoding really matters whenever you're interacting with the outside world. In Epsilon's case, the outside world is a USB host. And it turns out we decided to go for a peek/poke approach: the host can read and write to random memory location on the device running Epsilon. That's how we transfer Python scripts for example.

This approach has three huge upsides:

1. It uses a very simple, open and documented protocol
2. It's very simple to implement on the device
3. The host gets to do the heavy lifting and as a result can do pretty much whatever it wants without explicit support from the device.

Of course, this comes with a big theoretical drawback: the host could wreak havoc in the device's memory space.

Pretty much the same reasoning could be applied to string encoding: sure UTF-8 would be theoretically better. But in practice, it's extremely convenient to have a fixed-length encoding and to use only a single byte per codepoint.

Of course there is a drawback: it won't scale to non-latin scripts (Asian, Cyrilic, etc…). But on the other hand it's very simple to implement and would already make the situation a lot better for a significant part of our user base.

That's why we believe a custom charset (based off Latin1 so as not to reinvent the wheel too much) seems like a good compromise for a first step.

[Ecco]

Another thing comes to mind: it'd be great to have support for non-ascii characters (no matter what encoding is settled on) in the source. Sure we could use a generator, it'd be nice to be able to write things such as assert_simplifies_to("0", "π×0") straight in the source code.

[boricj]

@Ecco I'm not convinced the 8-bit proprietary charset is the way to go here, even if it will take a bit of work to accomplish UTF-8. Latin1 is known to lack lots of accents/characters for western languages and precomposed characters take up a lot of glyphs and codepoints to handle.

C++ can embed UTF-8 literals, so things like assert_simplifies_to(u8"0", u8"π×0") are indeed possible. Variable-length encodings are problematic with raw pointers, but a wrapper class with enough syntactic sugar can act as a drop-in substitute. Also, not all components in epsilon actually care about individual codepoints, we don't have to rework all string code to switch from char* to UTF-8-aware to make this work (things like https://github.com/numworks/epsilon/blob/master/apps/settings/sub_menu/about_controller.cpp#L53 or simple strlen()/strcat()/strcpy() manipulations don't need modifications).

I get that converting a code base to UTF-8 is not a very interesting thing to work on nor will most users even notice the difference. However, epsilon is not a closed system and the USB host link is a gateway to the rest of the entire world. Things do break and the host will hose the device when the gateway is not 100% foolproof (#698) and the road to encoding gateways is paved with good intentions...

[Ecco]

The big problem I have with using UTF-8 is not UTF-8 _per se_. Variable-length encoding sure is a bit more complicated, but that can be dealt with. That being said, what I think would be very hard is to actually figure out a clean and nice subset of Unicode features to support.

For example, if you accept UTF-8 input, you shall be ready to encounter:

• Normalization problems : U+2126 and U+03A9 should both render as uppercase omega.
• Exotic glyph layouts : right-to-left text, some asian scripts that are supposed to be laid out vertically, or sometimes one on top of another

Sure, you could decide on some rules (e.g. "only use NFKD normalized string, and avoid codepoints in the range XXX...YYY"), but that seems really complicated.

What's great with a custom charset is that all the assumptions are clear : you only support a given set of glyphs, which will be laid out left-to-right, which will all have the same length, etc… It not only makes the code smaller, but it also makes the "contract" a lot more simple.

[boricj]

The issues you've listed are about _full_ Unicode support (and to a lesser extent fonts), which we absolutely _shouldn't_ attempt.

There's no reason to implement support for bi-directional text, ligatures, zero width joiners on emojis and other exotic/insane features as long as epsilon's font doesn't have glyphs that require those. After all, why would we implement right-to-left text rendering if there are no Arabian (and other applicable scripts) glyphs in our font?

What we do need to do when adopting Unicode is explicitely define the set of supported codepoints. Then, when a script gets transferred to the calculator, the workshop only needs to perform NFD, check if any codepoints in a script are not in that set and warn the user that there are unsupported codepoints in the script (which would've prevented #698 even with a proprietary charset).

Now, if people do upload unsupported Unicode codepoints to the calculator despite our warnings, all they shall get is � for each unknown codepoint on rendering. Just like on a desktop computer.

_tl;dr_ Just because we add Unicode support doesn't mean we must implement support for things not in our supported charset (i.e. � on any unrecognised codepoint).

[vardai]

I see implementing a subset of Unicode as a great opportunity. Even if we wanted to add Japanese or another non-latin language, we would only need those characters actually used in the translation.

[Ecco]

the workshop only needs to perform NFD, check if any codepoints in a script are not in that set and warn the user that there are unsupported

I definitely agree: in both cases, there would need to be a "translation" layer. And that's precisely what's bugging me: it seems easier to say "hey we use a custom encoding" rather than "well, we support Unicode, but be careful, only a subset. So many things will work but you still really have to pay attention to what you're throwing at the device". But I agree that might not be such a big deal…

[Ecco]

Custom charset

Pros

• Very easy to implement
• Most efficient at runtime (1 glyph == 1 byte)
• The interface with the outer world is clear: a translation is required

Cons

• Non-standard
• A translation is required
• Will not scale well if we want more than 255 glyphs
• Cannot support separate diacritics as-is
• Requires a translation at the source level to write things such as assert_simplifies_to("0", "π×0")

UTF-8 + Partial Unicode support

Pros

• Uses a standard
• Future-proof
• Has built-in support for separate diacritics if we want to use them (à can be à or a + `)
• Wipes the need for a translation at the source level (assert_simplifies_to("0", "π×0") can be expected to work as-is).

Cons

• One character can be >1 byte. Shouldn't make much of a difference though since UTF-8 is backwards-compatible with 7-bits ASCII.
• Still needs a translation layer, which is somewhat non-obvious (but standardized).
• Text rendering is more complicated. Namely, the glyph lookup will be more complicated than something along the lines of Glyph g = GlyphTable[codepoint];.

[Ecco]

Quick question: how would you handle "x̅"? It doesn't exist as a single Unicode codepoint, so in that case one MUST add support for sequences… I guess we could use a codepoint in a Private Use Area. At least someone did already and assigned 0xE75E to LATIN SMALL LETTER X WITH MACRON

[boricj]

Use combining characters, like so: U+0078 U+0304 = x̄

We will want to support combining characters anyway because that's the only sane way to handle accents and diacriticals ; precomposed characters will consume egregious amounts of Flash we can't spare. We don't have to overly complicate said support of combining characters, as my PoC demonstrated that a global offset for capital characters and a global offset for everything else works fine in practice.

Now, there will most likely be proprietary characters that aren't available at all in Unicode, at the very least the empty symbol for EmptyExpression (but who knows, maybe there's one for that in all those 137,000+ characters). We should use Private Use Areas for these and pretend they are not officially supported in the workshop.

[RubenNumworks]

UTF-8 is supported since recently and a reasonable subset of glyphs is provided.
See https://github.com/numworks/epsilon/tree/master/ion/include/ion/unicode and https://github.com/numworks/epsilon/tree/master/ion/src/shared/unicode.