Rack: Does not support UTF-8 (non-ASCII-7) strings on Windows

[odahan]

OS (if applicable): W10
Version (or "dev" if compiling from source): 0.5

Notes module does not take into account letters like éàçÏô etc... Worse, it does strange things when such letters are typed (like cursor going backward).

[Strum]

An unfortunate choice of words for your topic there @odahan

i'll just leave it at that.

lol

[odahan]

@Strum I don't get it :-( My english is far from perfect but I'm always pleased to learn something new. It is out of topic here, but if you can explain I'll learn something new thanks to this bug :-)

[Strum]

@odahan

https://www.urbandictionary.com/define.php?term=french%20letter

[odahan]

LOL. What a strange expression :-) Thanks I know something new and useful, I will use "French characters" instead (don't say me it means another strange thing !).

[Strum]

@odahan , I think you're right with that.

:)

English is a funny language.

[odahan]

English is a funny language.
French too... a world of rules but with as many exceptions than regular cases !

[AndrewBelt]

The issue here is that TextField only treats ASCII-7 for now.

[odahan]

Annoying.

[AndrewBelt]

Feel free to send a PR. (But first wait a few days because I have some large changes that would make merges difficult at the moment.) I don't have time to fix it for 0.6, maybe 1.0.

[odahan]

I know until 1.0 there are more important things to do and fix.
I was not suspecting a modern app like VCV was using libraries not having native unicode support, that's why I was reporting this as a bug.

[AndrewBelt]

Simply using libraries with UTF-8 support does not automatically give you UTF-8 support.
The relevant code if you want to design/research a solution is in https://github.com/VCVRack/Rack/blob/master/src/ui/TextField.cpp

[HSBallina]

Wouldn't it be possible to point the Rack-folders somewhere else than having it pointing to the documents folder? For me, as for many other users (I presume), this is kind of a showstopper. Or am I missing something obvious here?

[AndrewBelt]

@HSBallina Yes, see https://vcvrack.com/manual/Installing.html#command-line-usage

[nesdnuma]

The issue is still present on Rack 1.1.3/Windows 10.

[AndrewBelt]

As of several months ago, Rack v2 fully supports UTF-8 on Windows by converting to/from UTF-16 when interacting with all Windows APIs.

Attention all Rack plugin developers: All std::string and char* strings are considered UTF-8 regardless of platform. In addition, fopen() is hijacked to actually call fopen_u8(), a custom function I wrote to handle UTF-8 filenames. For all other OS functions such as the Windows API, fstream, and libraries you might use that don't automatically handle UTF-8 strings, you must convert to/from UTF-16 and then call the wide versions of Windows functions, using the following functions in the rack::string:: namespace:

std::string U16toU8(const std::wstring& w);
std::wstring U8toU16(const std::string& s);