Cppcoreguidelines: New guidelines regarding characters and strings

[ParadineG]

I will say out multiple possible guidelines, might have gotten some wrong, because not reading with care. But all you can improve them or discard. Under portability I also mean some sort of direct communication between two programs on two different platform.

1. Use Unicode ( u8"", u"" and U"") instead of implementation defined( "" and L"") characters and strings
   Reason: Portability.
2. Don´t compare characters of different type with each other.
   Reason: They are likely to have different values.
3. Don´t compare u8"" and "" strings between each other.
   Reason: They might not be same.
4. Don´t compare signed, unsigned and plain characters and strings with each other.
   Reason: They might not be same.
5. Prefer to use same type of character and string for one project
   Reason: Its easy to accidentally mix them up.
6. Prefer to have characters unsigned?
   Reason: Portability. Plain char is implementation defined.
7. Don't use byte/single character based string manipulations on variable length encodings like UTF-8, UTF-16.
   Exception: UTF-32 as it is fixed-length?
   Reason: Letter might be combination of multiple characters.
8. Care with endianness for UTF-16 and UTF-32?
   Reason: Portability

[alphaONE2]

UTF-32 is only "fixed length" in the sense that it maps every possible Unicode code point to a single code unit. A single _grapheme_ can (and in many cases _must_) still be represented by multiple code points.

Examples:

• ö can be respresented as either U+00F6 (SMALL LETTER O WITH DIAERESIS) or as U+006F (LATIN SMALL LETTER O) followed by U+0308 (COMBINING DIEARESIS). Both forms should be considered equivalent.
• नि is U+0928 (DEVANAGARI LETTER NA) followed by U+093F (DEVANAGARI VOWEL SIGN I).

[MikeGitb]

Although I like the general direction, there are a few problems, when it comes to the details:

1. Use Unicode ( u8"", u"" and U"") instead of implementation defined( "" and L"") characters and strings
   Reason: Portability.

The windows API (and consequently all std functions/classes that use it) doesn't support utf8, but requires either ANSI with codepages or UTF-16 encoding. That effectively makes

1. Prefer to use same type of character and string for one project
   Reason: Its easy to accidentally mix them up.

almost impossible to follow on windows. I'm no expert in this, but I think what one should strife for here is to clearly separate what parts of a program us what kind of strings (e.g. core logic and network facing stuff only use utf8. System-API-facing stuff uses wchar).

I wonder if the general rule about handling unicode strings should be to use a library,that has actually full unicode support.

1. Prefer to have characters unsigned?
   Reason: Portability. Plain char is implementation defined.

I don't think that makes too much sense - in effect it would mean that you'd have to replace all occurrences of std::string, std::string_view, const char*, ostream/istream and string literals with their unsigned char counterparts and I don't think there even is a unsigned version of e.g. std::cout. All this for a very questionable benefit.

I think the rule should rather be: "Don't treat chars as numbers!". The only area, where I know this to be useful is translating numbers to digits and vice versa (int_var = char_var - '0' - which works correctly for both signed and unsigned). Vice versa, you imho shouldn't use plain char for anything but characters: If you need a 8 bit number use unsigned char or signed char and if you need raw bytes use either gsl::byte or if that is not possible uint8_t (You can argue whether this should be unsigned char or uint8_t, but I consider it a pro that this will give you a compiler error on platforms that don't have 8bit bytes).

[tamaskenez]

That effectively makes [using the same type of character and string for one project] almost impossible to follow on windows.

It's possible, there's a small library, nowide that provides utf8 versions of standard library functions/classes so you can use utf8 everywhere.

[ParadineG]

Thank You @alphaONE2 for explaining it in detail.
Thank You @tamaskenez for pointing example out.
Thank You @MikeGitb for feedback.
Yes, there are many APIs that use other than Unicode. And you can always have them converted whether through wrappers, third party library or some middleware between.
Yes because this will introduce for some programs having additional encoding forms. It would require have parts of program have different encoding. In that case clear code separation would be required. Like between portable part and non-portable on.

1. If you can use same type of character and string for one project.
   Reason: Its easy to accidentally mix them up.
   Exception: If you need to use more because external factors, separate encodings clearly from each other in different parts of code.

About whether char should be signed, unsigned or plain. Plain char really is best solution. I simply thought it needed to be talked out for this guideline once more.
Your proposed rule is good, but we could have one better.

1. Don´t use char beyond control characters and basic Latin set.
   Reason: portability
   Exception: If you need to use extended Latin set or multi-byte characters use strings instead single characters.

u8"" form will actually enforce it automatically, but only sometimes "" form. You can´t have problem with sign if you don´t need to use one.
Thought I don´t no how it should affect other types character type like wchar and other Unicode characters.

[AndrewPardoe]

We could potentially mention comparing char to integer in the guidelines.

For all of these, we could potentially have a rule to prefer UTF-8 when you need something beyond ASCII. Don't use byte/single character manipulations when not working with ASCII.

Should we consider a gsl::Unicode or gsl::Code_point to make it obvious when you're dealing with a code point versus a numerical value? Specifically a code point type that's independent of encoding.

Neil, you have a contact who does quite a bit of work in character sets? It would be great to get a minimal subset of rules and potentially a type in the GSL.

(Edit: corrected reference from char_utf8_t to a GSL type.)

[cubbimew]

technically, char is already a UTF-8 code unit type, regardless of what encoding a given program may reinterpret it as: by definition in [intro.memory] and as the type of u8'x' and as the element type of u8"貓".

The guidelines are about how we want C++ to look in the near-term future. Why not just say "use of the type char for any character encoding other than UTF-8 belongs in the 1980s"?

[neilmacintosh]

@tahonermann do you have a view here?

[tahonermann]

Thanks for the invite to the discussion @neilmacintosh!

I think the proposed guidelines are pretty reasonable. I very much like the spirit of 2, 3, 4, 7, and 8, though I think the phrasing could be updated to place more emphasis on encodings and character sets. And yes, UTF-32 is a fixed length encoding (grapheme clusters are a higher level abstraction).

1. Use Unicode ( u8"", u"" and U"") instead of implementation defined( "" and L"") characters and strings
   Reason: Portability.

I think this could be more concisely expressed as "Use Unicode" combined with "don't assume that the ordinary ("") and wide (L"") string encodings are Unicode. I've seen many cases of developers trained on Windows that believe wide (L"") strings are UTF-16 everywhere. They aren't. Similarly, developers trained on modern Linux systems are subject to bias towards ordinary ("") strings being UTF-8 everywhere. They aren't (at least, not yet).

1. Prefer to use same type of character and string for one project
   Reason: Its easy to accidentally mix them up.

This is a hard one. Encoding restrictions are generally imposed by requirements. In general, I agree that an independent body of work should be optimized around a single internal encoding and transcode as necessary to meet other requirements. However, this can lead to performance issues. I think libraries compose more naturally and run more efficiently if they are flexible with regard to encoding requirements. But that can also lead to interfaces like Microsoft's TCHAR based ones. Such interfaces work, but have obvious drawbacks.

There have been suggestions to introduce the notion of a "system" encoding that libraries could be optimized for. Presumably, for modern Linux systems this would be UTF-8 and for Windows, UTF-16 (though Microsoft has been investing more in UTF-8 recently). I think there isn't yet consensus on what the right approach is here.,

1. Prefer to have characters unsigned?
   Reason: Portability. Plain char is implementation defined.

The code units of ordinary strings are plain char, so plain char can't really be avoided. I agree with avoiding use of plain (and signed) char as the underlying code unit type of custom encodings though.

[neilmacintosh]

Thanks very much to everyone who contributed their expertise in this discussion!

@ParadineG would you like to put together a PR that reflects the comments from @tahonermann? We're looking for something fairly concise here, if possible.

[neithernut]

Although the overall idea of the proposal seems reasonable, interacting with the OS may still require a different encoding. Statements like:

That effectively makes [using the same type of character and string for one project] almost impossible to follow on windows.

It's possible, there's a small library, nowide that provides utf8 versions of standard library functions/classes so you can use utf8 everywhere.

(from @tamaskenez) are just not true, at least not if a file system is involved.

For example, consider some idiot storing files using "broken" file names. Either because he/she messed up or the operating system used decided to store file names in something like Latin1. Given external HDDs and people using outdated OSses and/or broken configurations, this is surprisingly common.

If you list directories and query a file's path, you usually get the bare bytes making up the file name from the operating system. It may be complete garbage if viewed as a string of arbitrary encoding, but you can still use it as a _handle_ to open the file. However, when processing/parsing the string as UTF-8, you may get anything ranging from complete garbage to strings which may contain sequences referring to invalid code points. Generally, you will not be able to reconstruct the original byte sequence, even if you know the original encoding.

Now consider a application storing file names as UTF-8 and a bunch of files exhibiting file names with an invalid encoding. Depending on how badly the software is written, it may crash either trying to parse the string, trying to present the file name to the user and or storing it in a database, or trying to open() the file based on a file name. Even if it doesn't crash, it may be impossible to open the file (although it is there) when you convert the file name to UTF-8 and back.

Hence, I'd always advocate keeping around file names (and other strings effectively serving as a handle) as received from the OS. In most cases though, you will probably still convert them to UTF-8, e.g. for presenting them to the user.

[tahonermann]

Hence, I'd always advocate keeping around file names (and other strings effectively serving as a handle) as received from the OS. In most cases though, you will probably still convert them to UTF-8, e.g. for presenting them to the user.

I agree with regard to retaining the original code units of the file name. For presentation purposes, file names should be interpreted as though encoded in the current (locale sensitive, run-time) execution encoding.

[tamaskenez]

Statements like:

[...] you can use utf8 everywhere.

are just not true, at least not if a file system is involved.
For example, consider some idiot storing files using "broken" file names.

I guess we all agree that both using utf8 everywhere or using OS-specific encodings / opaque handles have their pros and cons. The question is only how we weigh those.

Here I value simplicity over supporting broken files because broken file names are just like physical defects on the disk, should be handled by a disk repair tool. General apps should not be more complex just because we wanted to implement a tiny bit of disk repair tool functionality in them.

[cubbimew]

I have a lot of "broken" (encoded in something other than UTF-8) file names on my linux box and didn't find a way to work with them other than by manual "repair": changing the locale breaks display of good filenames and filesystem-level encoding option is even more coarse. Would be nice if there was a file manager with the option to choose filename encoding for a given file..

But doesn't C++ now have path::u8path for system-independent view of a file name and path::native for use as an opaque handle to OS API?