Runtime: Please add APIs to lowercase a UTF32 code point

[gafter]

To perform case-independent identifier comparison for VB, we case map to lowercase (as recommended by the Unicode specification). We use System.Globalization.CultureInfo.ToLower(char c). Unfortunately, there is no version of this API that works on UTF32 code points (that result from surrogate pairs).

Please add such an API.

[tannergooding]

@gafter, can you not use string char.ConvertFromUtf32(int utf32) and then string.ToLower(CultureInfo)?

An overload that directly took int utf32 or char highSurrogate, char lowSurrogate would probably be more performant, however.

[stephentoub]

You should be able to use:
C# public static int ToLower (this ReadOnlySpan<char> source, Span<char> destination, System.Globalization.CultureInfo culture);
from MemoryExtensions, even if a bit cumbersome for this specific use-case.

[gafter]

@tannergooding That would require a string allocation on every use.

@stephentoub Can you please tell me more about this API? Where do I find the docs, and in what platform is it? Does it return the new code point? Or do I have to decode the resulting surrogates in the destination myself?

[gafter]

@stephentoub That API is not usable from VB because of the use of Span. What API should I use from VB?

[GrabYourPitchforks]

We also have this exact API proposed as part of the UTF-8 work (via the static method UnicodeScalar/Rune.ToLower).

In your scenario, is it possible to perform sequence comparisons instead of char-by-char comparisons? That is, if a method bool Equal(ReadOnlySpan<char> identifier1, ReadOnlySpan<char> identifier2, CultureInfo culture, bool caseSensitive) were available, would you be able to call that instead?

[stephentoub]

That API is not usable from VB because of the use of Span. What API should I use from VB?

From VB, either the ones Tanner suggests, requiring a string allocation, or fix VB to allow using span ;), or create an int-to-int wrapper in C# that uses this under the covers. Or wait for the new APIs Levi mentions.

[GrabYourPitchforks]

You can also use this String.Compare overload if you happen to have a backing string and all you're trying to do is check two substrings for equality in a case-insensitive fashion using a specific culture. It'll be faster than iterating character by character.

[stephentoub]

Can you please tell me more about this API? Where do I find the docs, and in what platform is it? Does it return the new code point? Or do I have to decode the resulting surrogates in the destination myself?

It's the equivalent of String.ToLower(CultureInfo), except the input is a ReadOnlySpan<char> instead of a String, and the output is written to the destination Span<char> instead of to an allocated string. It's intended for use where String.ToLower was previously being used but where you want to avoid allocation, which is why I said it'd be a bit cumbersome for your desired use-case of targeting a single UTF32 codepoint; you would need to take the two resulting chars (presumably that you stackalloc'd into a Span<char>) and combine them yourself.

[gafter]

@GrabYourPitchforks Our compilers do not allocate backing strings for the source code. We use these APIs as part of identifying where the identifiers begin and end, so we do not know ahead of time what sequence we are working with.

[tannergooding]

That would require a string allocation on every use.

@gafter, only in the scenario where you had a surrogate (this would allow the scenario to be handled now and be improved later when a non-allocating API was added). Given that surrogate characters are the exception, not the norm, I wouldn't imagine it wouldn't be terrible as a first iteration.

You can check if the character needs surrogate handling using char.IsSurrogate (there are (string s, int index) and (char high, char low) overloads). There is also a char.IsSurrogate(char) overload for checking if it is an upper/lower surrogate character (if all you have is a stream and need to check characters individually).

[CyrusNajmabadi]

@gafter BasicCodeAnalysis has a dependency on CodeAnalysis. So couldn't you write any helpers you needed tehre in C#, and then just reference them from BasicCodeAnalysis? That would make it so you could use any sort of ref/span/whatever API and still have thigns work from VB.

[stephentoub]

Until Levi's new UnicodeScalar stuff is available, this is what a helper on top of MemoryExtensions.ToLower might look like:
C# public static void ToLower(ref char high, ref char low, CultureInfo culture) { Span<char> src = stackalloc char[2] { high, low }; Span<char> dst = stackalloc char[2]; MemoryExtensions.ToLower(src, dst, culture); high = dst[0]; low = dst[1]; }

[GrabYourPitchforks]

@gafter The API Rune.ToLower is now checked in. There's a culture-invariant overload and a culture-aware overload. Please let me know if this works for your use case.

[gafter]

Perfect!

[stephentoub]

@GrabYourPitchforks, can we close this now?

[GrabYourPitchforks]

Yup!