Corefxlab: Utf8String design proposal

[GrabYourPitchforks]

Utf8String design discussion - last edited 14-Sep-19

Utf8String design overview

Audience and scenarios

Utf8String and related concepts are meant for modern internet-facing applications that need to speak "the language of the web" (or i/o in general, really). Currently applications spend some amount of time transcoding into formats that aren't particularly useful, which wastes CPU cycles and memory.

A naive way to accomplish this would be to represent UTF-8 data as byte[] / Span<byte>, but this leads to a usability pit of failure. Developers would then become dependent on situational awareness and code hygiene to be able to know whether a particular byte[] instance is meant to represent binary data or UTF-8 textual data, leading to situations where it's very easy to write code like byte[] imageData = ...; imageData.ToUpperInvariant();. This defeats the purpose of using a typed language.

We want to expose enough functionality to make the Utf8String type _usable_ and _desirable_ by our developer audience, but it's not intended to serve as a full drop-in replacement for its sibling type string. For example, we might add Utf8String-related overloads to existing APIs in the System.IO namespace, but we wouldn't add an overload Assembly.LoadFrom(Utf8String assemblyName).

In addition to networking and i/o scenarios, it's expected that there will be an audience who will want to use Utf8String for interop scenarios, especially when interoperating with components written in Rust or Go. Both of these languages use UTF-8 as their native string representation, and providing a type which can be used as a data exchange type for that audience will make their scenarios a bit easier.

Finally, we should afford power developers the opportunity to improve their throughput and memory utilization by limiting data copying where feasible. This doesn't imply that we must be allocation-free or zero-copy for every scenario. But it does imply that we should investigate common operations and consider alternative ways of performing these tasks as long as it doesn't compromise the usability of the mainline scenarios.

It's important to call out that Utf8String is not intended to be a replacement for string. The standard UTF-16 string will remain the core primitive type used throughout the .NET ecosystem and will enjoy the largest supported API surface area. We expect that developers who use Utf8String in their code bases will do so deliberately, either because they're working in one of the aforementioned scenarios or because they find other aspects of Utf8String (such as its API surface or behavior guarantees) desirable.

Design decisions and type API

To make internal Utf8String implementation details easier, and to allow consumers to better reason about the type's behavior, the Utf8String type maintains the following invariants:

• Instances are __immutable__. Once data is copied to the Utf8String instance, it is unchanging for the lifetime of the instance. All members on Utf8String are thread-safe.

• Instances are __heap-allocated__. This is a standard reference type, like string and object.

• The backing data is __guaranteed well-formed UTF-8__. It can be round-tripped through string (or any other Unicode-compatible encoding) and back without any loss of fidelity. It can be passed verbatim to any other component whose contract requires that it operate only on well-formed UTF-8 data.

• The backing data is __null-terminated__. If the Utf8String instance is pinned, the resulting byte* can be passed to any API which takes a LPCUTF8STR parameter. (Like string, Utf8String instances can contain embedded nulls.)

These invariants help shape the proposed API and usage examples as described throughout this document.

[Serializable]
public sealed class Utf8String : IComparable<Utf8String>, IEquatable<Utf8String>, ISerializable
{
    public static readonly Utf8String Empty; // matches String.Empty

    /*
     * CTORS AND FACTORIES
     *
     * These ctors all have "throw on invalid data" behavior since it's intended that data should
     * be faithfully retained and should be round-trippable back to its original encoding.
     */

    public Utf8String(byte[]? value, int startIndex, int length);
    public Utf8String(char[]? value, int startIndex, int length);
    public Utf8String(ReadOnlySpan<byte> value);
    public Utf8String(ReadOnlySpan<char> value);
    public Utf8String(string value) { }

    // These ctors expect null-terminated UTF-8 or UTF-16 input.
    // They'll compute strlen / wcslen on the caller's behalf.

    public unsafe Utf8String(byte* value);
    public unsafe Utf8String(char* value);

    public static Utf8String Create<TState>(int length, TState state, SpanAction<byte, TState> action);

    // "Try" factories are non-throwing equivalents of the above methods. They use a try pattern instead
    // of throwing if invalid input is detected.

    public static bool TryCreateFrom(ReadOnlySpan<byte> buffer, out Utf8String? value);
    public static bool TryCreateFrom(ReadOnlySpan<char> buffer, out Utf8String? value);

    // "Loose" factories also perform validation, but if an invalid sequence is detected they'll
    // silently fix it up by performing U+FFFD substitution in the returned Utf8String instance
    // instead of throwing.

    public static Utf8String CreateFromLoose(ReadOnlySpan<byte> buffer);
    public static Utf8String CreateFromLoose(ReadOnlySpan<char> buffer);
    public static Utf8String CreateLoose<TState>(int length, TState state, SpanAction<byte, TState> action);

    // "Unsafe" factories skip validation entirely. It's up to the caller to uphold the invariant
    // that Utf8String instances only ever contain well-formed UTF-8 data.

    [RequiresUnsafe]
    public static Utf8String UnsafeCreateWithoutValidation(ReadOnlySpan<byte> utf8Contents);
    [RequiresUnsafe]
    public static Utf8String UnsafeCreateWithoutValidation<TState>(int length, TState state, SpanAction<byte, TState> action);

    /*
     * ENUMERATION
     *
     * Since there's no this[int] indexer on Utf8String, these properties allow enumeration
     * of the contents as UTF-8 code units (Bytes), as UTF-16 code units (Chars), or as
     * Unicode scalar values (Runes). The enumerable struct types are defined at the bottom
     * of this type.
     */

    public ByteEnumerable Bytes { get; }
    public CharEnumerable Chars { get; }
    public RuneEnumerable Runes { get; }

    // Also allow iterating over extended grapheme clusters (not yet ready).
    // public GraphemeClusterEnumerable GraphemeClusters { get; }

    /*
     * COMPARISON
     *
     * All comparisons are Ordinal unless the API takes a parameter such
     * as a StringComparison or CultureInfo.
     */

    // The "AreEquivalent" APIs compare UTF-8 data against UTF-16 data for equivalence, where
    // equivalence is defined as "the texts would transcode as each other".
    // (Shouldn't these methods really be on a separate type?)

    public static bool AreEquivalent(Utf8String? utf8Text, string? utf16Text);
    public static bool AreEquivalent(Utf8Span utf8Text, ReadOnlySpan<char> utf16Text);
    public static bool AreEquivalent(ReadOnlySpan<byte> utf8Text, ReadOnlySpan<char> utf16Text);

    public int CompareTo(Utf8String? other);
    public int CompareTo(Utf8String? other, StringComparison comparisonType);

    public override bool Equals(object? obj); // 'obj' must be Utf8String, not string
    public static bool Equals(Utf8String? left, Utf8String? right);
    public static bool Equals(Utf8String? left, Utf8String? right, StringComparison comparisonType);
    public bool Equals(Utf8String? value);
    public bool Equals(Utf8String? value, StringComparison comparisonType);

    public static bool operator !=(Utf8String? left, Utf8String? right);
    public static bool operator ==(Utf8String? left, Utf8String? right);

    /*
     * SEARCHING
     *
     * Like comparisons, all searches are Ordinal unless the API takes a
     * parameter dictating otherwise.
     */

    public bool Contains(char value);
    public bool Contains(char value, StringComparison comparisonType);
    public bool Contains(Rune value);
    public bool Contains(Rune value, StringComparison comparisonType);
    public bool Contains(Utf8String value);
    public bool Contains(Utf8String value, StringComparison comparisonType);

    public bool EndsWith(char value);
    public bool EndsWith(char value, StringComparison comparisonType);
    public bool EndsWith(Rune value);
    public bool EndsWith(Rune value, StringComparison comparisonType);
    public bool EndsWith(Utf8String value);
    public bool EndsWith(Utf8String value, StringComparison comparisonType);

    public bool StartsWith(char value);
    public bool StartsWith(char value, StringComparison comparisonType);
    public bool StartsWith(Rune value);
    public bool StartsWith(Rune value, StringComparison comparisonType);
    public bool StartsWith(Utf8String value);
    public bool StartsWith(Utf8String value, StringComparison comparisonType);

    // TryFind is the equivalent of IndexOf. It returns a Range instead of an integer
    // index because there's no this[int] indexer on the Utf8String type, and encouraging
    // developers to slice by integer indices will almost certainly lead to bugs.
    // More on this later.

    public bool TryFind(char value, out Range range);
    public bool TryFind(char value, StringComparison comparisonType, out Range range);
    public bool TryFind(Rune value, out Range range);
    public bool TryFind(Rune value, StringComparison comparisonType, out Range range);
    public bool TryFind(Utf8String value, out Range range);
    public bool TryFind(Utf8String value, StringComparison comparisonType, out Range range);

    public bool TryFindLast(char value, out Range range);
    public bool TryFindLast(char value, StringComparison comparisonType, out Range range);
    public bool TryFindLast(Rune value, out Range range);
    public bool TryFindLast(Rune value, StringComparison comparisonType, out Range range);
    public bool TryFindLast(Utf8String value, out Range range);
    public bool TryFindLast(Utf8String value, StringComparison comparisonType, out Range range);

    /*
     * SLICING
     *
     * All slicing operations uphold the "well-formed data" invariant and
     * validate that creating the new substring instance will not split a
     * multi-byte UTF-8 subsequence. This check is O(1).
     */

    public Utf8String this[Range range] { get; }

    public (Utf8String Before, Utf8String? After) SplitOn(char separator);
    public (Utf8String Before, Utf8String? After) SplitOn(char separator, StringComparison comparisonType);
    public (Utf8String Before, Utf8String? After) SplitOn(Rune separator);
    public (Utf8String Before, Utf8String? After) SplitOn(Rune separator, StringComparison comparisonType);
    public (Utf8String Before, Utf8String? After) SplitOn(Utf8String separator);
    public (Utf8String Before, Utf8String? After) SplitOn(Utf8String separator, StringComparison comparisonType);

    public (Utf8String Before, Utf8String? After) SplitOnLast(char separator);
    public (Utf8String Before, Utf8String? After) SplitOnLast(char separator, StringComparison comparisonType);
    public (Utf8String Before, Utf8String? After) SplitOnLast(Rune separator);
    public (Utf8String Before, Utf8String? After) SplitOnLast(Rune separator, StringComparison comparisonType);
    public (Utf8String Before, Utf8String? After) SplitOnLast(Utf8String separator);
    public (Utf8String Before, Utf8String? After) SplitOnLast(Utf8String separator, StringComparison comparisonType);

    /*
     * INSPECTION & MANIPULATION
     */

    // some number of overloads to help avoid allocation in the common case
    public static Utf8String Concat<T>(params IEnumerable<T> values);
    public static Utf8String Concat<T0, T1>(T0 value0, T1 value1);
    public static Utf8String Concat<T0, T1, T2>(T0 value0, T1 value1, T2 value2);

    public bool IsAscii();

    public bool IsNormalized(NormalizationForm normalizationForm = NormalizationForm.FormC);

    public static Utf8String Join<T>(char separator, params IEnumerable<T> values);
    public static Utf8String Join<T>(Rune separator, params IEnumerable<T> values);
    public static Utf8String Join<T>(Utf8String? separator, params IEnumerable<T> values);

    public Utf8String Normalize(NormalizationForm normalizationForm = NormalizationForm.FormC);

    // Do we also need Insert, Remove, etc.?

    public Utf8String Replace(char oldChar, char newChar); // Ordinal
    public Utf8String Replace(char oldChar, char newChar, StringComparison comparison);
    public Utf8String Replace(char oldChar, char newChar, bool ignoreCase, CultureInfo culture);
    public Utf8String Replace(Rune oldRune, Rune newRune); // Ordinal
    public Utf8String Replace(Rune oldRune, Rune newRune, StringComparison comparison);
    public Utf8String Replace(Rune oldRune, Rune newRune, bool ignoreCase, CultureInfo culture);
    public Utf8String Replace(Utf8String oldText, Utf8String newText); // Ordinal
    public Utf8String Replace(Utf8String oldText, Utf8String newText, StringComparison comparison);
    public Utf8String Replace(Utf8String oldText, Utf8String newText, bool ignoreCase, CultureInfo culture);

    public Utf8String ToLower(CultureInfo culture);
    public Utf8String ToLowerInvariant();

    public Utf8String ToUpper(CultureInfo culture);
    public Utf8String ToUpperInvariant();

    // The Trim* APIs only trim whitespace for now. When we figure out how to trim
    // additional data we can add the appropriate overloads.

    public Utf8String Trim();
    public Utf8String TrimStart();
    public Utf8String TrimEnd();

    /*
     * PROJECTING
     */

    public ReadOnlySpan<byte> AsBytes(); // perhaps an extension method instead?
    public static explicit operator ReadOnlySpan<byte>(Utf8String? value);
    public static implicit operator Utf8Span(Utf8String? value);

    /*
     * MISCELLANEOUS
     */

    public override int GetHashCode(); // Ordinal
    public int GetHashCode(StringComparison comparisonType);

    // Used for pinning and passing to p/invoke. If the input Utf8String
    // instance is empty, returns a reference to the null terminator.

    [EditorBrowsable(EditorBrowsableState.Never)]
    public ref readonly byte GetPinnableReference();

    public static bool IsNullOrEmpty(Utf8String? value);
    public static bool IsNullOrWhiteSpace(Utf8String? value);

    public override string ToString(); // transcode to UTF-16

    /*
     * SERIALIZATION
     * (Throws an exception on deserialization if data is invalid.)
     */

    // Could also use an IObjectReference if we didn't want to implement the deserialization ctor.
    private Utf8String(SerializationInfo info, StreamingContext context);
    void ISerializable.GetObjectData(SerializationInfo info, StreamingContext context);

    /*
     * HELPER NESTED STRUCTS
     */

    public readonly struct ByteEnumerable : IEnumerable<byte> { /* ... */ }
    public readonly struct CharEnumerable : IEnumerable<char> { /* ... */ }
    public readonly struct RuneEnumerable : IEnumerable<Rune> { /* ... */ }
}

public static class MemoryExtensions
{
    public static ReadOnlyMemory<byte> AsMemory(Utf8String value);
    public static ReadOnlyMemory<byte> AsMemory(Utf8String value, int offset);
    public static ReadOnlyMemory<byte> AsMemory(Utf8String value, int offset, int count);
}

Non-allocating types

While Utf8String is an allocating, heap-based, null-terminated type; there are scenarios where a developer may want to represent a segment (or "slice") of UTF-8 data from an existing buffer without incurring an allocation.

The Utf8Segment (alternative name: Utf8Memory) and Utf8Span types can be used for this purpose. They represent a view into UTF-8 data, with the following guarantees:

• They are _immutable_ views into _immutable_ data.
• They are _guaranteed_ well-formed UTF-8 data. (Tearing will be covered shortly.)

These types have Utf8String-like methods hanging off of them as instance methods where appropriate. Additionally, they can be projected as ROM<byte> and ROS<byte> for developers who want to deal with the data at the raw binary level or who want to call existing extension methods on the ROM and ROS types.

Since Utf8Segment and Utf8Span are standalone types distinct from ROM and ROS, they can have behaviors that developers have come to expect from string-like types. For example, Utf8Segment (unlike ROM<char> or ROM<byte>) can be used as a key in a dictionary without jumping through hoops:

Dictionary<Utf8Segment, int> dict = ...;

Utf8String theString = u"hello world";
Utf8Segment segment = theString.AsMemory(0, 5); // u"hello"

if (dict.TryGetValue(segment, out int value))
{
    Console.WriteLine(value);
}

Utf8Span instances can be compared against each other:

Utf8Span data1 = ...;
Utf8Span data2 = ...;

int hashCode = data1.GetHashCode(); // Marvin32 hash

if (data1 == data2) { /* ordinal comparison of contents */ }

An alternative design that was considered was to introduce a type Char8 that would represent an 8-bit code unit - it would serve as the elemental type of Utf8String and its slices. However, ReadOnlyMemory<Char8> and ReadOnlySpan<Char8> were a bit unweildy for a few reasons.

First, there was confusion as to what ROS<Char8> actually meant when the developer could use ROS<byte> for everything. Was ROS<Char8> actually providing guarantees that ROS<byte> couldn't? (No.) When would I ever want to use a lone Char8 by itself rather than as part of a larger sequence? (You probably wouldn't.)

Second, it introduced a complication that if you had a ROM<Char8>, it couldn't be converted to a ROM<byte>. This impacted the ability to perform text manipulation and then act on the data in a binary fashion, such as sending it across the network.

Creating segment types

Segment types can be created safely from Utf8String backing objects. As mentioned earlier, we enforce that data in the UTF-8 segment types is well-formed. This implies that an instance of a segment type cannot represent data that has been sliced in the middle of a multibyte boundary. Calls to slicing APIs will throw an exception if the caller tries to slice the data in such a manner.

The Utf8Segment type introduces additional complexity in that it could be torn in a multi-threaded application, and that tearing may invalidate the well-formedness assumption by causing the torn segment to begin or end in the middle of a multi-byte UTF-8 subsequence. To resolve this issue, any instance method on Utf8Segment (including its projection to ROM<byte>) must first validate that the instance has not been torn. If the instance has been torn, an exception is thrown. This check is _O(1)_ algorithmic complexity.

It is possible that the developer will want to create a Utf8Segment or Utf8Span instance from an existing buffer (such as a pooled buffer). There are zero-cost APIs to allow this to be done; however, they are unsafe because they easily allow the developer to violate invariants held by these types.

If the developer wishes to call the unsafe factories, they must maintain the following three invariants hold.

1. The provided buffer (ROM<byte> or ROS<byte>) remains "alive" and immutable for the duration of the Utf8Segment or Utf8Span's existence. Whichever component receives a Utf8Segment or Utf8Span - however the instance has been created - must never observe that the underlying contents change or that dereferencing the contents might result in an AV or other undefined behavior.

2. The provided buffer contains only well-formed UTF-8 data, and the boundaries of the buffer do not split a multibyte UTF-8 sequence.

3. For Utf8Segment in particular, the caller __must not__ create a Utf8Segment instance wrapped around a ROM<byte> in circumstances where the component which receives the newly created Utf8Segment might tear it. The reason for this is that the "check that the Utf8Segment instance was not torn across a multi-byte subsequence" protection is only reliable when the Utf8Segment instance is backed by a Utf8String. The Utf8Segment type makes a best effort to offer protection for other backing buffers, but this protection is not ironclad in those scenarios. This could lead to a violation of invariant (2) immediately above.

The type design here - including the constraints placed on segment types and the elimination of the Char8 type - also draws inspiration from the Go, Swift, and Rust communities.

public readonly ref struct Utf8Span
{
    public Utf8Span(Utf8String? value);

    // This "Unsafe" ctor wraps a Utf8Span around an arbitrary span. It is non-copying.
    // The caller must uphold Utf8Span's invariants: that it's immutable and well-formed
    // for the lifetime that any component might be consuming the Utf8Span instance.
    // Consumers (and Utf8Span's own internal APIs) rely on this invariant, and
    // violating it could lead to undefined behavior at runtime.

    [RequiresUnsafe]
    public static Utf8Span UnsafeCreateWithoutValidation(ReadOnlySpan<byte> buffer);

    // The equality operators and GetHashCode() operate on the underlying buffers.
    // Two Utf8Span instances containing the same data will return equal and have
    // the same hash code, even if they're referencing different memory addresses.

    [EditorBrowsable(EditorBrowsableState.Never)]
    [Obsolete("Equals(object) on Utf8Span will always throw an exception. Use Equals(Utf8Span) or == instead.")]
    public override bool Equals(object? obj);
    public bool Equals(Utf8Span other);
    public bool Equals(Utf8Span other, StringComparison comparison);
    public static bool Equals(Utf8Span left, Utf8Span right);
    public static bool Equals(Utf8Span left, Utf8Span right, StringComparison comparison);
    public override int GetHashCode();
    public int GetHashCode(StringComparison comparison);
    public static bool operator !=(Utf8Span left, Utf8Span right);
    public static bool operator ==(Utf8Span left, Utf8Span right);

    // Unlike Utf8String.GetPinnableReference, Utf8Span.GetPinnableReference returns
    // null if the span is zero-length. This is because we're not guaranteed that the
    // backing data has a null terminator at the end, so we don't know whether it's
    // safe to dereference the element just past the end of the span.

    public ReadOnlySpan<byte> Bytes { get; }
    public bool IsEmpty { get; }
    [EditorBrowsable(EditorBrowsableState.Never)]
    public ref readonly byte GetPinnableReference();

    // For the most part, Utf8Span's remaining APIs mirror APIs already on Utf8String.
    // There are some exceptions: methods like ToUpperInvariant have a non-allocating
    // equivalent that allows the caller to specify the buffer which should
    // contain the result of the operation. Like Utf8String, all APIs are assumed
    // Ordinal unless the API takes a parameter which provides otherwise.

    public static Utf8Span Empty { get; }

    public ReadOnlySpan<byte> Bytes { get; } // returns ROS<byte>, not custom enumerable
    public CharEnumerable Chars { get; }
    public RuneEnumerable Runes { get; }

    // Also allow iterating over extended grapheme clusters (not yet ready).
    // public GraphemeClusterEnumerable GraphemeClusters { get; }

    public int CompareTo(Utf8Span other);
    public int CompareTo(Utf8Span other, StringComparison comparison);

    public bool Contains(char value);
    public bool Contains(char value, StringComparison comparison);
    public bool Contains(Rune value);
    public bool Contains(Rune value, StringComparison comparison);
    public bool Contains(Utf8Span value);
    public bool Contains(Utf8Span value, StringComparison comparison);

    public bool EndsWith(char value);
    public bool EndsWith(char value, StringComparison comparison);
    public bool EndsWith(Rune value);
    public bool EndsWith(Rune value, StringComparison comparison);
    public bool EndsWith(Utf8Span value);
    public bool EndsWith(Utf8Span value, StringComparison comparison);

    public bool IsAscii();

    public bool IsEmptyOrWhiteSpace();

    public bool IsNormalized(NormalizationForm normalizationForm = NormalizationForm.FormC);

    public Utf8String Normalize(NormalizationForm normalizationForm = NormalizationForm.FormC);
    public int Normalize(Span<byte> destination, NormalizationForm normalizationForm = NormalizationForm.FormC);

    public Utf8Span this[Range range] { get; }

    public SplitResult SplitOn(char separator);
    public SplitResult SplitOn(char separator, StringComparison comparisonType);
    public SplitResult SplitOn(Rune separator);
    public SplitResult SplitOn(Rune separator, StringComparison comparisonType);
    public SplitResult SplitOn(Utf8String separator);
    public SplitResult SplitOn(Utf8String separator, StringComparison comparisonType);

    public SplitResult SplitOnLast(char separator);
    public SplitResult SplitOnLast(char separator, StringComparison comparisonType);
    public SplitResult SplitOnLast(Rune separator);
    public SplitResult SplitOnLast(Rune separator, StringComparison comparisonType);
    public SplitResult SplitOnLast(Utf8String separator);
    public SplitResult SplitOnLast(Utf8String separator, StringComparison comparisonType);

    public bool StartsWith(char value);
    public bool StartsWith(char value, System.StringComparison comparison);
    public bool StartsWith(Rune value);
    public bool StartsWith(Rune value, StringComparison comparison);
    public bool StartsWith(Utf8Span value);
    public bool StartsWith(Utf8Span value, StringComparison comparison);

    public int ToChars(Span<char> destination);

    public Utf8String ToLower(CultureInfo culture);
    public int ToLower(Span<byte> destination, CultureInfo culture);

    public Utf8String ToLowerInvariant();
    public int ToLowerInvariant(Span<byte> destination);

    public override string ToString();

    public Utf8String ToUpper(CultureInfo culture);
    public int ToUpper(Span<byte> destination, CultureInfo culture);

    public Utf8String ToUpperInvariant();
    public int ToUpperInvariant(Span<byte> destination);

    public Utf8String ToUtf8String();

    // Should we also have Trim* overloads that return a range instead
    // of the span directly? Does this actually enable any new scenarios?

    public Utf8Span Trim();
    public Utf8Span TrimStart();
    public Utf8Span TrimEnd();

    public bool TryFind(char value, out Range range);
    public bool TryFind(char value, StringComparison comparisonType, out Range range);
    public bool TryFind(Rune value, out Range range);
    public bool TryFind(Rune value, StringComparison comparisonType, out Range range);
    public bool TryFind(Utf8Span value, out Range range);
    public bool TryFind(Utf8Span value, StringComparison comparisonType, out Range range);

    public bool TryFindLast(char value, out Range range);
    public bool TryFindLast(char value, StringComparison comparisonType, out Range range);
    public bool TryFindLast(Rune value, out Range range);
    public bool TryFindLast(Rune value, StringComparison comparisonType, out Range range);
    public bool TryFindLast(Utf8Span value, out Range range);
    public bool TryFindLast(Utf8Span value, StringComparison comparisonType, out Range range);

    /*
     * HELPER NESTED STRUCTS
     */

    public readonly ref struct CharEnumerable { /* pattern match for 'foreach' */ }
    public readonly ref struct RuneEnumerable { /* pattern match for 'foreach' */ }

    public readonly ref struct SplitResult
    {
        private SplitResult();

        [EditorBrowsable(EditorBrowsable.Never)]
        public void Deconstruct(out Utf8Span before, out Utf8Span after);
    }
}

public readonly struct Utf8Segment : IComparable<Utf8Segment>, IEquatable<Utf8Segment>
{
    private readonly ReadOnlyMemory<byte> _data;

    public Utf8Span Span { get; }

    // Not all span-based APIs are present. APIs on Utf8Span that would
    // return a new Utf8Span (such as Trim) should be present here, but
    // other APIs that return bool / int (like Contains, StartsWith)
    // should only be present on the Span type to discourage heavy use
    // of APIs hanging directly off of this type.

    public override bool Equals(object? other); // ok to call
    public bool Equals(Utf8Segment other); // defaults to Ordinal
    public bool Equals(Utf8Segment other, StringComparison comparison);

    public override int GetHashCode(); // Ordinal
    public int GetHashCode(StringComparison comparison);

    // Caller is responsible for ensuring:
    // - Input buffer contains well-formed UTF-8 data.
    // - Input buffer is immutable and accessible for the lifetime of this Utf8Segment instance.
    public static Utf8Segment UnsafeCreateWithoutValidation(ReadOnlyMemory<byte> data);
}

Supporting types

Like StringComparer, there's also a Utf8StringComparer which can be passed into the Dictionary<,> and HashSet<> constructors. This Utf8StringComparer also implements IEqualityComparer<Utf8Segment>, which allows using Utf8Segment instances directly as the keys inside dictionaries and other collection types.

The Dictionary<,> class is also being enlightened to understand that these types have both non-randomized and randomized hash code calculation routines. This allows dictionaries instantiated with _TKey = Utf8String_ or _TKey = Utf8Segment_ to enjoy the same performance optimizations as dictionaries instantiated with _TKey = string_.

Finally, the Utf8StringComparer type has convenience methods to compare Utf8Span instances against one another. This will make it easier to compare texts using specific cultures, even if that specific culture is not the current thread's active culture.

public abstract class Utf8StringComparer : IComparer<Utf8Segment>, IComparer<Utf8String?>, IEqualityComparer<Utf8Segment>, IEqualityComparer<Utf8String?>
{
    private Utf8StringComparer(); // all implementations are internal

    public static Utf8StringComparer CurrentCulture { get; }
    public static Utf8StringComparer CurrentCultureIgnoreCase { get; }
    public static Utf8StringComparer InvariantCulture { get; }
    public static Utf8StringComparer InvariantCultureIgnoreCase { get; }
    public static Utf8StringComparer Ordinal { get; }
    public static Utf8StringComparer OrdinalIgnoreCase { get; }

    public static Utf8StringComparer Create(CultureInfo culture, bool ignoreCase);
    public static Utf8StringComparer Create(CultureInfo culture, CompareOptions options);
    public static Utf8StringComparer FromComparison(StringComparison comparisonType);

    public abstract int Compare(Utf8Segment x, Utf8Segment y);
    public abstract int Compare(Utf8String? x, Utf8String? y);
    public abstract int Compare(Utf8Span x, Utf8Span y);
    public abstract bool Equals(Utf8Segment x, Utf8Segment y);
    public abstract bool Equals(Utf8String? x, Utf8String? y);
    public abstract bool Equals(Utf8Span x, Utf8Span y);
    public abstract int GetHashCode(Utf8Segment obj);
    public abstract int GetHashCode(Utf8String obj);
    public abstract int GetHashCode(Utf8Span obj);
}

Manipulating UTF-8 data

CoreFX and Azure scenarios

• What exchange types do we use when passing around UTF-8 data into and out of Framework APIs?

• How do we generate UTF-8 data in a low-allocation manner?

• How do we apply a series of transformations to UTF-8 data in a low-allocation manner?

  • Leave everything as Span<byte>, use a special Utf8StringBuilder type, or something else?

  • Do we need to support UTF-8 string interpolation?

  • If we have builders, who is ultimately responsible for lifetime management?

  • Perhaps should look at ValueStringBuilder for inspiration.

  • A MutableUtf8Buffer type would be promising, but we'd need to be able to generate Utf8Span slices from it, and if the buffer is being modified continually the spans could end up holding invalid data. Example below:

  MutableUtf8Buffer buffer = GetBuffer();
  Utf8Span theSpan = buffer[0..1];
  
  buffer.InsertAt(0, utf8("💣")); // U+1F483 ([ F0 9F 92 A3 ])
  
  // 'theSpan' now contains only the first byte ([ F0 ]).
  // Trying to use it could corrupt the application.
  //
  // Any such mutable UTF-8 type would necessarily be unsafe. This
  // also matches Rust's semantics: direct byte manipulation can only
  // take place within an unsafe context.
  // See:
  // * https://doc.rust-lang.org/std/string/struct.String.html#method.as_mut_vec
  // * https://doc.rust-lang.org/std/primitive.str.html#method.as_bytes_mut
  
  




• Some folks will want to perform operations in-place.

Sample operations on arbitrary buffers

(Devs may want to perform these operations on arbitrary byte buffers, even if those buffers aren't guaranteed to contain valid UTF-8 data.)

• Validate that buffer contains well-formed UTF-8 data.

• Convert ASCII data to upper / lower in-place, leaving all non-ASCII data untouched.

• Split on byte patterns. (Probably shouldn't split on runes or UTF-8 string data, since we can't guarantee data is well-formed UTF-8.)

These operations could be on the newly-introduced System.Text.Unicode.Utf8 static class. They would take ROS<byte> and Span<byte> as input parameters because they can operate on arbitrary byte buffers. Their runtime performance would be subpar compared to similar methods on Utf8String, Utf8Span, or other types where we can guarantee that no invalid data will be seen, as the APIs which operate on raw byte buffers would need to be defensive and would probably operate over the input in an iterative fashion rather than in bulk. One potential behavior could be skipping over invalid data and leaving it unchanged as part of the operation.

Sample Utf8StringBuilder implementation for private use

internal ref struct Utf8StringBuilder
{
    public void Append<T>(T value) where T : IUtf8Formattable;
    public void Append<T>(T value, string format, CultureInfo culture) where T : IUtf8Formattable;

    public void Append(Utf8String value);
    public void Append(Utf8Segment value);
    public void Append(Utf8Span value);

    // Some other Append methods, resize methods, etc.
    // Methods to query the length.

    public Utf8String ToUtf8String();

    public void Dispose(); // when done with the instance
}

// Would be implemented by numeric types (int, etc.),
// DateTime, String, Utf8String, Guid, other primitives,
// Uri, and anything else we might want to throw into
// interpolated data.
internal interface IUtf8Formattable
{
    void Append(ref Utf8StringBuilder builder);
    void Append(ref Utf8StringBuilder builder, string format, CultureInfo culture);
}

Code samples and metadata representation

The C# compiler could detect support for UTF-8 strings by looking for the existence of the System.Utf8String type and the appropriate helper APIs on RuntimeHelpers as called out in the samples below. If these APIs don't exist, then the target framework does not support the concept of UTF-8 strings.

Literals

Literal UTF-8 strings would appear as regular strings in source code, but would be prefixed by a _u_ as demonstrated below. The _u_ prefix would denote that the return type of this literal string expression should be Utf8String instead of string.

Utf8String myUtf8String = u"A literal string!";

// Normal ldstr to literal UTF-16 string in PE string table, followed by
// call to helper method which translates this to a UTF-8 string literal.
// The end result of these calls is that a Utf8String instance sits atop
// the stack.

ldstr "A literal string!"
call class System.Utf8String System.Runtime.CompilerServices.RuntimeHelpers.InitializeUtf8StringLiteral(string)

The _u_ prefix would also be combinable with the _@_ prefix and the _$_ prefix (more on this below).

Additionally, literal UTF-8 strings __must__ be well-formed Unicode strings.

// Below line would be a compile-time error since it contains ill-formed Unicode data.
Utf8String myUtf8String = u"A malformed \ud800 literal string!";

Three alternative designs were considered. One was to use RVA statics (through ldsflda) instead of literal UTF-16 strings (through ldstr) before calling a "load from RVA" method on RuntimeHelpers. The overhead of using RVA statics is somewhat greater than the overhead of using the normal UTF-16 string table, so the normal UTF-16 string literal table should still be the more optimized case for small-ish strings, which we believe to be the common case.

Another alternative considered was to introduce a new opcode ldstr.utf8, which would act as a UTF-8 equivalent to the normal ldstr opcode. This would be a breaking change to the .NET tooling ecosystem, and the ultimate decision was that there would be too much pain to the ecosystem to justify the benefit.

The third alternative considered was to smuggle UTF-8 data in through a normal UTF-16 string in the string table, then call a RuntimeHelpers method to reinterpret the contents. This would result in a "garbled" string for anybody looking at the raw IL. While that in itself isn't terrible, there is the possibility that smuggling UTF-8 data in this manner could result in a literal string which has ill-formed UTF-16 data. Not all .NET tooling is resilient to this. For example, xunit's test runner produces failures if it sees attributes initialized from literal strings containing ill-formed UTF-16 data. There is a risk that other tooling would behave similarly, potentially modifying the DLL in such a manner that errors only manifest themselves at runtime. This could result in difficult-to-diagnose bugs.

We may wish to reconsider this decision in the future. For example, if we see that it is common for developers to use large UTF-8 literal strings, maybe we'd want to dynamically switch to using RVA statics for such strings. This would lower the resulting DLL size. However, this would add extra complexity to the compilation process, so we'd want to tread lightly here.

Constant handling

class MyClass
{
    public const Utf8String MyConst = u"A const string!";
}

// Literal field initialized to literal UTF-16 value. The runtime doesn't care about
// this (modulo FieldInfo.GetRawConstantValue, which perhaps we could fix up), so
// only the C# compiler would need to know that this is a UTF-8 constant and that
// references to it should get the same (ldstr, call) treatment as stated above.

.field public static literal class System.Utf8String MyConst = "A const string!";

String concatenation

There would be APIs on Utf8String which mirror the string.Concat APIs. The compiler should special-case the + operator to call the appropriate overload n-ary overload of Concat.

Utf8String a = ...;
Utf8String b = ...;

Utf8String c = a + u", " + b; // calls Utf8String.Concat(...)

Since we expect use of Utf8String to be "deliberate" when compared to string (see the beginning of this document), we should consider that a developer who is using UTF-8 wants to stay in UTF-8 during concatenation operations. This means that if there's a line which involves the concatenation of both a Utf8String and a string, the final type post-concatenation should be Utf8String.

Utf8String a = ...;
string b = ...;

Utf8String concatFoo = a + b;
string concatBar = (object)a + b; // compiler can't statically determine that any argument is Utf8String

This is still open for discussion, as the behavior may be surprising to people. Another alternative is to produce a build warning if somebody tries to mix-and-match UTF-8 strings and UTF-16 strings in a single concatenation expression.

If string interpolation is added in the future, this shouldn't result in ambiguity. The $ interpolation operator will be applied to a literal Utf8String or a literal string, and that would dictate the overall return type of the operation.

Equality comparisons

There are standard == and != operators defined on the Utf8String class.

public static bool operator ==(Utf8String a, Utf8String b);
public static bool operator !=(Utf8String a, Utf8String b);

The C# compiler should special-case when either side of an equality expression is known to be a literal null object, and if so the compiler should emit a referential check against the null object instead of calling the operator method. This matches the if (myString == null) behavior that the string type enjoys today.

Additionally, equality / inequality comparisons between Utf8String and string should produce compiler warnings, as they will never succeed.

Utf8String a = ...;
string b = ...;

// Below line should produce a warning since it will end up being the equivalent
// of Object.ReferenceEquals, which will only succeed if both arguments are null.
// This probably wasn't what the developer intended to check.

if (a == b) { /* ... */ }

I attempted to define operator ==(Utf8String a, string b) so that I could slap [Obsolete] on it and generate the appropriate warning, but this had the side effect of disallowing the user to write the code if (myUtf8String == null) since the compiler couldn't figure out which overload of operator == to call. This was also one of the reasons I had opened https://github.com/dotnet/csharplang/issues/2340.

Marshaling behaviors

Like the string type, the Utf8String type shall be marshalable across p/invoke boundaries. The corresponding unmanaged type shall be LPCUTF8 (equivalent to a BYTE* pointing to null-terminated UTF-8 data) unless a different unmanaged type is specified in the p/invoke signature.

If a different [MarshalAs] representation is specified, the stub routine creates a temporary copy in the desired representation, performs the p/invoke, then destroys the temporary copy or allows the GC to reclaim the temporary copy.

class NativeMethods
{
    [DllImport]
    public static extern int MyPInvokeMethod(
        [In] Utf8String marshaledAsLPCUTF8,
        [In, MarshalAs(UnmanagedType.LPUTF8Str)] Utf8String alsoMarshaledAsLPCUTF8,
        [In, MarshalAs(UnmanagedType.LPWStr)] Utf8String marshaledAsLPCWSTR,
        [In, MarshalAs(UnmanagedType.BStr)] Utf8String marshaledAsBSTR);
}

If a Utf8String must be marshaled from native-to-managed (e.g., a reverse p/invoke takes place on a delegate which has a Utf8String parameter), the stub routine is responsible for fixing up invalid UTF-8 data before creating the Utf8String instance (or it may let the Utf8String constructor perform the fixup automatically).

Unmanaged routines must not modify the contents of any Utf8String instance marshaled across the p/invoke boundary. Utf8String instances are assumed to be immutable once created, and violating this assumption could cause undefined behaviors within the runtime.

There is no default marshaling behavior for Utf8Segment or Utf8Span since they are not guaranteed to be null-terminated. If in the future the runtime allows marshaling {ReadOnly}Span<T> across a p/invoke boundary (presumably as a non-null-terminated array equivalent), library authors may fetch the underlying ReadOnlySpan<byte> from the Utf8Segment or Utf8Span instance and directly marshal that span across the p/invoke boundary.

Automatic coercion of UTF-16 literals to UTF-8 literals

If possible, it would be nice if UTF-16 literals (not arbitrary string instances) could be automatically coerced to UTF-8 literals (via the ldstr / call routines mentioned earlier). This coercion would only be considered if attempting to leave the data as a string would have caused a compilation error. This could help eliminate some errors resulting from developers forgetting to put the _u_ prefix in front of the string literal, and it could make the code cleaner. Some examples follow.

// String literal being assigned to a member / local of type Utf8String.
public const Utf8String MyConst = "A literal!";

public void Foo(string s);
public void Foo(Utf8String s);

public void FooCaller()
{
    // Calls Foo(string) since it's an exact match.
    Foo("A literal!");
}

public void Bar(object o);
public void Bar(Utf8String s);

public void BarCaller()
{
    // Calls Bar(object), passing in the string literal,
    // since it's the closest match.
    Bar("A literal!");
}

public void Baz(int i);
public void Baz(Utf8String s);

public void BazCaller1()
{
    // Calls Baz(Utf8String), passing in the UTF-8 literal,
    // since there's no closer match.
    Baz("A literal!");
}

public void BazCaller2(string someInput)
{
    // Compiler error. The input isn't a literal, so no auto-coercion
    // takes place. Dev should call Baz(new Utf8String(someInput)).
    Baz(someInput);
}

public void Quux<T>(ReadOnlySpan<T> value);
public void Quux(Utf8String s);

public void QuuxCaller()
{
    // Calls Quux<char>(ReadOnlySpan<char>), passing in the string literal,
    // since string satisfies the constraints.
    Quux("A literal!");
}

public void Glomp(Utf8Span value);

public void GlompCaller()
{
    // Calls Glomp(Utf8Span), passing in the UTF-8 literal, since there's
    // no closer match and Utf8String can be implicitly cast to Utf8Span.
    Glomp("A literal!");
}

UTF-8 String interpolation

The string interpolation feature is undergoing significant churn (see https://github.com/dotnet/csharplang/issues/2302). I envision that when a final design is chosen, there would be a UTF-8 counterpart for symmetry. The internal IUtf8Formattable interface as proposed above is being designed partly with this feature in mind in order to allow single-allocation Utf8String interpolation.

ustring contextual language keyword

For simplicity, we may want to consider a contextual language keyword which corresponds to the System.Utf8String type. The exact name is still up for debate, as is whether we'd want it at all, but we could consider something like the below.

Utf8String a = u"Some UTF-8 string.";

// 'ustring' and 'System.Utf8String' are aliases, as shown below.

ustring b = a;
Utf8String c = b;

The name ustring is intended to invoke "Unicode string". Another leading candidate was utf8. We may wish not to ship with this keyword support in v1 of the Utf8String feature. If we opt not to do so we should be mindful of how we might be able to add it in the future without introducing breaking changes.

An alternative design to use a u suffix instead of a u prefix. I'm mostly impartial to this, but there is a nice symmetry to having the characters u, $, and @ all available as prefixes on literal strings.

We could also drop the u prefix entirely and rely solely on type targeting:

ustring a = "Literal string type-targeted to UTF-8.";
object b = (ustring)"Another literal string type-targeted to UTF-8.";

This has implications for string interpolation, as it wouldn't be possible to prepend both the (ustring) coercion hint and the $ interpolation operator simultaneously.

Switching and pattern matching

If a value whose type is statically known to be Utf8String is passed to a switch statement, the corresponding case statements should allow the use of literal Utf8String values.

Utf8String value = ...;

switch (value)
{
    case u"Some literal": /* ... */
    case u"Some other literal": /* ... */
    case "Yet another literal": /* target typing also works */
}

Since pattern matching operates on input values of arbitrary types, I'm pessimistic that pattern matching will be able to take advantage of target typing. This may instead require that developers specify the u prefix on Utf8String literals if they wish such values to participate in pattern matching.

A brief interlude on indexers and IndexOf

Utf8String and related types do not expose an elemental indexer (this[int]) or a typical IndexOf method because they're trying to rid the developer of the notion that bytewise indices into UTF-8 buffers can be treated equivalently as charwise indices into UTF-16 buffers. Consider the naïve implementation of a typical "string split" routine as presented below.

void SplitString(string source, string target, StringComparison comparisonType, out string beforeTarget, out string afterTarget)
{
    // Locates 'target' within 'source', splits on it, then populates the two out parameters.
    // ** NOTE ** This code has a bug, as will be explained in detail below.

    int index = source.IndexOf(target, comparisonType);
    if (index < 0) { throw new Exception("Target string not found!"); }

    beforeTarget = source.Substring(0, index);
    afterTarget = source.Substring(index + target.Length, source.Length - index - target.Length);
}

One subtlety of the above code is that when culture-sensitive or case-insensitive comparers are used (such as _OrdinalIgnoreCase_ in the above example), the target string doesn't have to be an exact char-for-char match of a sequence present in the source string. For example, consider the UTF-16 string "GREEN" ([ 0047 0052 0045 0045 004E ]). Performing an _OrdinalIgnoreCase_ search for the substring "e" ([ 0065 ]) will result in a match, as 'e' (U+0065) and 'E' (U+0045) compare as equal under an _OrdinalIgnoreCase_ comparer.

As another example, consider the UTF-16 string "preſs" ([ 0070 0072 0065 017F 0073 ]), whose fourth character is the _Latin long s_ 'ſ' (U+017F). Performing an _OrdinalIgnoreCase_ search for the substring "S" ([ 0053 ]) will result in a match, as 'ſ' (U+017F) and 'S' (U+0053) compare as equal under an _OrdinalIgnoreCase_ comparer.

There are also scenarios where the length of the match within the search string might not be equal to the length of the target string. Consider the UTF-16 string "encyclopædia" ([ 0065 006E 0063 0079 0063 006C 006F 0070 00E6 0064 0069 0061 ]), whose ninth character is the ligature 'æ' (U+00E6). Performing an _InvariantCultureIgnoreCase_ search for the substring "ae" ([ 0061 0065 ]) will result in a match at index 8, as "æ" ([ 00E6 ]) and "ae" ([ 0061 0065 ]) compare as equal under an _InvariantCultureIgnoreCase_ comparer.

This result is interesting and should give us pause. Since "æ".Length == 1 and "ae".Length == 2, the arithmetic at the end of the method will actually result in the wrong substrings being returned to the caller.

beforeTarget = source.Substring(0, 8 /* index */); // = "encyclop"
afterTarget = source.Substring(
    10 /* index + target.Length */,
    2 /* source.Length - index - target.Length */); // = "ia" (expected "dia"!)

Due to the nature of UTF-16 (used by string), when performing an _Ordinal_ or an _OrdinalIgnoreCase_ comparison, the length of the matched substring within the source will always have a char count equal to target.Length. The length mismatch as demonstrated by "encyclopædia" above can only happen with a culture-sensitive comparer or any of the _InvariantCulture_ comparers.

However, in UTF-8, these same guarantees do not hold. Under UTF-8, only when performing an _Ordinal_ comparison is there a guarantee that the length of the matched substring within the source will have a byte count equal to the target. All other comparers - including _OrdinalIgnoreCase_ - have the behavior that the byte length of the matched substring can change (either shrink or grow) when compared to the byte length of the target string.

As an example of this, consider the string "preſs" from earlier, but this time in its UTF-8 representation ([ 70 72 65 C5 BF 73 ]). Performing an _OrdinalIgnoreCase_ for the target UTF-8 string "S" ([ 53 ]) will match on the ([ C5 BF ]) portion of the source string. (This is the UTF-8 representation of the letter 'ſ'.) To properly split the source string along this search target, the caller need to know not only where the match was, _but also how long the match was within the original source string_.

This fundamental problem is why Utf8String and related types don't expose a standard IndexOf function or a standard this[int] indexer. It's still possible to index directly into the underlying byte buffer by using an API which projects the data as a ROS<byte>. But for splitting operations, these types instead offer a simpler API that performs the split on the caller's behalf, handling the length adjustments appropriately. For callers who want the equivalent of IndexOf, the types instead provide TryFind APIs that return a Range instead of a typical integral index value. This Range represents the matching substring within the original source string, and new C# language features make it easy to take this result and use it to create slices of the original source input string.

This also addresses feedback that was given in a previous prototype: users weren't sure how to interpret the result of the IndexOf method. (Is it a byte count? Is it a char count? Is it something else?) Similarly, there was confusion as to what parameters should be passed to a this[int] indexer or a Substring(int, int) method. By having the APIs promote use of Range and related C# language features, this confusion should subside. Power developers can inspect the Range instance directly to extract raw byte offsets if needed, but most devs shouldn't need to query such information.

API usage samples

__Scenario:__ Split an incoming string of the form "LastName, FirstName" into individual _FirstName_ and _LastName_ components.

// Using Utf8String input and producing Utf8String instances
void SplitSample(ustring input)
{
    // Method 1: Use the SplitOn API to find the ',' char, then trim manually.

    (ustring lastName, ustring firstName) = input.Split(',');
    if (firstName is null) { /* ERROR: no ',' detected in input */ }

    lastName = lastName.Trim();
    firstName = firstName.Trim();

    // Method 2: Use the SplitOn API to find the ", " target string, assuming no trim needed.

    (ustring lastName, ustring firstName) = input.Split(u", ");
    if (firstName is null) { /* ERROR: no ", " detected in input */ }
}

// Using Utf8Span input and producing Utf8Span instances
void SplitSample(Utf8Span input)
{
    // Method 1: Use the SplitOn API to find the ',' char, then trim manually.

    (Utf8Span lastName, Utf8Span firstName) = input.Split(',');
    lastName = lastName.Trim();
    firstName = firstName.Trim();
    if (firstName.IsEmpty) { /* ERROR: trailing ',', or no ',' detected in input */ }

    // Method 2: Use the SplitOn API to find the ", " target string, assuming no trim needed.

    (Utf8Span lastName, Utf8Span firstName) = input.Split(", ");
    if (firstName.IsEmpty) { /* ERROR: trailing ", ", or no ", " detected in input */ }
}

Additionally, the SplitResult struct returned by Utf8Span.Split implements both a standard IEnumerable<T> pattern and the C# _deconstruct_ pattern, which allows it to be used separately from enumeration for simple cases where only a small handful of values are returned.

Utf8Span str = ...;

// The result of Utf8Span.Split can be used in an enumerator

foreach (Utf8Span substr in str.Split(','))
{
    /* operate on substr */
}

// Or it can be used in tuple deconstruction
// (See docs for description of behavior for each arity.)

(Utf8Span before, Utf8Span after) = str.Split(',');
(Utf8Span part1, Utf8Span part2, Utf8Span part3, ...) = str.Split(',');

__Scenario:__ Split a comma-delimited input into substrings, then perform an operation with each substring.

// Using Utf8String input and producing Utf8String instances
// The Utf8Span code would look  identical (sub. 'Utf8Span' for 'ustring')

void SplitSample(ustring input)
{
    while (input.Length > 0)
    {
        // 'TryFind' is the 'IndexOf' equivalent. It returns a Range instead
        // of an integer index because there's no this[int] indexer on Utf8String.

        if (!input.TryFind(',', out Range matchedRange))
        {
            // The remainder of the input string is empty, but no comma
            // was found in the remaining portion. Process the remainder
            // of the input string, then finish.

            ProcessValue(input);
            break;
        }

        // We found a comma! Substring and process.
        // The 'matchedRange' local contains the range for the ',' that we found.

        ProcessValue(input[..matchedRange.Start]); // fetch segment to the left of the comma, then process it
        input = input[matchedRange.End..]; // set 'input' to the remainder of the input string and loop
    }

    // Could also have an IEnumerable<ustring>-returning version if we wanted, I suppose.
}

Miscellaneous topics and open questions

__What about comparing UTF-16 and UTF-8 data?__

Currently there is a set of APIs Utf8String.AreEquivalent which will decode sequences of UTF-16 and UTF-8 data and compare them for ordinal equality. The general code pattern is below.

ustring a = ...;
string b = ...;

// The below line fails to compile because there's no operator==(Utf8String, string) defined.

bool result = (a == b);

// The below line is probably what the developer intended to write.

bool result = ustring.AreEquivalent(a, b);

// The below line should compile since literal strings can be type targeted to Utf8String.

bool result = (a == "Hello!");

Do we want to add an operator==(Utf8String, string) overload which would allow easy == comparison of UTF-8 and UTF-16 data? There are three main downsides to this which caused me to vote no, but I'm open to reconsideration.

1. The compiler would need to special-case if (myUtf8String == null), which would now be ambiguous between the two overloads. (If the compiler is already special-casing null checks, this is a non-issue.)

2. The performance of UTF-16 to UTF-8 comparison is much worse than the performance of UTF-16 to UTF-16 (or UTF-8 to UTF-8) comparison. When the representation is the same on both sides, certain shortcuts can be implemented to avoid the _O(n)_ comparison, and even the _O(n)_ comparison itself can be implemented as a simple _memcmp_ operation. When the representations are heterogeneous, the opportunity for taking shortcuts is much more restricted, and the _O(n)_ comparison itself has a higher constant factor. Developers might not expect such a performance characteristic from an equality operator.

3. Comparing a Utf8String against a literal string would no longer go through the fast path, as target typing would cause the compiler to emit a call to operator==(Utf8String, string) instead of operator==(Utf8String, Utf8String). The comparison itself would then have the lower performance described by bullet (2) above.

One potential upside to having such a comparison is that it would prevent developers from using the antipattern if (myUtf8String.ToString() == someString), which would result in unnecessary allocations. If we are concerned about this antipattern one way to address it would be through a Code Analyzer.

__What if somebody passes invalid data to the "skip validation" factories?__

When calling the "unsafe" APIs, callers are fully responsible for ensuring that the invariants are maintained. Our debug builds could double-check some of these invariants (such as the initial Utf8String creation consisting only of well-formed data). We could also consider allowing applications to opt-in to these checks at runtime by enabling an MDA or other diagnostic facility. But as a guiding principle, when "unsafe" APIs are called the Framework should trust the developer and should have as little overhead as possible.

__Consider consolidating the unsafe factory methods under a single unsafe type.__

This would prevent pollution of the type's normal API surface and could help write tools which audit use of a single "unsafe" type.

Some of the methods may need to be extension methods instead of normal static factories. (Example: Unsafe slicing routines, should we choose to expose them.)

Potential APIs to enlighten

_System_ namespace

Include Utf8String / Utf8Span overloads on Console.WriteLine. Additionally, perhaps introduce an API Console.ReadLineUtf8.

_System.Data.*_ namepace

Include generalized support for serializing Utf8String properties as a primitive with appropriate mapping to nchar or nvarchar.

_System.Diagnostics.*_ namespace

Enlighten EventSource so that a caller can write Utf8String / Utf8Span instances cheaply. Additionally, some types like ActivitySpanId already have ROS<byte> ctors; overloads can be introduced here.

_System.Globalization.*_ namespace

The CompareInfo type has many members which operate on string instances. These should be spanified foremost, and Utf8String / Utf8Span overloads should be added. Good candidates are Compare, GetHashCode, IndexOf, IsPrefix, and IsSuffix.

The TextInfo type has members which should be treated similarly. ToLower and ToUpper are good candidates. Can we get away without enlightening ToTitleCase?

_System.IO.*_ namespace

BinaryReader and BinaryWriter should have overloads which operate on Utf8String and Utf8Span. These overloads could potentially be cheaper than the normal string / ROS<char> based overloads, since the reader / writer instances may in fact be backed by UTF-8 under the covers. If this is the case then writing is simple projection, and reading is validation (faster than transcoding).

File: WriteAllLines, WriteAllText, AppendAllText, etc. are good candidates for overloads to be added. On the read side, there's ReadAllTextUtf8 and ReadAllLinesUtf8.

TextReader.ReadLine and TextWriter.Write are also good candidates to overload. This follows the same general premise as BinaryReader and BinaryWriter as mentioned above.

Should we also enlighten SerialPort or GPIO APIs? I'm not sure if UTF-8 is a bottleneck here.

_System.Net.Http.*_ namespace

Introduce Utf8StringContent, which automatically sets the _charset_ header. This type already exists in the _System.Utf8String.Experimental_ package.

_System.Text.*_ namespace

UTF8Encoding: Overload candidates are GetChars, GetString, and GetCharCount (of Utf8String or Utf8Span). These would be able to skip validation after transcoding as long as the developer hasn't subclassed the type.

Rune: Add ToUtf8String API. Add IsDefined API to query the OS's NLS tables (could help with databases and other components that need to adhere to strict case / comparison processing standards).

TextEncoder: Add Encode(Utf8String): Utf8String and FindFirstIndexToEncode(Utf8Span): Index. This is useful for HTML-escaping, JSON-escaping, and related operations.

Utf8JsonReader: Add read APIs (GetUtf8String) and overloads to both the ctor and ValueTextEquals.

JsonEncodedText: Add an EncodedUtf8String property.

Regex is a bit of a special case because there has been discussion about redoing the regex stack all-up. If we did proceed with redoing the stack, then it would make sense to add first-class support for UTF-8 here.

[benaadams]

Even though byte / charu8 is the underlying elemental type of Utf8String, none of the APIs outside of the constructor actually take those types as input. The input parameter types to IndexOf and similar APIs is UnicodeScalar, which represents an arbitrary Unicode scalar value and can be 1 - 4 code units wide when transcoded to UTF-8.

Does that mean

var ss = s.Substring(s.IndexOf(','));

Would be a double traversal? i.e. any use of IndexOf would lead to a double traversal for its return value to be meaningful?

[GrabYourPitchforks]

Yes, I know this is dated _from the future_! :)
It's our agenda and review doc for the in-person meeting before it goes to wider community review. Not everything is captured here, especially things related to runtime interaction.

[GrabYourPitchforks]

@benaadams No, it's a single traversal, just like if _s_ were typed as System.String in your example. The IndexOf is O(n) up to the first found ',' character (using a vectorized search if available), and the Substring is O(n) from the indexed position to the end of the string. So the total number of bytes observed is index /* IndexOf */ + (Length - index) /* memcpy */ = Length = single traversal.

[benaadams]

But if IndexOf is returning the number of UnicodeScalars which can be 1-4 bytes; passing that int return value into Substring doesn't it then have to rescan from the start of the Utf8String to find that start position? i.e. IndexOf isn't returning (int scalarPosition, int byteOffset)

[GrabYourPitchforks]

APIs that operate on indices (like IndexOf, Substring, etc.) go by code unit count, not scalar count.

(I get that it might be confusing since _enumeration_ of Utf8String instances goes by scalar, not by code unit, so now we have a disparity on the type. That's why I'd proposed as an open question that maybe we kill the enumerator entirely and just have Bytes and Scalars properties, which removes the disparity.)

[stephentoub]

Thanks, Levi. Some questions/comments:

1. Should be straightforward and O(1) to create a Utf8String instance from an existing String / ReadOnlySpan or from a ReadOnlySpan coming in from the wire.

I don't understand how this is possible. With Utf8String as a reference type, getting the data into it will necessitate a memcpy at a minimum, which is not O(1).

Must allow querying total length (in code units) as O(1) operation.

I would expect a requirement would also be being able to query the total length in bytes in O(1) (which is also possible with string).

The five requirements below are drawn from String

This is already making some trade-offs. If I've read the data off the wire, I already have it in some memory, which I can then process as a ReadOnlySpan<byte>. To use it as a Utf8String, I then need to allocate and copy. So we're trading off usability for perf. I'm a bit surprised that's the right trade-off for the target audience, but the doc also doesn't specify who the target developers are, provide example scenarios for where/how this will be used, etc.

public ReadOnlySpan Bytes { get; }
public ReadOnlyMemory AsMemory();

Why is the to-memory conversion called AsMemory but the to-span conversion called Bytes?

public bool Contains(UnicodeScalar value);

I'm surprised not to see overloads of methods like Contains (IndexOf, EndsWith, etc.) that accept string or char. For char, even if you add an implicit cast from char to UnicodeScalar, we just had that discussion about not relying on implicit casts from a usability perspective in cases like this. And for string, with the currently defined methods someone would need to actually convert a string to a Utf8String, which is not cheap, in order to call these methods.

public int IndexOfAny(ReadOnlySpan value);
public int LastIndexOfAny(ReadOnlySpan value);

string.{Last}IndexOfAny calls this argument anyOf.

public Utf8String ToLowerInvariant();
public Utf8String ToUpperInvariant();

Presumably Utf8String will have culture support and will also have ToLower/Upper methods that are culture-sensitive?

public int IndexOf(UnicodeScalar value);
public int IndexOf(UnicodeScalar value, int startIndex);

What does the return value mean? Is that the number of the byte offset of the UnicodeScalar, or is it the number of the UnicodeScalar? Similarly, for startIndex. Assuming it's the number of UnicodeScalars, if I want to get Bytes and index into it starting at this UnicodeScalar, how do I convert that UnicodeScalar-offset to a byte offset?

Once culture support comes online, we should add CompareTo and related APIs.

From a design discussion perspective, I would think we'd want this outline to represent the ultimate shape we want, and the implementations can throw NotImplementedException until the functionality is available (before it ships).

public readonly struct UnicodeScalar

What's the plan for integration of this with the existing unicode support in .NET? For example, how do I get a System.Globalization.UnicodeCategory for one of these?

public readonly struct Utf8StringSegment

Similar questions related to the APIs on Utf8String.

And, presumably we wouldn't define any APIs (outside of Utf8String/Utf8StringSegment) that accept a Utf8String, instead accepting a Utf8StringSegment, since the former can cheaply convert to the latter but not vice versa?

For me, it also begs the question why do we need both? If we're going to have Utf8StringSegment, presumably that becomes the thing that most APIs would be written in terms of, because it can cheaply represent both the whole and slices. And once you have that, which effectively has the same surface area as Utf8String, why not just make it Utf8String, still as a struct, and get rid of the class-equivalent and duplication. It can then be constructed from a byte[] or a ReadOnlyMemory<byte> without any extra allocation or copying, can be cheaply sliced, etc. Utf8StringSegment (when named Utf8String) is then essentially as a nice wrapper / package for a lot of the functionality that exists in System.Memory as static methods.

n.b. This type is not pinnable because we cannot guarantee null termination.

I don't see why we'd place this restriction. Arrays don't guarantee null termination but are pinnable. Lots of types don't guarantee null termination but are pinnable.

// Pass a Utf8String instance across a p/invoke boundary

I would hope that before or as part of enabling this, we add support for Span<T> and ReadOnlySpan<T>. We still have debt to be paid down there and should address that before adding this as well.

Culture-aware processing code is currently implemented in terms of UTF-16 across all platforms. We don't expect this to change appreciably in the near future, which means that any operations which use culture data will almost certainly require two transcoding steps, making them expensive for UTF-8 data.

I didn't understand this part. Don't both Windows and ICU provide UTF8-based support in addition to the UTF16-based support that's currently being used?

Other stuff

Equivalents for String.Format?

[whoisj]

Don't both Windows and ICU provide UTF8-based support in addition to the UTF16-based support that's currently being used?

Not that I know of. Windows is, with very good legacy reasons, very UTF-16/UCS-2 focused.

[whoisj]

What about Equals(string other) or CompareTo(string other) ?

Seems like not implementing this would make it difficult for existing ecosystems to adopt this type.

[KrzysztofCwalina]

1. The proposal lists servers and IoT as main scenarios. I think we need to add ML.NET. They explicitly requested UTF8 string support.
2. The ML.NET team requires allocation free slicing. I am not sure if they need the slices to be heap-friendly or not. Something you should research.
3. It would be good to drill into reasons for each of the pri 0 requirements. They all start with "must" and some are very limiting.
4. I think the requirements should include slicing (even if we decide that slices are a different type and/or not heapable). Non-allocating slicing is a must have for high performance string manipulation.
5. As a validation exercise, it would be good to rewrite ASP.NET platform server using this string (the code now uses custom AsciiString) and see if we can keep the same performance.
6. EndsWith (and all similar operations) should have overloads that take ReadOnlySpan<some_type>, and C# should support conveniently creating literals of this span on the stack, e.g. (pseudocode): myString.EndsWith(stackalloc u8"World!"). Currently all the APIs that Utf8String (which allocates) and scalar (which is a single "char", i.e. not super useful).
7. In the language support section you state that a literal assignment to Utf8String will result in conversion. Why? We should do target typing in the case you outline and avoid any conversions at runtime.
8. Nit: I find the "u8" prefix super ugly.
9. We use ReadOnlySpan<Char> as a representation of a slice of UTF16 string. You are proposing we use Utf8StringSegment. Is the discrepancy ok?
10. Re Open Question #1: I don't think doing LINQ over scalars is a good practice.

[nil4]

The signature public Utf8String[] Split(Utf8String separator) implies a lot of allocations and memory copies.

First, an array must be allocated for the return value.

Then, each element in the array must be a copy of each match, into a newly-allocated buffer, as Utf8String mandates null-termination but the input will not have nulls after each separator.

If I understand this correctly, except for the trivial case when the separator is not present at all, this signature would basically require copying the whole input string.

Would it make sense to return a custom enumerator of Utf8StringSegment instead, similar to SplitByScalarEnumerator or SplitBySubstringEnumerator?

[svick]

I think the biggest issue with the proposed API is confusion between UTF8 code units and Unicode scalar values, especially when it comes to lengths and indexes. Would it make sense to alleviate that confusion by more explicit names, like ByteLength instead of Length or startByteIndex instead of startIndex?

---

c# [EditorBrowsable(EditorBrowsableState.Never)] public static Utf8String DangerousCreateWithoutValidation(ReadOnlySpan<byte> value);

Is EditorBrowsableState.Never the right way to hide dangerous methods? I don't like it, because it means such methods are hard to use, when I think the actual goal is to limit their discoverability, not their usability. Wouldn't putting them into a separate type be a better solution, similar to how dangerous Span APIs were put into the MemoryMarshal type?

---

One potential workaround is to make the JIT recognize a ldstr opcode immediately followed by a newobj Utf8String(string) opcode. This pattern can be special-cased to behave similarly to the standalone ldstr today, where the address of the literal String (or Utf8String) object is known at JIT time and a single mov reg, imm instruction is generated.

Would this mean that if I write new Utf8String("foo"), which would produce the same sequence of opcodes, it might not actually create a new instance of Utf8String? I think that would be very confusing, since it's not how any other type behaves, not even string. It would also be a violation of the C# specification, which says that for a class, new has to allocate a new instance:

The run-time processing of an object_creation_expression of the form new T(A), […] consists of the following steps:

• If T is a class_type:
  
  
  
  • A new instance of class T is allocated. […]
  
  

---

What is the relationship between UnicodeScalar and Rune (https://github.com/dotnet/corefx/issues/24093)?

---

We can also consider introducing a type StringSegment which is the String-backed analog of this type.

There was an issue about creating StringSegment in corefx, which was closed a month ago, with the justification that ReadOnlyMemory<char> and ReadOnlySpan<char> are good enough: https://github.com/dotnet/corefx/issues/20378. Does that mean it's now on the table again?

---

The code comments on the StringSegment type go into much more detail on the benefits of this type when compared to ReadOnlyMemory<T> / ReadOnlySpan<T>.

Where can I find those comments? I didn't find the StringSegment type in any dotnet repo.

---

More generally, with this proposal we will have: string, char[], Span<char>, ReadOnlySpan<char>, Memory<char>, ReadOnlyMemory<char>, Utf8String, byte[], Span<byte>, ReadOnlySpan<byte>, Memory<byte> and ReadOnlyMemory<byte>. Do we really need Utf8StringSegment as yet another string-like type?

[GrabYourPitchforks]

I don't understand how this is possible. With Utf8String as a reference type, getting the data into it will necessitate a memcpy at a minimum, which is not O(1).

Yes, this is a typo.

I would expect a requirement would also be being able to query the total length in bytes in O(1) (which is also possible with string).

This is possible via Utf8String.Length or Utf8String.Bytes.Length, both of which return the byte count.

I'm surprised not to see overloads of methods like Contains (IndexOf, EndsWith, etc.) that accept string or char.

I struggled with this, and the reason I ultimately decided not to include it is because I think the majority of calls to these methods involve searching for _literal_ substrings, and I'd rather rely on a one-time compiler conversion of the search target from UTF-16 to UTF-8 than a constantly-reoccurring runtime conversion from UTF-16 to UTF-8. I'm concerned that the presence of these overloads would encourage callers to inadvertently use a slow path that requires transcoding. We can go over this in Friday's discussion.

What's the plan for integration of [UnicodeScalar] with the existing unicode support in .NET?

I had planned APIs like UnicodeScalar.GetUnicodeCategory() in a future release, but we can go over them in Friday's meeting.

We use ReadOnlySpan as a representation of a slice of UTF16 string. You are proposing we use Utf8StringSegment. Is the discrepancy ok?

Check the comment at the top of https://github.com/dotnet/corefxlab/blob/utf8string/src/System.Text.Utf8/System/Text/StringSegment.cs. It explains in detail why I think this type provides significant benefits that we can't get simply from using ReadOnlySpan<char>.

It would also be a violation of the C# specification, which says that for a class, new has to allocate a new instance.

We do violate the specification in a few cases. For instance, new String(new char[0]) returns String.Empty. Not a new string that happens to be equivalent to String.Empty - _the actual String.Empty instance itself_. Similarly, the Roslyn compiler can sometimes optimize new statements away. See for example https://github.com/dotnet/roslyn/commit/13adbac980ba771d8128449476b6b00021cde203.

What is the relationship between UnicodeScalar and Rune (dotnet/corefx#24093)?

UnicodeScalar is validated: it is _contractually guaranteed_ to represent a value in the range U+0000..U+D7FF or U+E000..U+10FFFF. Scalars have unique transcodings to UTF-8 and UTF-16 code unit sequences. Such transcoding operations are guaranteed always to succeed. Rune (which is not in this proposal) wraps a 32-bit integer which is ostensibly a Unicode code point value but which is not required to be valid. This means that developers consuming invalid Rune instances must be prepared for some operations on those instances to fail.

[svick]

@GrabYourPitchforks

For instance, new String(new char[0]) returns String.Empty. Not a new string that happens to be equivalent to String.Empty - _the actual String.Empty instance itself_.

I didn't know that, interesting.

Similarly, the Roslyn compiler can sometimes optimize new statements away. See for example https://github.com/dotnet/roslyn/commit/13adbac980ba771d8128449476b6b00021cde203.

As far as I can tell, that commit is about Span<T>, which is a struct, so it doesn't violate the C# specification.

UnicodeScalar is validated: it is _contractually guaranteed_ to represent a value in the range U+0000..U+D7FF or U+E000..U+10FFFF. […] Rune (which is not in this proposal) wraps a 32-bit integer which is ostensibly a Unicode code point value but which is not required to be valid.

That doesn't sound like a good enough reason to have two different types to me, especially since you can create an invalid UnicodeScalar. Maybe the two groups could work together to create a single type for representing Unicode scalar values?

[GrabYourPitchforks]

As far as I can tell, that commit is about Span, which is a struct, so it doesn't violate the C# specification.

new byte[] { ... } isn't a struct type. :)

That doesn't sound like a good enough reason to have two different types to me

This proposal assumes that Rune is never committed. So there's only one type in the end.

[whoisj]

I see that it's already committed, but can I just go on record as saying that UnicodeScalar is just a plain terrible name? It really is. It's long, it's generic enough to mean nearly nothing, and it is not even a term the Unicode group uses. I had the same complaints about Rune (with the exception that Rune is at least short`).

This type really ought to be named Character or CodePoint.

I'm mostly OK with the rest of it, though it would be nice if .Split didn't have to allocate quite as much. The underlying data is already read-only - can't Span<T> be used here or something?

[svick]

@whoisj

I see that it's already committed, but can I just go on record as saying that UnicodeScalar is just a plain terrible name? It really is. It's long, it's generic enough to mean nearly nothing, and it is not even a term the Unicode group uses.

"Unicode Scalar Value" is the term Unicode uses for this.

This type really ought to be named Character or CodePoint.

"Character" doesn't really mean anything (Unicode lists 4 different meanings) and would be easily confused with System.Char/char.

"Code Point" is closer, but that term includes invalid Unicode Scalar Values (the range from U+D800 to U+DFFF).

[benaadams]

It's long, ...

The question is, what would the C# keyword be? (Int32 vs int); something like uchar is short 😉 or nchar to match databases

[whoisj]

The question is, what would the C# keyword be? (Int32 vs int); something like uchar is short 😉 or nchar to match databases

This.

Will there be a language word for the type? If there is, you can call the type ThatUnicodeValueWhichNobodyCouldAgreeOnAGoodNameForSoThisIsIt for all I care. I vote for c8 but I also like Rust. Keeping C# in mind, uchar seems the like to no-brainer to me.

@svick yeah, I know that "chartacter" is nearly meaningless hence my suggesting it. I prefer "code point" because how on Earth are you going to prevent me from writing invalid values to a UnicodeScalar's memory? Preventing unsafe is a recipe for a performance disaster; and making unsafe (the real meaning of the word) assumptions about what values a block of memory can contain will lead to fragile and exploitable software design.

[GrabYourPitchforks]

how on Earth are you going to prevent me from writing invalid values to a UnicodeScalar's memory?

Nobody's stopping you. In fact, there's a public static factory that skips validation and allows you to create such an invalid value. But if you do this you're now violating the contractual guarantees offered by the type, I'd recommend _not_ doing this. :)

To be clear, creating an invalid UnicodeScalar won't AV the process or anything quite so dire. But it could make the APIs behave in very strange and unexpected manners, leading to errors on the consumption side. For example, UnicodeScalar.Utf8SequenceLength could return -17 if constructed from invalid input. Such are the consequences of violating invariants.

Unlike the UnicodeScalar type, the Utf8String type specifically _does not_ offer a contractual guarantee that instances of the type contain only well-formed UTF-8 sequences.

[whoisj]

In fact, there's a public static factory that skips validation and allows you to create such an invalid value.

Sure, great, but a lot of the data being read into these structures will be coming from external sources. Very happy to hear that there's no validation steps being taking as the data is read in (because it would be horribly expensive), but still very concerned about:

But if you do this you're now violating the contractual guarantees offered by the type

BUT there is no guarantee - you've said so in your previous statement. There's an assumption, but no guarantee; so let's be careful how we describe this.

[GrabYourPitchforks]

The Utf8String and UnicodeScalar types make different contractual guarantees. I'll try to clarify them.

The Utf8String type _encourages_ but does not _require_ the caller to provide it a string consisting of only valid UTF-8 sequences. All APIs hanging off it have well-defined behaviors even in the face of invalid input. For example, enumerating scalars over an ill-formed Utf8String instance will return U+FFFD when an invalid subsequence is encountered. (Not just that, but the number of bytes we skip in the face of an invalid subsequence is also well-defined and predictable.) This extends to ToUpperInvariant() / ToLowerInvariant() and other manipulation APIs. Their behavior is well-defined even in the face of invalid input.

Exception: If you construct a Utf8String instance and use unsafe code or private reflection to manipulate its data after it has been constructed, the APIs have undefined behavior.

The UnicodeScalar type _requires_ construction from a Unicode scalar value. The API behavior is only well-defined when the instance itself is well-formed. If the caller knows ahead of time that the value it's providing is valid, it can call the "skip validation" factory method. If the instance members off a UnicodeScalar instance misbehave, it means that the caller who originally constructed it violated an invariant at construction time.

The reason for the difference is that it's going to be common to construct a Utf8String instance from some unknown data coming in over i/o. It's _not_ common to construct a UnicodeScalar instance from arbitrary data. Instances of this type are generally constructed from enumerating over UTF-8 / UTF-16 data, and significant bit twiddling needs to happen during enumeration anyway in order to transcode the original data stream into a proper scalar value. Detection of invalid subsequences would necessarily need to occur during enumeration, which means the caller _already_ has the responsibility of fixing up invalid values. The "skip validation" factory is simply a convenience for callers who have already performed this fixup step to avoid the additional validation logic in hot code paths.

So when I use the term "contractual guarantee", it's really shorthand for "This API behaves as expected as long as the caller didn't do anything untoward while constructing the instance. If the API misbehaves, take it up with whoever constructed this instance, as they expressly ignored the overloads that tried to save them from themselves and went straight for the 'I know what I'm doing' APIs."

[GrabYourPitchforks]

FWIW, the reason for this design is that it means that _consumers_ of these types don't have to worry about any of this. Just call the APIs like normal and trust that they'll give you sane values. If you take UnicodeScalar as a parameter to your method, you don't need to perform an IsValid check on it before you use it. Rely on the type system's enforcement to have prohibited the caller from even constructing a bad instance in the first place. (Modulo the caller doing something explicitly marked as dangerous, of course.)

This philosophy is different from the Rune proposal, where if you take a Rune as a parameter into your method you need to perform an IsValid check as part of your normal parameter validation logic since there's otherwise no guarantee that the type was constructed correctly.

[whoisj]

I suppose those are safe-enough trade-offs. Still, too bad the name has to be so unwieldy. :man_shrugging:

[GrabYourPitchforks]

The name doesn't have to be unwieldy. If there's consensus that it should be named Rune or similar, I'll relent on the naming. :)

[KrzysztofCwalina]

We should not call it a "Rune" if it's not a representation for the Unicode Code Point, i.e. let's not hijack a good unambiguous term and use it for something else.

[Tornhoof]

ᚺᛖᛚᛚᛟ᛫ᚹᛟᚱᛚᛞ
Are you sure about Rune? It's a Unicode Block after all.
Maybe rather grapheme?

[KrzysztofCwalina]

I think in graphemics (branch of science studying writing) rune is indeed a grapheme. I think in software engineering, rune is a code point. But possibly it's not such a clear cut as I think. The point I was trying to make is using "rune" to mean Unicode Scalar would be at least yet another overload of the word "rune".

[Tornhoof]

Thank you for clarifying that.

[benaadams]

The UnicodeScalar is a more sane type to use than char as it always encompasses a full character rather than potentially half a character, which char can do.

However, while UnicodeScalar is fine for a library type it isn't great for common usage for people that don't like var as you'd go from the less correct

foreach (char ch in str)
{
    // ...
}

to the more correct

foreach (UnicodeScalar scalar in ustr)
{
    // ...
}

which is a less desirable, very verbose, code representation for a single character.

[KrzysztofCwalina]

Aside for reusable library developers, I don't think many people will be using (referring to) UnicodeScalar. Most people will create or get an instance of Utf8String, and the instance methods on this type, and most of these either don't use UnicodeScalar, or will be used by passing a literal ('a' or 65). If somehow the type becomes super popular, we can think about adding a language alias.

[benaadams]

I think you might be underestimating the usage...

ReadOnlySpan<UnicodeScalar> values = new UnicodeScalar[] {'😊','😎','😥','🎄'};
int index = str.IndexOfAny(values);

Or iterating over chars to determine what emoji-range the character fits into for "sentiment" analysis

[KrzysztofCwalina]

Yeah, my teenage daughter also thinks I underestimate the value and usage of emojis :-) And I think you both might be right :-)

[benaadams]

Unless you are doing interop; Utf8String is probably going to be the go to string type. Its smaller when ascii; deals with whole characters and is more efficient when transferring on the wire (as its already in the correct format so needs no transcoding)

Whereas string is twice the size when ascii and deals in half characters; you have to worry about byte order; and often you have to repeatedly transcode to utf8. In my experience people just ignore the half-character issue so most current string handling is wrong*. Using Utf8String means most string handling would be correct by default.

So I think use of UnicodeScalar may be higher than anticipated.

*From my limited observations

[whoisj]

However, while UnicodeScalar is fine for a library type it isn't great for common usage for people that don't like var as you'd go from the less correct

QFT. This is my issue, @benaadams has hit the nail on the head here. Fairly sure we cannot expect char and string to be co-opted by UnicodeScalar and Utf8String but it would be lovely if they could be.

[migueldeicaza]

Disagree with the idea that this won’t be used extensively: it will be used everywhere a char is used today. Almost every use of Char needs to be replaced by this.

You can search for ‘char’ in any .NET code base to get an idea of how often people use this primitive type.

char is essentially broken for proper processing, we only survive because most people brush this off as ‘something went wrong on a corner case’.

Basically, .NET today encourages subtly broken code by default which you can make correct with a lot of work.

We need to strive to create an environment where we make it easy for people to write correct code from the start, for them to fall into the pit of success.

Long names like ‘UnicodeScalar’ are just going to prevent people from embracing it by default. The notion that “we will wait and see if there is demand” is a self fulfilling prophesy into failure.

Research wise, just grep for ‘rune’ in the go codebase to show you how misguided the idea that his is a rare type.

Go here ensures that any beginner gets correct code from the start. We ended up in the other extreme.

[benaadams]

Yeah, my teenage daughter also thinks I underestimate the value and usage of emojis :-)

Windows even has an emoji keyboard...

We've seen a huge rise in the use of emoji and the odder variants of unicode (script, upside down letters, lookalike chars etc) in our consumer focused applications. Initially we we resistant to it; but at this point its basically time to wholly embrace it as just the way things are.

So we will be moving to Utf8String for everything when its available (other than system calls); and using it for all new projects.

Also ASP.NET Core should consider moving precompiled Razor pages to be Utf8String based rather than string based and going though Encoding.UTF8 for every static string in every page request.

[benaadams]

Also all these methods are broken for supplementary plane chars/emoji

partial class String
{
    bool Contains(char ...);
    int IndexOf(char ...);
    int IndexOfAny(char[] ...);
    bool EndsWith(char ...);
    string Join<T>(char, IEnumerable<T>);
    string Join(char, ...);
    int LastIndexOf(char, ...);
    int LastIndexOfAny(char, ...);
    string PadLeft(int, char);
    string PadRight(int, char);
    string Replace(char, char);
    string[] Split(char ...);
    string[] Split(char[] ...);
    bool Trim(char);
    bool Trim(char[]);
    bool TrimEnd(char);
    bool TrimEnd(char[]);
    bool TrimStart(char);
    bool TrimStart(char[]);
}

So should the following additional overloads be added to regular string?

partial class String
{
    bool Contains(UnicodeScalar ...);
    int IndexOf(UnicodeScalar ...);
    int IndexOfAny(ReadOnlySpan<UnicodeScalar> ...);
    bool EndsWith(UnicodeScalar ...);
    string Join<T>(UnicodeScalar, IEnumerable<T>);
    string Join(UnicodeScalar, ...);
    int LastIndexOf(UnicodeScalar, ...);
    int LastIndexOfAny(UnicodeScalar, ...);
    string PadLeft(int, UnicodeScalar);
    string PadRight(int, UnicodeScalar);
    string Replace(UnicodeScalar, UnicodeScalar);
    string[] Split(UnicodeScalar ...);
    string[] Split(ReadOnlySpan<UnicodeScalar> ...);
    bool Trim(UnicodeScalar);
    bool Trim(ReadOnlySpan<UnicodeScalar>);
    bool TrimEnd(UnicodeScalar);
    bool TrimEnd(ReadOnlySpan<UnicodeScalar>);
    bool TrimStart(UnicodeScalar);
    bool TrimStart(ReadOnlySpan<UnicodeScalar>);
}

Or would they go via an implicit conversion to string; and the string overloads? Or would the C# compiler change the embedded type from UnicodeScalar to string depending on what overload was available at the call site?

This would have been popular/memed recently for some unknown reason:

string.Join('👏', words)

[ufcpp]

A example of why we should use UnicodeScalar is the C# compiler. It has a bug on using surrogate pairs:

class Program
{
    static void Main()
    {
        // Error CS1056 Unexpected character
        int 𩸽 = 2; // CJK Extension B
        int 𒀀 = 3; // Cuneiform
        int 𓀀 = 5; // Egyptian Hieroglyph
        System.Console.WriteLine(𩸽 * 𒀀 * 𓀀);
    }
}

Unicode category of these characters are Lo (Other Letter). And Lo characters can be used for identifiers in the C# spec.

Many language other than C# can use surrogate pairs correctly.

Go:

package main
import "fmt"
func main() {
    𩸽 := 2
    𒀀 := 3
    𓀀 := 5
    fmt.Println(𩸽 * 𒀀 * 𓀀)
}

Java:

public class HelloWorld
{
  public static void main(String[] args)
  {
    int 𩸽 = 2;
    int 𒀀 = 3;
    int 𓀀 = 5;
    System.out.print(𩸽 * 𒀀 * 𓀀);
  }
}

[KrzysztofCwalina]

Let me clarify: I definitely don't think we should be using "char" or any such type in Utf8String APIs. The APIs need to be 100% reliable, and UnicodeScalar is the only way to do it, i.e I am a big fan of UnicodeScalar.
What i was saying is that I don't think users will have to refer to the type often, e.g
utf8String.Split('a') would call a method taking UnicodeScalar, and C# compiler would target type the parameter, i.e. there would be not Char created and then converted to UnicodeScalar; C# would create scalar value directly.

[whoisj]

The APIs need to be 100% reliable, and UnicodeScalar is the only way to do it

Agreed on the concept that a 32-bit Unicode scalar type needs to be present, and the string (notice not string) types need to utilize that and not the 16-bit char type.

I think the type will not be referred to often enough to justify adding language alias, at least not initially.

I disagree. I tend to have code littered with char[] and const char declarations. I just opened a random project and did a search for \bchar\b and in 72 files I got 155 hits. Having to type / look at UnicodeScalar in place of all of those char keywords would be... uh... less than optimal, yes - let's put it that way because it sound pleasant and professional: "less than optimal".

[GrabYourPitchforks]

Since the question of graphemes came up, I'll mention that we've been punting on the idea of having graphemes as a first-class citizen in the framework. (By "grapheme", I mean interpreting the 2-scalar sequence [ U+1F474 Older Man Emoji, U+1F3FF Fitzpatrick Type 6 Skin Modifier ] as the single grapheme "Older man with Fitzpatrick type 6 skin".) The reason we had been punting on this is that it tends to be more of a UI / text editor concern - not a general framework concern - and there is the existing TextElementEnumerator type if you're willing to pull up your sleeves.

@migueldeicaza, I think you had some early thoughts on this a while back. Has your thinking changed on what kind of support we should have in-box for this? Is this really a concern for the BCL, or does it properly belong in a separate package?

[GrabYourPitchforks]

Some other feedback now that I'm rereading this thread.

@benaadams - I agree that we should add UnicodeScalar-accepting APIs to System.String if there's a need to do so. If the majority use case is that developers are searching for literals, the existing APIs like String.IndexOf("<my multi-char emoji>") already work just fine.

@whoisj - Regarding your const char fields, what if the compiler could implicitly convert char to UnicodeScalar? That way your call site would look like myUtf8String.IndexOf(my_const_char). We can also consider adding an explicit conversion operator between the two types.

[migueldeicaza]

Some comments on the API from last week:

Given that it should be possible to create Utf8Strings from invalid operations, we need a way of returning a value that indicates that there was an error processing the utf8 sequence in the buffer and indicate that this was caused due to this error. NStack has a port of Go's libraries that do this.

It is not clear why Length of Runes should be O(1), seems like a waste of space, specially considering that iterating over the values is not O(1) anyways.

When processing utf8strings you really want to have access to the byte-length, that is missing.

I would add a few things:

• Utf8String to Rune array (spit it out as an array of Runes, or an IList)
• It is not clear what Length is, whether it is the number of bytes in the buffer, or the number of runes on it. Intuitively it is the number of bytes, if so, we should add a RuneCount property that returns the number of runes.
• Split on Rune is missing
• Join methods.

There is a proposal for UnicodeScalar, you should lift the operations I submitted before on a better named type, Rune that has a comprehensive API that I have been using for a while.

I don’t think that it is a good idea to limit UnicodeScalar to valid values, I think you should instead have an IsValid method.

My feeling is that if you want a string with training wheels we have System.String already - but a case should be made for a System.UnicodeString that is made up 32-bit tunes, that has O(1) indexing capabilities.

One nice capability that the Go API has is that enumerating over the Runes in the string is not limited to obtaining the individual runes, but also the offset where the rune was found. In NStack, I have a similar method that returns a tuple (int index, Rune rune) that achieves this.

[GrabYourPitchforks]

See also the UTF-8 string scenario and design philosophy document (https://github.com/dotnet/corefxlab/issues/2368).

[GrabYourPitchforks]

There's some confusion over runtime complexity of the Utf8String length APIs. Fetching the code unit count ("byte length", if you will) is O(1) complexity. The Utf8String type internally doesn't keep track of the total number of scalars ("runes", if you will), and there's no API on Utf8String to fetch this count. There are other APIs which will give you this information, but the proposal here doesn't expose those APIs on Utf8String.

[GrabYourPitchforks]

I've added the missing APIs to the Tuesday review. Thanks for the eagle eyes @migueldeicaza!

[whoisj]

As a note, it seems to me that Utf8String.Length ought to be the byte length of the underlying array (it is an array, isn't it - are we considering using linked buffers or something - doesn't really matter).

Given that Utf8String are likely immutable, and therefore the count of characters cannot be cached in the type after initialization; and nobody in their right mind would suggest computing the actual character count of a Utf8String at allocation, we likely need a .Count() method which does the calculation when invoked.

The Utf8String.Enumerator implementation ought to be interesting. There are a lot of way to do this, none of which are particularly pleasant. 😕

@GrabYourPitchforks if you looking for any parallel implementations (aside from the stuff by @migueldeicaza) let me know and I can send you a few links to internal source we use to handle Git strings (Git is Utf8 through-and-through) - ironically, we called our type StringUtf8 😏

Also, if you're just plain sick of my feedback let me know and I'll go sulk in a corner quietly :grin:

As a complete, and mostly unrelated side-note, I've always hate the String API which take StringComparison. I would so incredibly rather every API took a StringComparer implementation and we could leave the annoying, and mostly useless StringComparision enumeration in the bin.

Any chance we can avoid dragging it into the future via this API set? 🙏 :bow: 🙏

[whoisj]

My feeling is that if you want a string with training wheels we have System.String already - but a case should be made for a System.UnicodeString that is made up 32-bit tunes, that has O(1) indexing capabilities.

On this topic, I cannot count the number of times that a p/invoke call to some library has returned corrupted or invalid string values; and since System.String lacks any ability to self validate, I always end up adding validation routines to project. Seems like something we ought to avoid in the future, ala @migueldeicaza recommendation (this is what I was trying, and failed to illuminate above).

Oh, and in case it needs to be said: much ❤️ and 🙇 for @GrabYourPitchforks for even working on this API. It is long over due and is such a hot topic; it's pure heroism to work on it. 😃

[GrabYourPitchforks]

@whoisj By self-validate a System.String instance, you mean looking for mismatched surrogate pairs? I considered in this project adding validation APIs to both Utf8String and String but ultimately decided against it for a few reasons. I didn't want developers to feel obligated to call it before consuming the instances. And for String in particular, it's generally very difficult to create a malformed instance in the first place without bit-twiddling a char[]. If you're running into the need to validate in a production app I'm certainly willing to reconsider those decisions.

@whoisj What's your concern with StringComparison? Are your scenarios working with specific cultures rather than the invariant culture or the current thread's culture?

@migueldeicaza Interestingly, we considered a fully UTF-32 string type a few weeks ago, and I don't think it's a crazy idea. The primary scenario we came up with was a text editor or other UI-based application. I think if we wanted to give that scenario proper respect we'd also want to consider grapheme representation in the framework and plumb it through as an in-box concept. Do you think server applications might need this in addition to UI applications? (As an aside, C++'s std::wstring on most non-Windows platforms is UTF-32, and it seemingly doesn't enjoy wide use.)

[whoisj]

@whoisj What's your concern with StringComparison? Are your scenarios working with specific cultures rather than the invariant culture or the current thread's culture?

More often that not, I work on library code that interops with external software. I primarily am concerned with Orignal and OrdinalCaseInsensitive, very-very rarely do I need to care about culture.

Often, I need to produce custom string comparers, when this happens every entry-point on string that takes a StringComparison becomes useless to me and I end-up re-implementing them.

What kind of custom string comparer could one be writing that isn't provided by NetFx? Well, several projects I've worked on recently needed custom file system path comparers, and for the compare to chosen based on conditions. For example, on Windows paths that do not begin with "\?\" treat '/' and '\' interchangeably, thus a custom comparer is necessary. I could just
C# OrdinalCaseInsensitive.Equals(lhsPath.Replace('/', '\\'), rhsPath.Replace('/', '\\'))
but we literally compare thousands of paths in certain cases and thrashing the HEAP with needless string allocations is really terrible. So instead, I have chunking logic which breaks on path separators... blah, blah, blah.

Now consider the logic necissary for something like bool IsChildPath(string parentPath, string childPath). Internally I could use string.StartsWith(...) but it doesn't accept a StringComparer, instead it takes a StringComparison. Leaving me to author my own static bool StartsWith(this String value, StringComparer comparer) method. If System.String.StartsWith accepted a StringComparer life would just be better.

@whoisj By self-validate a System.String instance, you mean looking for mismatched surrogate pairs?

Exactly.

I didn't want developers to feel obligated to call it before consuming the instances.

Developers should not feel obligated to use validation API, but the lack of a validation API can cause heartburn. Consider the developer who is writing software that reads data from a stream or shared memory. There's always a change something got corrupted, so having a built-in way to validate the data would be rather useful. Perhaps developers writing code like this are rare enough that NetFx doesn't need it, in which case I'll continue to keep writing my own. 😁

... we considered a fully UTF-32 string type a few weeks ago, and I don't think it's a crazy idea. Do you think server applications might need this in addition to UI applications?

I can see utility in an indexable string type, but it'll be very specialized. I'd much rather see the work your doing here stay the focus. UTF-8 as the internal encoding for character data is extremely valuable, especially when memory isn't plentiful and cheap.

... oh, as an aside - are there going to be Utf8StringComparer types provided? If so, have you thought about the implementation details yet?

[mburbea]

Here's a thought for the Utf8String type, what if we allow you to use single quotes to be the literal?
With the exception of the case of a single character that could be a char, I believe we can make it safe.
Otherwise people may forget to prefix it. It can also be target typed in the case of a single character literal.
e.g.
```cs
var empty = ''; // empty Utf8String
var emoji = '🎄'; // Utf8String of a tree emoji. Can't be a char, as it is a pair of surrogates.
var word = 'word';
var @char = ','; // System.Char with the value 0x20
Utf8String singleCharString = ' ';

[benaadams]

I assume single quotes would be UnicodeScalar?

UnicodeScalar emoji = '🎄'; // valid
var emoji = '🎄🎄'; // compile error
Utf8String emoji = "🎄🎄"; // valid

[ufcpp]

@mburbea

There are too many characters indistinguishable in appearance.

var x1 = "A"; // U+41, ASCII
var x2 = "Ä"; // U+C4, Latin-1 but not ASCII
var x3 = "Α"; // U+391, UTF-16, but not Latin-1
var x4 = "𝙰"; // U+1D670, 2 characters in UTF-16

var y1 = "❤"; // U+2764, 1 character in UTF-16
var y2 = "🧡"; // U+1F9E1, 1 unicode scalar, but 2 chars in UTF-16

var z1 = "🤹"; // U+1F939, 1 unicode scalar
var z2 = "🤹‍♀️"; // U+1F939 U+200D U+2640 U+FE0F, 1 grapheme, 4 scalars
var z3 = "🤹‍♂️"; // U+1F939 U+200D U+2642 U+FE0F, 1 grapheme, 4 scalars

[ufcpp]

// char
char c1 = 'a';
var c2 = 'a'; // valid. inferred as char

// UnicodeScalar
char u1 = '🎄'; // invalid
UnicodeScalar u2 = '🎄'; // I want this is valid
Utf8String s = '🎄'; // NEVER valid

var u3 = '🎄'; // Should be valid? inferred as UnicodeScalar? I don't want, but...
// This is similar to integer literals
var x = 4294967295; // uint
var y = 4294967296; // long

[whoisj]

@ufcpp I believe you're crossing the concepts of grapheme and scalars, which you actually point out yourself above. Given that the value type needs a fixed size, and given the hardware that exists today and for the foreseeable future, I'd guess that we'd land on a 32-bit scalar value type; whereas a grapheme type would likely need to be 128-bit. We need to be very careful here to not confuse content and composition.

The Unicode organization says that all Unicode character values can be represented with 20 bits or less, that means our containing type ought to be 32-bit (smallest power-of-two larger than 20).

All of that said, I have developed multiple (sad I know) Utf-8 interop libraries at this point in my career, and often I take the "easy" route of defining a 64-bit code-point type. Example (truncated):

```C#
[StructLayout(LayoutK.Explicit, Size = 8)]
unsafe struct Codepoint
{
[FieldOffset(0)]
private fixed sbyte _utf8[8];

[FieldOffset(0)]
private fixed char _utf16[4];

[FieldOffset(0)]
private fixed uint _utf32[2];

}
```

[ufcpp]

@whoisj
What I want to say is that type inference from a character is confusing. ❤ and 🧡 have very similar shape but ❤ can be char while 🧡can not.

a grapheme type would likely need to be 128-bit.

needs more. For instance: 👩🏻‍👩🏿‍👧🏼‍👧🏾 is a grapheme which consist of 11 scalars, 41 bytes in UTF-8.

[whoisj]

👩🏻‍👩🏿‍👧🏼‍👧🏾 selects and copies as 4 discreet characters for me; sure the kerning is unusual, but the net effect is visually pleasing. Appears to be (or very similar to) the string "\u1F469, ‍\u1F468, ‍\u1F467, ‍\u1F467".

Regardless, I think you're correct - grapheme would probably be treated as a string-like (read: array of scalars) value.

[ufcpp]

← 👩🏻‍👩🏿‍👧🏼‍👧🏾 in Windows 10, Chrome.

see also https://www.youtube.com/watch?v=2U_RvWbXGtI

According to Unicode® Standard Annex #29, any number of pictographics joined with ZWJ (U+200D) is a grapheme. For instance 👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻‍👩🏻, (U+1F469 U+1F3FB U+200D)×n is a grapheme, treated as one character in Windows 10.

[whoisj]

any number of pictographics joined with ZWJ (U+200D) is a grapheme.

So it is a string-type then 😁

[GrabYourPitchforks]

@whoisj We were considering adding the appropriate IComparer<Utf8String> interface to the existing StringComparer class. It's a little unorthodox, but it satisfies the "my code that I typed using muscle memory works correctly" nicety.

[benaadams]

Will Utf8String have value equality with string? e.g.

public static bool operator ==(Utf8String left, string right) 
public static bool operator ==(string left, Utf8String right) 

As string and Utf8String have known encodings; so should be able to be compared without allocating the other type to change the encoding to compare.

Similarly

ValueEquals(this ReadOnlySpan<char>, ReadOnlySpan<Char8>)
ValueEquals(this ReadOnlySpan<Char8>, ReadOnlySpan<char>)

[GrabYourPitchforks]

@benaadams Undecided. I've been experimenting with operator ==(Utf8String, string) as a convenience method for equality comparisons. I've also been experimenting with adding those operators with a "referential equality only" implementation and slapping [Obsolete] on them to discourage their use. In either case we can't add the overloads until we figure out what to do about https://github.com/dotnet/csharplang/issues/2340.

I'm not opposed to having methods that perform equality comparisons of UTF-8 directly against UTF-16. The difficulty IMO is figuring out how to expose these methods while at the same time making as optimized as possible code that developers are likely to write in the real world.

[benaadams]

Ah, yes, unfortunate :-/

[GrabYourPitchforks]

Heads up to anybody following this - I've updated the proposal description as of 04 Sep 2019. There are significant changes from the previous iteration from last year.

Summary of changes:

• Removal of Char8 (and Char8[], ReadOnlySpan<Char8>, etc.) in favor of explicit slice type Utf8Span.
• Utf8String / Utf8Span now include as part of their contract that they represent well-formed UTF-8 data.
• More APIs to operate on slices, including slices of raw bytes (not necessarily valid UTF-8), and allowing callers to bypass validation if they promise to uphold the normal Utf8String / Utf8Span invariants.
• Better C# language integration.
• Removal of the this[int] indexer on Utf8String in favor of Range-based slicing / substring APIs. There was considerable feedback that the indexer was confusing, so now enumeration is performed directly via the Bytes, Chars, or similar property.

[idg10]

Ultra-pedantic point, but sometimes this sort of thing ends up in the docs... In the discussion at the top of this issue, you have code including the following:

```csharp
buffer.InsertAt(0, utf8("💣")); // U+1F483 ([ F0 9F 92 A3 ])
````

I think the comment is inaccurate. Codepoint U+1F483 (DANCER) looks like 💃. The character you have in the code is U+1F4A3 BOMB (not to be confused with U+FEFF). You should change it either to:

```csharp
buffer.InsertAt(0, utf8("💣")); // U+1F4A3 ([ F0 9F 92 A3 ])
````

or

```csharp
buffer.InsertAt(0, utf8("💃")); // U+1F483 ([ F0 9F 92 83 ])
````
depending on which you actually meant. I'm guessing it was the first, since the UTF8-encoded version you already have in the comments is consistent with the character in the code.

[CyrusNajmabadi]

Hey Corefxlab. I'm highly concerned about how the approach taken here will likely lead to an API that will only be suitable in niche scenarios, and will have significant issues that prevent widespread adoption (and thus limit the ability for the ecoystem to get the sorts of perf benefits that would be very desirable through broad usage of utf8 strings over 16bit encoded strings. I've made two large posts on the topic over at csharplang here and here.

I specifically outline how this approach will face major problems with adoption in libraries, leading to a chicken/egg dilemma where libraries will not be able to adopt this API because of breaking end user libraries/app, and end user library/apps won't be able to use it because of libraries. I use Rolsyn as an example of a library that would benefit hugely from utf8 but would also likely never be able to move to it in the form specified here (and at csharplang).

I also outline an approach that would be costly, but would actually lead to a path where the ecosystem as a whole could actually find a path to move over to utf8 en masse. With that in mind, here are copies of hte posts i wrote there which may hopefully spark a substantive discussion here toward producing a solution that gives the biggest benefits to the broadest groups possible:

[CyrusNajmabadi]

Post #1

--

I'm not a fan of this approach as it treats utf8 strings as something other that then needs to be brought in through a side-channel.

It seems this fundamentally could not be picked up by a library author. i.e. if i have a library and i'm already using System.String (highly highly likely), i can't switch to utf8 strings because it will break all my consumers.

And, if i don't use utf8 strings, similarly my consumers will be less likely to as well since they would not want the costs marshalling to/from all libs.

--

I talked to @jcouv about this and the approach that feels like it would be most likely to succeed would be to provide a way to switch the .net runtime to/from utf8 mode (on a process boundary most likely). The benefits here are:

1. users can switch over everything entirely to utf8 when it is acceptable for their domain.
2. most apps would immediately get a near 50% reduction in memory for all their strings (iirc measurements showed that 90%+ of all strings are simple ascii).
3. apps/libraries get switched over all at once based on the needs of the final consumers.

There is a downside in this that often gets brought up. Namely that utf8 strings do have different perf behavior for some ops over strings (namely indexing). However, this doesn't actually seem like a critical problem to me. First, remember that what i'm proposing involves a switch (either opt-in or opt-out) to use ut8 across the board. As such, if someone is in a domain where they index heavily and get a perf hit, they can not use utf8 until they address that problem. Second, i think the problem seems somewhat overblown in terms of how bad it is. We can likely break string indexing up into two domains:

1. people streaming through a string with monotonically increasing indexes. This can be addressed by:
   
   
   
   1. pushing those people (with analyzers) to use iterators instead.
   
   
   2. having the runtime be slightly smarter with string indexing. like many utf8 systems out there it could store additional information in the runtime about the last index operation that happened on hte last few strings. If the user passes in str[i] and then str[i + 1] the information about the locatin collected in the first op can be used to make the second fast.
   
   
2. people randomly accessing string indices. This seems like this would be a very small subset of users. And, if that space was truly important, they:
   
   
   
   1. could opt-out of utf8 strings
   
   
   2. could use some new type that guaranteed constant random access for a string. maybe a new Utf16String, or just a char[] or ImmutableArray
   
   

Basically, it feels like there is a path that can get us to a future where almost everyone (final consumers and libraries alike) are on utf8 and the entire ecosystem gets the massive memory savings. It comes at the complexity of having opt-in/out and potentially needing some analyzers/classes for the people using strings in uncommon ways today. However, it seems much better to me than introducing a new utf8 string type that is highly unlikely to be picked up.

[CyrusNajmabadi]

Post #2:

---

As an example of how we have a problem, take a look at Roslyn itself, including the entire Roslyn API we ship.

1. it is massively System.String based everywhere.
2. It uses a huge amount of memory internally with strings. IIRC measurements have shown it's >50% of our memory usage in compiler and IDE.

How could Roslyn itself possibly get the benefits of utf8 strings?

1. We could try switching to it internally, but our marshalling points between the internal and public layers would kill us. For example, every time the IDE accessed a string-property exposed by the compiler, we would take a marshalling hit. And we access those string-properties continuously.
2. We could try to expose both types of strings somehow? allowing consumers to move to utf8 when possible, while still having the System.String property. But how would this look? .Name and .Name8? How would memory not explode in such a world?
3. We could switch wholesale over to utf8 strings for our entire surface area. But that would break 100% of the ecosystem out there.

Effectively, afaict, a project like Roslyn could never move to utf8. And we're one of the projects that would benefit the most here. We likely would save gigabytes of memory on real projects on user boxes.

So, as mentioned in teh start, this overall approach seems highly limited and constraining. it will only help projects that are isolated and can completely switch over without having to worry about dependencies. The overall ecosystem will find it nearly impossible to switch.

Conversely, the approach I outlined gives a path forward that allows big saving immediately across the board, with appropriate mechanisms for people to deal with rare problems if they arise. Then, if problems do occur in some places, they can be fixed up without holding the rest of the ecosystem back.

[miloush]

I agree with @CyrusNajmabadi. If this is a type meant for developers to work with UTF-8 data, then it belongs to Utf8Encoding - I don't see why any of the functionality above should not be available for UTF-16 strings for example.

If this is to save space or improve speed, it should be CLR's business to decide how strings are represented in memory and it should be completely transparent to the developer (at least in the safe context). The UTF-8 encoding could then be a no-op from internal representation.

I would prefer a runtime solution even to a compiler switch.

[CyrusNajmabadi]

It's important to call out that Utf8String is not intended to be a replacement for string. The standard UTF-16 string will remain the core primitive type used throughout the .NET ecosystem and will enjoy the largest supported API surface area.

This is highly unfortunate and it seems extremely limiting and (not to be too negative) very shortsighted. utf8 strings are enormously valuable across the ecosystem. I brought up roslyn as a case, however I've seen data over hte years showing how many projects would routinely get double-digit percentage memory savings (often 30-40% or more) by using utf8 strings.

This is sufficiently valuable to warrant trying to go for an approach that doesn't treat these as an "out of band" solution to a few side cases, but rather just making it the approach for strings for .net. Note that this sort of change is not something that is without precedent. Many platforms have made such a switch because of hte absolutely staggering benefits, even when it comes with some issues that may have to be dealt with in niche cases. A great example of this are webrowsers. AFAICT not, all the major browsers have switched over to an internal utf8 representation. Even though this did potentially impact some programs, the benefits were so unbelievably huge to the ecosystem that it was worth it. I feel the same holds for .net and it would be very unfortunate to not work toward the same end state.

[CyrusNajmabadi]

I would prefer a runtime solution even to a compiler switch.

FWIW, that's what i'm proposing. This is a switch at the runtime level. The switch allows for the ecosystem to move over at a pace that makes sense for the end consumer. Consumers that don't experience pain moving to utf8 (i.e. because none of their code/libs contains patterns that fall off a perf cliff with utf8 strings) can just move right over. Consumers/libs that have problems can advise that people not opt in just yet while they update themselves to be good utf8 citizens.

Importantly, libs can continue to work with both types of consumers. For example, Roslyn could work with utf16 consumers as well as utf8 consumers.

Eventually, once enough of the ecosystem had moved over, this switch could then change its default. For example, being opt-out instead of opt-in. And then, once enough time had passed and hte ecosystem had demonstrated sufficient adoption and acceptance of utf8, the code to even handle the utf16 switch could potentially be removed.

Does this entail more work in some areas? Yes:

1. The runtime needs to be able to have a switch when loading that determines internal string representation. That is new with my proposal.
2. runtime needs to provide the APIs that work properly on utf8 and utf16 strings (though that is true with this proposal as wlel). Except now it's all through one surface area instead of two.
3. it's likely that tools/analyzers need to be written to help people move bad patterns to good ones.

However, this also entails less work in others. For example:

1. we don't need a new string API.
2. we don't need to update vast amounts of APIs to accept this new Utf8String.
3. libraries don't need to update themselves. they are automatically usable by utf16 platform consumers and utf8 platform consumers.

The proposal from corefxlabs reduces cost in the wrong places. It effectively makes it simple to add a new type, while making it effectively impossibly expensive to get real adoption across the ecosystem**. The proposal i'm putting forth puts more cost on corefx/coreclr, but provides an actual migration/adoption path for the entire ecosystem to get the benefits of utf8 strings in a gradual manner.

--

** If you think the corefxlabs approach is not impossibly expensive for libraries to adopt, then I ask that you please outline exactly how a library like Roslyn could ever move to utf8 strings (which would be hugely beneficial) without the serious problems i raise in https://github.com/dotnet/corefxlab/issues/2350#issuecomment-548101966. Any real utf8string plan, in my opinion, needs to actually come along with a serious and substantive analysis and outline of how established libraries can actually move over to this in an cost effective manner. "cost effective" including, but not limited to:

1. actual development cost to implement/maintain.
2. actual runtime cost in terms of memory usage for supporting utf16 and utf8 strings.
3. actual runtime cost in terms of CPU to deal with conversions between strings
4. actual cost to consumers to adopt this lib

[stephentoub]

@CyrusNajmabadi, what are you proposing be done for code that does:
```C#
fixed (char* ptr = str)
{
...
}

```C#
ReadOnlySpan<char> span = str;
...

?

The above has always been a sticking point in the myriad number of times in the past (long and recent) that a mode that makes System.String be UTF8 instead of UTF16 has been discussed.

[CyrusNajmabadi]

@CyrusNajmabadi, what are you proposing be done for code that does:

As a starting point, recall that my proposal starts with a runtime switch at (i believe) the process level (or whatever normal boundary we think of where the heap/jit/etc. exist for coreclr).

fixed (char* ptr = str)

So, when answering that question we have to consider two cases:

1. the process/clr was launched with 16bit strings (i.e. what we have today). In that case, there is no issue and you get the same behavior as today. That's not very interesting, so from now on i'm just going to ignore this case and not restate it.

2. the process/clr was was launched with utf8 strings. For now, let's assume that is an opt-in switch. In this case, i would be ok with the above throwing since it assumed something about the internal representation of a string. This could be something that could also be found by an analyzer as well. Unsafe code is exactly that case that needs special handling and which is why we would likely need the switch to begin with.

So now that we're throwing what can the author/library do about this? Well, i think teh right approach would be to tell people: hey you wrote unsafe code. If you want to be able to move to the brave new world you need to consider how your unsafe code should operate with a utf8 representation. i.e. you can do something like this:

c# void Foo() { if (!RuntimeWhatever.IsUtf8) { fixed (char* c = ...) } else { fixed (byte* b = ... // or, ideally, maybe there are other built-in intrinsics that can work well for you now that // we're so focused on perf. so you don't need unsafe anymore. } }

Or ReadOnlySpan<char> span = str;

I would do the same. This would throw in utf8 mode and that's ok with me. It just indicates to me that this library isn't ready to be loaded in a utf8-mode clr. And we shoudl make that clear and provide very strong and easy guidance on how to provide the alternative algorithm on a string that works in utf8 mode using the utf8 apis, or ROS<byte> or utf8segement/span/etc.

--

The core idea here is that ideally the above is the rarer case and can be isolated to libraries with clear work on what they can do to get "utf8/utf16 split" clean. Once clean, they can then be loaded into a CLR with either environment and can work fine. The point of hte split/option at the CLR level is to actually ensure that when the above happens (and it will happen) people have a way to stay working until their dependencies get clean.

However, the major benefit of htis approach is that this can happen at the pace that makes sense for the different parts of the ecosystems. Some people may be able to move over immediately. Some may have a tiny amount of code (or some libs with tiny amounts of code) that need a little work. They can patch quickly and release a new version that gets them moved over.

Some corners of the ecosystem may take a bunch of time. For example, i could easily envision a string-based api (maybe some really powerful regex system) that is so dependent on unsafe code that it takes a significant amount of time to move over. In my world, that's ok. They continue to work on utf16 systems, and people can continue using utf16 until that lib is fixed up. Importantly though, this doesn't limit others who aren't blocked by a need to have that lib.

Does that makes sense @stephentoub ?

[stephentoub]

Does that makes sense @stephentoub ?

It makes sense in that I understand what you're proposing. Thanks.

[mburbea]

Even safe code would be impacted:

foreach(char c in myString){
} // silently casts elements from byte to char.

What if the switch also changes char to mean a utf8 byte?

[miloush]

@mburbea in my view that example should not cast from byte to char, it should behave exactly as now and it's runtime job to ensure that regardless of the internal representation it chooses for string

[CyrusNajmabadi]

Even safe code would be impacted:

I don't see that being the case. Supporting that scenario seems like it would be fine as string would still expose a performant, same-behavior char-enumerator in utf8 mode.

That woudl actually be a big benefit here as i imagine this could be done fairly easily. I also expect the runtime to detect and optimize cases like this:

```c#
for (int i = 0; i < v.length; i++)
{
v[i]
}

This sort of striding access to a utf8 string can totally be done efficiently through the use of a little extra runtime storage.  There is *ample* precedent for this with many runtimes doing exactly this to prevent falling off an `n^2` perf cliff.  i.e. say i write the common sort of "string splitting" loop like so:

(note: code not tested):

```c#
List<StringSpan> Split(string s, char c)
{
     int currentIndex = 0;
     int next
     while ((next = s.IndexOf(c, from: currentIndex) >= 0)
     {
          s.SubSpan(from: currentIndex, to: next)
     }
}

This woudl normally be n^2 as the .SubSpan operator would have to go scan through the utf8 string again to find that index within it.

However, this doesn't have to be the case the runtime can do things like keep an internal cache somewhere of the last few strings to the last index operation on it (with the presumption that the next indexing operation will likely benefit from the last index returned). So, when called again, it can use the previous position stored with the previous index to more quickly find the next position for the next index.

It would def add complexity. But it would make things stay close to linear, preventing immediately perf cliffs for common cases.

For code that was literally random, you would fall off a perf cliff. However, like with 'unsafe' this would be a case that would warrant telling people: you shoudl continue using utf16 strings until you can update that algorithm to be more efficient with a variable-width encoding of strings.

[CyrusNajmabadi]

It makes sense in that I understand what you're proposing. Thanks.

Absolutely! Note that i don't pretend my approach is easy or without serious areas of concern. My overall point is that this area is so impactful, and 'strings' (in the abstract sense) are so core to the entirely .NET development model through, that it warrants a much more "complete story" that actually acknowledges and tries to come up with a plan for how the ecosystem can actually fully embrace this, instead of it only being a side channel for niche scenarios.

With Span/Ref/etc. i was totally ok with it being super advanced and something that could be bolted on. With the 'string' space, i dont' see that being at all viable. And i see not having a plan here for fully embracing utf8 as something that will significantly effect .net and its entire ecosystem in the future.

[CyrusNajmabadi]

From discussing things on gitter here, i'd like to add some clarifying points:

1. The major thrust of what i'm proposing is that utf8 is merely an internal representation for strings. it's not a separate concept.
2. the current string API that we have today operates with the exact same semantics it has today. i.e. this[int] means exactly the same even with the internal representation being utf8.
   
   
   
   1. Note: "semantics" here doesn't include runtime performance.
   
   
   2. this doesn't include unsafe.
   
   
   3. this doesn't include code which can peek into the internal representation (i.e. ROS).
   
   

As such, things like 'char' still remain a 16bit concept.

Along with teh above, I'm 100% ok with corefx layering on additional apis onto System.String (or silbling classes). For example, having the ability to iterate a string by char, codepoint, unicode-scaler, grapheme-cluster, etc. makes a ton of sense. However, those APIs could be provided transparently on top of System.Strings witha utf16 or utf8 internal representation.

The idea here is to keep System.String so nearly identical to what we have today that we can transparently move lots of code over to using it, and have a low cost way to migrate code that can't immediateley move over.

[GrabYourPitchforks]

Thanks @CyrusNajmabadi for your thoughtful feedback! It'll take me some time to digest all of it. But for now let me respond to two points:

You had mentioned we should detect fixed (char* ptr = str) and throw if UTF-8 strings are enabled. As far as I can tell this is not viable, as the runtime can't reliably detect this pattern. I am not intimately familiar with the details of why this is not feasible, but I am going on what folks knowledgeable about the runtime have relayed to me. This means that instead of seeing an exception, the application could exhibit incorrect behavior, including data corruption or buffer overruns. These problems might not manifest themselves in testing and might only occur in production scenarios.

You had mentioned that "semantics" should not include runtime performance. One of the scenarios that I had envisioned for Utf8String was a guarantee on the algorithmic complexity of each API, which would allow the type to be utilized and reasoned about in perf-sensitive scenarios. I imagine there exist applications which make assumptions about the algorithmic complexity of various APIs on string as well. It's entirely possible that I'm concerning myself too much with this and that I should deprioritize it - and your comments are excellent food for thought on this topic. But I wanted to at least provide some background.

Separately, there have been proposals on and off over an AnyString type, which is an abstraction over string data where the encoding isn't exposed via the public API surface. This approximates how Swift's string type works, though nowadays they use UTF-8 under the covers the majority of the time. I'm not keen on introducing yet another type to the mix, but I wanted to throw it out there as an example of how other ecosystems have handled the issue.

[GrabYourPitchforks]

There are also practical concerns around the ability to round-trip data if we use UTF-8 as the backing mechanism for string instances. Consider the following:

string original = "😀"; // = U+1F600 GRINNING FACE (utf-16: [ D83D DE00 ], utf-8: [ F0 9F 98 80 ])

string a = "\ud83d"; // not representable in utf-8
string b = "\ude00"; // not representable in utf-8
string c = a + b;

Console.WriteLine(original == c); // what does this print?

One option would be to use WTF-8 ("wobbly") instead of UTF-8, which would allow round-tripping this data. It would make string splitting / concatenation a little more difficult since it would involve fixing up data at the boundary. It would also mean that we're not actually using UTF-8 under the covers, which means that we can't internally convert the string to a ROS<byte> and blast it to wherever we want it to go. We'd still need to scan the data first, fixing up malformed subsequences. So this still gets the memory savings, but it limits where the runtime can truly take advantage of the fact that the underlying representation is UTF-8.

Ninja edit! WTF-8 is described at https://simonsapin.github.io/wtf-8/.

[CyrusNajmabadi]

You had mentioned we should detect fixed (char* ptr = str) and throw if UTF-8 strings are enabled. As far as I can tell this is not viable, as the runtime can't reliably detect this pattern

hrmm... that seems odd. can this be clarified? Isn't it calling, for example: ref char String.GetPinnableReference() (or RuntimeHelpers.OffsetToStringData)? That seems like something that could definitely be detected and blocked right?

[CyrusNajmabadi]

You had mentioned that "semantics" should not include runtime performance. One of the scenarios that I had envisioned for Utf8String was a guarantee on the algorithmic complexity of each API, which would allow the type to be utilized and reasoned about in perf-sensitive scenarios.

Note: i believe that would still be true under my proposal for most APIs. For example, if you called any of the new utf8-aware APIs they would certainly be able to give accurate perf characteristics.

Where i am saying that "runtime performance" wouldn't be guaranteed would be around a few of the existing utf16-index-based apis we have today. i.e. char this[int i]. That would stay constant on utf16-clr and would have some amount of non-exact perf characteristics for utf8-clr. However, new apis exposed would certainly be able to be provide more strict bounds since htey would know they could have a utf8 backed impl.

[CyrusNajmabadi]

These problems might not manifest themselves in testing and might only occur in production scenarios.

My recommendation would be to strongly advise a high amount of testing for people, and to move gently over to utf8-clr, with a willingness to roll back to utf16-clr if they do happen to run into issues. Unsafe code is certainly a dangerous area, and the more code and libraries that you depend on that have unsafe, the more time it might take to move over.

Other options to mitigate this might be to require a more stringent 'opt in' attribute from libraries, with warnings (or potentially errors) about using utf8-clr with libraries where the author didn't at least annotate with the attribute saying "i am really ready to be used in either mode".

The goal here though would be to allow many pieces of code to move over quickly and to raise aware (ideally with errors/mesages/etc) about pieces of code that can't so that those (hopefully) mostly isolated places can be fixed up in a reasonable timeframe.

[CyrusNajmabadi]

Separately, there have been proposals on and off over an AnyString type, which is an abstraction over string data where the encoding isn't exposed via the public API surface. This approximates how Swift's string type works, though nowadays they use UTF-8 under the covers the majority of the time. I'm not keen on introducing yet another type to the mix, but I wanted to throw it out there as an example of how other ecosystems have handled the issue.

Definitely. And there are things like how Java just has the their single string type, exposed as a 16bit-char construct, but internally the representation can be 8bit-fixed for ascii. There are all sorts of pros/cons for lots of these systems. Where you guys end up depends on what value you find to the different aspects of all of these.

For me personally, i weight the following extremely highly:

1. i want only one string type that is effectively used everywhere.
2. i want the ecosystem as a whole to be able to move in some amount of reasonable time.
3. i want that string type to sane semantics around it. i.e. one that properly can allow me to work at the rune/grapheme level sanely.

This precludes (for me) any approach that introduces a new string-like type. That's because those approaches give me '3', but basically fail on '1' and '2'. The type will not be used everywhere, and there doesn't seem to be any way to get the ecosystem to actually be able to move over reasonably at all.

The proposal i outline certainly haw downsides. It incurs cost on certain (hopefully small) areas of code out that if it wants to be able to work on a new utf8-clr. It may also have the potential to introduce negative runtime issues (perf/stability/etc.) even with lots of work done to avoid that. However, to me, that's a totally ok price to pay for the benefits if it can mean that the ecosystem will get a single good string type and the cost to move forward onto it is amortized low overall.

[CyrusNajmabadi]

There are also practical concerns around the ability to round-trip data if we use UTF-8 as the backing mechanism for string instances.

Curses. :)

However, i would personally try to avoid having the perfect be the enemy of the good. There look to be some hairy corner cases that might arise. But maybe those can be rare enough to hack in some solutions without having to throw the baby out with the bathwater.

It would also mean that we're not actually using UTF-8 under the covers, which means that we can't internally convert the string to a ROS and blast it to wherever we want it to go.

Or, alternatively, you can take approaches like the following:

1. allow strings to use certain of their header bits to indicate what the internal representation is.
2. you need ot scan strings anyways even with your proposal to ensure they're utf8 right? So we apply that technique to compute bits like: all these bytes are ascii, all these bytes are utf, all these bytes are wtf8, and potentially even these are just bytes.
3. with the header bits indicating what's up, you can optimize your runtime code accordingly, right? For example, if it is ascii/utf8 (likely the 99.9% case), you get the full "blasting" perf. But you also have your runtime fallback for the code that says "whoops, a wobbly string snuck in. write out things in some sensible manner".**

In effect, don't take the rare corner case and make it kill everything else. Optimize for what is certainly the real common cases in practice (ascii, and utf8-sane) and design for that. in practice that means nearly everyone will get nearly all the benefit nearly all of hte time. And the rare cases that don't can be addressed as people discover the issues in particular places and stamp them out.

** Note: this approach seems similar to how the runtime will stack-alloc small things, but has to have a fallback for when that might be too much for the stack. When i've spelunked into coreclr/fx i've definitely seen all sorts of code written to optimize the common fast path, with fallback for the uncommon-slow path as necessary. While this is certainly more burden on you guys as the owners of likely the largest amount of string-processing code out there, don't you think that burden is justified in order to get the savings and ease for the rest of the ecosystem?

Thoughts @GrabYourPitchforks ?

[GrabYourPitchforks]

Making the OffsetToStringData property throw is an interesting proposal. We looked at that property's callers when investigating whether strings could dynamically be backed by UTF-16 or ASCII, but it didn't occur to me to have the property getter simply throw. I'll shop around the idea of whether folks are comfortable having the fixed statement or implicit conversions from string to ROS<char> throw.

[CyrusNajmabadi]

I'll shop around the idea of whether folks are comfortable having the fixed statement or implicit conversions from string to ROS throw.

Great! Note that i think we can also go above and beyond that. For example, analyzers that find this sort of thing and link to appropriate docs or even potentially help fixup code (if we can determine common patterns and provide useful transformations).

This would be part of a "getting your code utf8 ready" package that Core could release ahead of time to help people prepare and get their code ready for a point where this CLR switch was available. The main idea would be to both help isolate code that is preventing people from moving forward, but also then help take those pieces of code and have them move forward to being internally ut8/16 clean to quickly unblock their dependencies.

[GrabYourPitchforks]

Below are some initial reactions and strawmanning from the global runtime switch suggestion. I'm still shopping this around, so this isn't intended to be an exhaustive summary.

Imagine three modes of operation for the runtime: UTF-16 (as it is today), UTF-8 aware only, and UTF-8 relaxed. The first of these three options is the world today, so I won't discuss that. The other two are detailed below.

__UTF-8 aware only__: The .csproj can be marked with "I'm aware of UTF-8 strings", and a few things will happen at compile time. First, any APIs which rely on string being UTF-16 would be compile-time errors. So pinning a string would generate a compile-time error, calling AsSpan() would be a compile-time error, and so on. In addition, we could _warn_ on some APIs like string indexing, as these APIs would continue to work but might have a very different performance profile if UTF-8 strings are enabled by the runtime.

The SDK would also expose alternatives to the methods that are no longer available. For example, you could imagine bool string.TryGetUtf16View(out Utf16StringView), and that returned object would allow things like pinning or a conversion to ROS<char>. The error messages could instruct the developers to call this instead to get compilation to succeed.

Additionally, the compiler would annotate the assembly with an attribute saying "I am UTF-8 string aware." When operating in _UTF-8 aware only_ mode, the runtime would forbid loading any assemblies which have not been annotated in such a manner. The advantage of this is that once loaded, your assembly behaves properly and shouldn't encounter any runtime exceptions. The downside is that quite literally every DLL in your application would need to be recompiled if you intend on running the application in this mode.

__UTF-8 relaxed__: This mode of operation is closest to @CyrusNajmabadi's proposal, which tells the runtime to flip the string backing store to UTF-8, but it does not place the restriction on which assemblies can be loaded into the process. The advantage is that you don't need to recompile the application and the entire object graph. The downside is as mentioned earlier: you could get unexpected failures at runtime, leading to a game of whack-a-mole when trying to run a real-world application in this mode.

It was also brought up that even if we have a global switch, it might still make sense to keep a standalone Utf8String type. For client applications which make heavy use of UI, it is natural to keep UTF-16 as the standard string backing store. But if you also need high-performance networking in your application, the parts of your app which touch the networking stack would use Utf8String (not string) as an exchange type amongst themselves.

[alrz]

UTF-8 relaxed

That would be a nice feature for trying out things without much inconvenience, but never safe for production - especially if the "strict" mode is not an option.

There could be a third variation derived from relaxed mode in which we store ascii bytes depending on the string content (similar to java), since in this mode we're going to scan strings for utf8 validity anyways, we can decide the storage strategy at runtime and possibly fallback to utf16, instead of throwing.

[YohDeadfall]

For client applications which make heavy use of UI, it is natural to keep UTF-16 as the standard string backing store.

Makes sense for Windows only. I haven't been working with UI for a long time, so am not constantly tracking changes in that area, but there are movements in making a cross platform framework. There was a discussion in WPF repo asking about making WPF cross platform, and attendees pointed out at Uno and if I remember correctly about others.

The idea of .NET Core is to run on any machine and any platform. Since version 3.0 it includes WinForms and WPF which are Windows only, but as I said a demand in platform independent UI exists, so there is a chance that something will born in the .NET Foundation (or acquired and included) and this thing won't use UTF-16.

[CyrusNajmabadi]

Could you clarify this bit @GrabYourPitchforks

For client applications which make heavy use of UI, it is natural to keep UTF-16

Why is UTF-16 natural for UI? Thanks!

[GrabYourPitchforks]

Why is UTF-16 natural for UI? Thanks!

Localization / globalization APIs and text processing functions (such as those exposed by NLS and ICU) tend to work best with UTF-16 since that's what their underlying tables are encoded in. Such API calls tend to be much more prominent in UI-based applications than in server-side applications.

[CyrusNajmabadi]

Interesting! Is that windows specific? Or do other platforms also do things that way? How do they do it efficiently if they're a primarily utf8 language/platform?

[GrabYourPitchforks]

Interesting! Is that windows specific? Or do other platforms also do things that way? How do they do it efficiently if they're a primarily utf8 language/platform?

Most platforms I'm aware of internally transcode the data to UTF-16 before doing localization / globalization-aware text processing, then they transcode the data back to UTF-8 on the way out. Some platforms provide UTF-8 specific APIs to help with "simple" operations like whitespace trimming or simple case conversion, but they don't encompass the full breadth of text processing APIs most devs would expect from a library. See http://userguide.icu-project.org/strings/utf-8 for some further details on this.

[CyrusNajmabadi]

Wouldn't the section "UTF-8 as Default Charset" in your link cover my proposal?

[GrabYourPitchforks]

Wouldn't the section "UTF-8 as Default Charset" in your link cover my proposal?

See https://sourceforge.net/p/icu/mailman/message/32031609/ for more information on this flag. It's used more for interop purposes (filenames, etc.) than it is for user-displayed text processing. See the bottom heading of http://site.icu-project.org/design/collation/v2/perf for perf characteristics of UTF-8 vs. UTF-16 text processing.

[CyrusNajmabadi]

Interesting. Doesn't seem worth it to me to have the 16bit representation at twice the cost, but i suppose thats' something up to the end developer. With the approach i'm advocating, it would be up to the final place where the app is run to decide if you wanted utf16 or utf8. I would be surprised if the perf benefit would be sufficient to keep a UI app wanting to stay on utf16. But, if it was that beneficial for them, they could certainly stay.

Thanks for the info!

[YohDeadfall]

@GrabYourPitchforks Client applications mostly don't require fast string processing since performance is limited by the user (: So filenames are not a case about which should care, but there are other stuff I know about (Windows related only):

• UI Automation,
• text processing (search, navigation).

The first one requires updated only on UI changes (visual tree changes in case of WPF), so it doesn't happen very often and automation tree is much less than UI tree.

The last one is related to text processing and searches in UI for a specific text (for an instance to find a matching element in a combo on in a list. In case of large data in which the uses wants to find something async processing makes sense to prevent freezes.

[webczat]

hmm I was wondering if utf-8 isn't actually potentially slower? For example I don't know how long my char type needs to be in advance, etc. In case of UTF-16 strings I believe that you have only two cases, either two chars or one char. If you assume you are unlikely to have surogate pairs in your string, it is even faster and does not require scanning the string even once to see how many code points you have there. So I always read it as compromise between UTF-8 where you cannot determine the size of char without reading it, and UTF-32 where char would be just too large. And found that useful thing. Of course if string is immutable you don't need to calculate this again and again each time, except if you iterate through the string then how much you move probably depends on context, no?

[airbreather]

hmm I was wondering if utf-8 isn't actually potentially slower?

@webczat It's "potentially" slower, yes. There are code points that are 3 bytes in UTF-8 and 2 bytes in UTF-16 (to my knowledge, this is mostly CJK). More bytes = more data to go through = slower.

And, of course, if you're interacting with an external component / system that already uses UTF-16 encoded strings, then of course it's going to be slower to translate them to UTF-8 (and back again, if needed) :grin:.

I'm curious about the use cases that you bring up? What problem are you trying to solve that requires you to know "how many code points you have there"?

e.g., if you're developing something like a text editor that needs to keep track of what column the caret is at, then counting code points it not enough: some code points are combining marks that modify the meaning of code points around them.

e.g., if you're trying to see if some text will fit in a Tweet for the Twitter API, then the correct answer depends on whether or not the text is normalized (NFC), so you'd have to scan the whole string anyway (unless you already happen to know it's already in NFC any only contains code points on the BMP, in which case yes, UTF-16 has a slight advantage).

[webczat]

hmmmm... twitter's notion of length of tweets? 140 characters? currently more I believe. what is a character? code point... most likely :)

[webczat]

and again how large is the unicode character if in utf8? we don't know. either the max possible nr of bytes or... for utf16 you either have a pair or a single code unit per char.hmmmm... twitter's notion of length of tweets? 140 characters? currently more I believe. what is a character? code point... most likely :)

[GrabYourPitchforks]

Twitter is a bit of a special case since they use a weighting system, so what they consider a "character" doesn't necessarily match what other programming languages (or even Unicode itself) consider a "character". You can read more about their design here, here, and here. There are some reports that the Twitter web service implementation doesn't necessarily match the library they make available on GitHub, but it should be enough to convey the general gist.

We're working on some .NET docs to better explain concepts like UTF-*, code points, scalar values, text lengths, and how they relate to more common jargon like "characters". The work item tracking that is at https://github.com/dotnet/docs/issues/15845. There's also a link from there to a rough draft of the document.

[webczat]

well. :) I didn't know most of that, thanks for explanation

[blankensteiner]

Will Utf8String make it into .NET 5?

[GrabYourPitchforks]

@blankensteiner It's still in an experimental phase. But it _is_ present in the .NET 5 nightlies. See https://github.com/dotnet/core-sdk#installers-and-binaries for instructions on downloading the latest nightlies. You'll have to pull in the package __System.Utf8String.Experimental__ to see the APIs. Pay special attention to the package source configuration listed at the _Installers and Binaries_ section of the page I just linked to, as it'll tell you how to configure your local NuGet client to see the nightly packages.

[tannergooding]

I really don't like that Utf8String is taking such a drastically different approach from String.
I do understand that a lot of the concepts can be hard/confusing and Utf8String makes some things more glaringly obvious (like the individual elements not representing a full unicode character).
However, I don't think making Utf8String drastically different is the right approach. It will hinder some existing scenarios you can use System.String for while not fixing the issue that remains on System.String.

Today, there is nothing that requires a String or even an individual Char to be "valid". "\uFFFF" and '\uFFFF' are both completely valid standalone values.
I am free to split a string at any point and provided I account for the split when processing the data, everything is fine. It is only a problem if you split the string at an arbitrary point and do not account for the split. This is because you may have to look at surrounding characters to understand the full context (including surrogates, certain uppercase/lowercase characters, puncation, etc).
I am free to use string + span to process any kind of data, and its only when I start dealing with something like System.Text.Encoding does actual validation and conversions start happening (such as invalid sequences becoming U+FFFD).

Because of all of this, string is very flexible and you can use it easily as a barebones primitive (basically treating it as an arbitrary char[]) to do my own implementations/logic. It realistically has no overhead, it has language support, and I can mold it to fit my needs. If I want to do proper unicode processing, I have separate APIs that deal with that. This includes things like the API overloads on char that take a string s, int index and account for surrogates or the new Rune APIs that do similar.

With the new proposed API, it is trying to track validation, doesn't provide indexing or even a strongly typed char8 primitive, and removes much of the flexibility that exists on string.
While I feel this will make some workloads faster (you can assume data is valid) and it will help some users handle processing of the data, I feel it will likely hurt users who were wanting a lower level primitive similar to string.
It also won't solve the problems that exist on string and where users will continue hitting them moving forward (partially because they are less obvious on string due to more valid characters being representable).

I would propose that instead we expose something more akin to the original proposal: https://github.com/dotnet/runtime/issues/933
It would essentially mirror the existing string API surface but over char8[] rather than char[]. Likewise, there would be a strongly typed char8 type that can be used for overload purposes and that would expose validation and helper methods much like char does today (as compared to short which it could have used instead).

We should then encourage and broaden guidance on using the Rune APIs for enumeration and valid handling of textual and linguistic data. This would ensure that users can write the same code for both string and utf8string inputs (only dealing in concepts of rune) when they are wanting to deal with known valid data. We could also expose a helper type that tracks validity or a set of additional APIs that assume valid inputs for the cases where that helps increase performance.
I believe this would make a more consistent world which ensures users can and know how to correctly handle text data while still allowing the new types to be used in a lower level mechanism (such as interop or non-server scenarios) if and when desired. It also ensures that the new types can be used for overload resolution and differentiation when that is important and that the language can easily expose literals for them in the future.

[LokiMidgard]

I'll would actually prefer the current approach over a more string like.

I especially dislike char8 it has the same problem that char also have. It suggests that the type represents a character which it does not. Sometimes it works but sometimes two or more char8 will represent one character.

Having a length and an indexer will let people use for loops to iterate over the utf8string and using the Length property to find out how many characters there are in the utf8string.

Using library's today that work with string often do not account for surrogate characters. If you put one in, there is a chance of breaking things. And I think the problem is not that those people writing those library's don't know what they do, but the API design of string leading them there. Today, there is nothing that requires a String or even an individual Char to be "valid". And that's the problem. You have a string, which many people regard as (UTF-16 encoded) text that is not valid text. You have a char that is not a character. You could say a string is a sequence of data that is in most cases text, but sometimes it is not.

I also think the current design is still flexible and low level enough. You can get a span of bytes from the string. This has length, can be indexed and you can split it where you wan't its the same you can do today with string I think. Correct my if I'm wrong but the only members missing are the indexer, index based operations and the Length property. And those are the members that will break surrogates.

To be honest what I want is a Framework type that represents text and I can get somehow information about the number of characters (Run's not char's) and string operations like replace, trim, etc.... The underling encoding doesn't matter as long as I can read and write the text to those encoding's when needed.

[LokiMidgard]

Do I understand it correctly that Range is working on bytes?

And that when I iterate over Runes I can sum the length of each Rune (assuming it has such a Property) to calculate the "index" of my current position?

```c#
public static Utf8String GetBracesWithContent(Utf8String txt, Rune openingBrace, Rune closingBrace braces){
int? opening = null;
int length =0;
foreach(var rune in txt.Runes){
if(rune == openingBrace && !opening.HasValue){
opening = length;
}
else if( rune == closingBrace && opening.HasValue){
return txt[new Range(opening.Value, length + rune.Length)];
}
length += rune.Length;
}
}

The idea is I think that not using an int index should prevent people from splitting Utf8String where this would not be valid. However writing `u"💣Hello World"[1..]` people could still think they slice `Rune`s with this method, as an convenient alternative to `SubString()`.

Maybe an custom "Range" Type could prevent this.

```c#
public readonly struct Utf8Index {
    public int ByteIndex {get;}
    public int RuneIndex {get;}
    public int RuneLength {get;}
}

This type could also have information about the position of the Run in the text, not only the byte position. Assume you need to find an opening and closing brace that has at least 3 Runes between it. Having the byte position is not enough. You would need to iterate the Runes between those braces again, to count them.

[tannergooding]

I especially dislike char8 it has the same problem that char also have. It suggests that the type represents a character which it does not. Sometimes it works but sometimes two or more char8 will represent one character.

That is most of my point. Most languages use the term char for character and it matches the https://www.unicode.org/glossary/#character definition of it:

(1) The smallest component of written language that has semantic value; refers to the abstract meaning and/or shape, rather than a specific shape (see also glyph), though in code tables some form of visual representation is essential for the reader’s understanding. (2) Synonym for abstract character. (3) The basic unit of encoding for the Unicode character encoding. (4) The English name for the ideographic written elements of Chinese origin

Unicode, at the lowest level, works on these basic units. This makes it powerful, extensible, and gives you full access to the underlying semantics because it is a primitive. String models this as does the existing Char type. It is then my opinion that Utf8String should match that for consistency, flexibility, and to allow the most advanced scenarios where necessary (it will be another "primitive").

There is then a higher concept of Code Point (https://www.unicode.org/glossary/#code_point) and Unicode Scalar Value (https://www.unicode.org/glossary/#unicode_scalar_value). The latter we model using Rune and is what most people should be operating on as it removes the need to consider surrogates. It doesn't however remove the need to consider surrounding characters or other bits of linguistic data which needs to be handled separately.

People working with string already need to consider the fact that characters may need to be combined or that is could be invalid. If they don't want to handle that themselves, they need to use Rune and the corresponding enumeration APIs which helps do that for them.

I would want the way users are told to interact with text data of any kind (string or utf8string, but also any other future unicode types) to be consistent.

• We shouldn't have one model for UTF-16 data and another model for UTF-8 data.
• We shouldn't restrict what utf8string data can represent as compared to string data
• We shouldn't have a validation mechanism that works for utf8string without working for string as well

Doing otherwise bifurcates the ecosystem and makes it so I do x when working with string but y when working with utf8string. It also means that optimizations or changes made for one may not become applicable or available for the other.

[GrabYourPitchforks]

Thanks for the recent discussion! These are all very good questions.

__Re: having different models for UTF-16 and UTF-8__, I agree that consistency would be ideal. And the API surface is largely the same across both types where it makes sense. The only places where differences are exposed is where either: (a) I've received feedback that the string concepts don't cleanly translate to Utf8String and that it confused the callers; or (b) it's a known pit of failure in string that I'm trying not to propagate.

As an example of these, consider the string indexer. The recommendation is to use Rune or a similar API instead of looking at individual char elements, but the simple fact is that the overwhelming majority of applications aren't exposed to surrogate code points. Their users aren't feeding them emoji or characters from lesser-used languages. These applications "just work" for the most part because processing these characters isn't really within their use case.

However, if these applications convert to Utf8String, that flips on its head. Even for applications operating primarily on English text, it's not unreasonable for smart quotes (“”) or accented characters (résumé) to make their way into the system. Now, suddenly, indexing by byte (or char8) gives incorrect data, where previously indexing by char gave correct data for the use cases they cared about.

This is such a fundamental difference between the two types that the feedback I received convinced me that the best course of action was to remove the indexer entirely. By removing the indexer, we force the caller to consider the operation they were trying to perform. Otherwise we'd run the risk of them using the same patterns that have worked in the past, now left wondering why they're receiving random production failures that never occurred during testing.

__Re: validating Utf8String but not string__, this was done for two reasons. First, more recent runtimes like Rust and Swift have proven that validating input in this manner is largely viable. It also mostly matches what .NET already does today under the covers, even if you don't see it.

When data comes in from i/o, it comes in as a series of bytes. It then needs to be _transcoded_ (often via Encoding.UTF8.GetChars or Encoding.UTF8.GetString) into a string or a char[] or a Span<char>. This conversion process necessarily validates data for well-formedness while changing its elemental representation.

It's actually exceedingly difficult in .NET to create a malformed string instance. (Here, I'm using "malformed" to mean "invalid UTF-16".) Either you started out with a malformed literal like "\ud800" in your sources (why?), or your transcoding routine produced ill-formed output (arguably a bug in the transcoder), or the caller incorrectly substringed a string instance in the middle of a surrogate pair.

That last scenario is arguably a bug in the application code since any string API they call on the newly malformed instance will likely result in garbage output. The only thing they could really do with any reliability is to treat the string instance as an opaque buffer, at which point we may as well drop the whole string ceremony and use raw arrays like char[]. In fact, if you look at existing APIs like Encoder.Convert which are designed to operate on opaque buffers of arbitrarily split text data, they largely use char[] instead of string.

By giving Utf8String a formal valid-by-construction constraint, we're codifying the rules that developers were already unknowingly playing by all along. Since Utf8String can be projected as a ReadOnlySpan<byte> or a ReadOnlyMemory<byte>, anybody who needs arbitrary slicing capability can perform that projection and call those slicing routines. Neither ReadOnlySpan<byte>.Slice nor ReadOnlyMemory<byte>.Slice is opinionated in how slicing should be performed beyond ensuring the call doesn't go out of bounds.

[tannergooding]

Now, suddenly, indexing by byte (or char8) gives incorrect data, where previously indexing by char gave correct data for the use cases they cared about.

Right. utf8string, due to using 8-bits to represent the base data (rather than 16-bits) can represent a lot less state and so you are more likely to encounter cases where you need to combining multiple characters to produce a usable unicode scalar.
I think increasing visibility around this and pushing users towards writing correct code by default by utilizing the Rune APIs is likely the correct thing here. It works for both utf8string and string without having to introduce new concepts and just increases visibility on an issue that people already have (even if it is something they aren't actively hitting). As we become a more connected world and things like emoji become more common place in everyday text, I imagine this is something that will become increasingly more problematic for users, even with string and so it isn't something I think we should only handle on utf8string but more generally.

When data comes in from i/o, it comes in as a series of bytes. It then needs to be transcoded (often via Encoding.UTF8.GetChars or Encoding.UTF8.GetString) into a string or a char[] or a Span. This conversion process necessarily validates data for well-formedness while changing its elemental representation.

Today this happens for anything using the System.Text APIs and with any API that works with an encoding. This is most, but not all of the framework, and includes things like System.IO or other APIs in System.Text, but does not include things like the APIs on System.String which take a char* or char[] or many of the marshaling APIs (PtrToStringUni for example).

Having this validation is great and helps ensure that pure .NET code is correct by default and helps avoid a lot of issues.

However, once you start doing interop with other languages, libraries, frameworks, or applications and you aren't doing communication over streams or the web, you will likely be working with APIs that take char[], char*, or Span<char> (or byte[], byte*, and Span<byte> for UTF8/ASCII data) and unless you are using an encoding API or needing to transcode (from UTF8 to UTF16 or vice-versa), you won't get any of the validation or fixups for invalid surrogates, etc. You might also not want any of that because it may be problematic or break some expectation for whatever you are communicating with. But, given that this is an object oriented environment, you likely want many of the niceties that come with language and framework integration.

That is, you likely want to be able to declare const Utf8String ColorSemnaticName = "COLOR"; (because doing static ReadOnlySpan<sbyte> ColorSemanticName => new sbyte[] { 0x43, 0x4F, 0x4C, 0x4F, 0x52, 0x00 }; isn't very readable or maintainable).
You likely want to be able to provide friendly overloads that take ReadOnlySpan<char> or string (and equivalents for Utf8String) so that users don't have to manually pin or convert their inputs.
You likely also want to have a strongly typed character both to avoid overload resolution problems but also to avoid confusion and clearly indicate what type of data you expect here. Is ReadOnlySpan<byte> a binary buffer, an ASCII buffer, a UTF8 buffer, something else? What happens if you have one overload that does take binary data and another overload that takes a string or some combination of both?

I think a lot of the changes here are good, but I think they aren't taking into consideration the same scenarios that exist for System.String or the cases where users do want and need to do lower level interop/communication and where they want or need a type that is more like the existing System.String.

[pentp]

In general, you've convinced me that the new safer concept is good, but I'd like to point out a very common scenario that's not possible with these safe APIs: searching for a char/string (usually ASCII) and then searching for another char/string after that position (often the same char/string) and then taking a slice between those. It's possible to get the first one with TryFind/TryFindLast, but there's no API to get the next one from a specified position.

Conversely, the SplitOn/SplitOnLast convenience functions are likely going to cause the same efficiency issues that string.Split causes today - very often only one slice is needed and others are just wasted allocations then.

[Joe4evr]

@pentp I don't see how it's not possible to make your scenario work:

// I assume target-typing a string literal "Just Works"™
Utf8String longString = "Lorem ipsum dolor sit amet, consectetur adipiscing elit";
ReadOnlySpan<Char8> span = longString.AsSpan();
Char8 needle = 'i';

span = span.Slice(span.IndexOf(needle) + 1); //span = "psum dolor sit amet, consectetur adipiscing elit"
span = span.Slice(0, span.IndexOf(needle));  //span = "psum dolor s"

[LokiMidgard]

@Joe4evr I think, AsSpan() is currently not part of the proposed API. Instead there is AsBytes() which returns a ReadonlySpan<byte>, but it would work almost the same I think.

@pentp I'm not confident on how much I understand on performance and stack allocation yet, but what I've seen so far it is much more preferable in most cases. The SplitOn method returns a SplitResult which is a readonly ref struct and this can be deconstructed to two Utf8Span which is
also a readonly ref struct so there should be no (heap) allocation at all, and on the stack only a view bytes are needed and no string memory is copied. So I don't think this method has the same performance footprint as the string.Split() method.

I assume you could also use the TryFind method and calculate the position of the next rune by using the start of the returned Range and its length. Then slice the original string from that position and take the first Rune/Char using the corresponding enumerator and look at the first element.

```c#
Utf8String longString = "Lorem ipsum dolor sit amet, consectetur adipiscing elit";
Rune needle = 'i';

if(!longString.TryFind(needle, out var range))
throw;
var (offset, length) = range.GetOffsetAndLength(longString.Length);
// create range from next char to end
var start = Range.StartAt(offset + length);

var rest = longString[start];
var enumerator = rest.Chars; // or runes
if(!enumerator.MoveNext())
throw;
var thisIsNextChar = enumerator.Current;
```

[GrabYourPitchforks]

There were samples in the issue description that show what a simple split would look like:

Utf8String utf8 = "Hello world!";
(Utf8Span before, Utf8Span after) = utf8.AsSpan().Split(' ');
Debug.Assert(before == u"Hello");
Debug.Assert(after == u"world");

See also the unit tests at https://github.com/dotnet/runtime/blob/81acc61da028deb65379bafb8f0762fb43f7de91/src/libraries/System.Utf8String.Experimental/tests/System/Utf8SpanTests.Manipulation.cs, which demonstrate further usage patterns.

[pentp]

OK, it looks like the actual code does support this use case efficiently by using Utf8Span. I was confused because the proposed API documentation seems to be way out of date.

[Serentty]

@tannergooding

That is most of my point. Most languages use the term char for character and it matches the https://www.unicode.org/glossary/#character definition of it:
Unicode, at the lowest level, works on these basic units. This makes it powerful, extensible, and gives you full access to the underlying semantics because it is a primitive. String models this as does the existing Char type.
There is then a higher concept of Code Point

Code points are the smallest meaningful unit of information in Unicode. Everything smaller is encoding-specific and just a matter of bit patterns and representation. If anything is a “character” in Unicode, it's a code point, not a UTF-16 code unit, which is a concept that doesn't even exist in UTF-8 and UTF-32. The existing string and char types do _not_ model the concept of a character as you quoted in a meaningful way. I agree that a name like char8 would be horribly misleading, but no less so than the current situation with char, whose name unfortunately could never realistically be changed.

Overall, I disagree with the assessment that char in .NET as it is now is any more “character-like” than a single byte of UTF-8 is. The only character-like thing about it is that as a mere heuristic, common characters often take a single 16-bit char, but even that is less and less true by the year.

@GrabYourPitchforks

This is such a fundamental difference between the two types that the feedback I received convinced me that the best course of action was to remove the indexer entirely.

I agree quite strongly with this approach. Rust doesn't have a _default_ indexing operation on strings, and makes you specify if you mean bytes, code points (called char in Rust), or (with additional library support) grapheme clusters. This sounded like a radical and annoying solution to me at first, but with heavy use I've come to find it quite practical and useful. A small learning hump can be very beneficial if it saves you a lot of effort down the line, as compared to an easier solution that “mostly works”.

[Serentty]

I really like the idea of having a keyword for this. The full name Utf8String is really not nice to type, especially when it has a number in it.

I also like the idea of overloading == for comparison with the normal string class, even though it's a bit costly in terms of performance. Then again, I would probably want to be able to have this implicitly convert to a normal string, which is probably not a good idea given the hidden allocation, so maybe my opinion on this isn't well thought-out yet. I think at least equality should be okay though.

[PathogenDavid]

@GrabYourPitchforks

It's still in an experimental phase.

Is there an overview somewhere of what's needed to get Utf8String out of the experimental phase? From my perspective it has seemed like Utf8String has been just around the corner for quite a while, but moving it to the runtimelab repo indicates otherwise.

[Serentty]

I'm also curious if certain features from Utf8String could be backported to string, especially the indexible properties. The biggest question would be whether trying to index in a way that doesn't make sense (but is still within the bounds of the string) should yield a replacement character, or throw an exception. Rust panics when you do this, which is a lot more extreme than an exception, so I think an exception could be reasonable if you try to—for example—index by runes into a string with unpaired surrogates before the rune you're fetching.

[GrabYourPitchforks]

@PathogenDavid The big blocker is waiting for feedback. We need folks to use this as an exchange type and help us validate the design. The nightly builds of .NET include the Utf8String type, and there's also a netstandard2.0 package available.

install-package System.Utf8String.Experimental -prerelease -source https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5/nuget/v3/index.json

(That's the normal package feed for all of our .NET 5 wave packages, experimental and stable.)

@Serentty We added some APIs to help with this. See Rune.GetRuneAt, which throws if the string instance contains malformed UTF-16 data. There's a non-throwing Try* version available as well. For simple enumeration scenarios, there's string.EnumerateRunes.

[Symbai]

And I'm waiting for it to be implemented before I will give any feedback. I'm really looking forward to it but you won't get the big feedback if you keep it hidden like that, forcing people to look out how to add it to their projects, which some of them won't do because its still experimental, which can also mean that it never gets greenlighted at all. So for productive projects this is a no-go.

If you really still feel insecure about making it "live", you better should blog post and make it a big notice everywhere you can in order to get the feedback. Otherwise it remains as it is just like now.. some folks are waiting for it and you are waiting for feedback and we never reach the same goal.

[PathogenDavid]

@PathogenDavid The big blocker is waiting for feedback. We need folks to use this as an exchange type and help us validate the design. The nightly builds of .NET include the Utf8String type, and there's also a netstandard2.0 package available.

I'm brave enough for that.

Just to clarify: That NuGet package still requires a nightly .NET SDK to be installed, right? (That's my understanding of https://github.com/dotnet/runtime/issues/41521 and a quick test with package 5.0.0-preview.8.20352.3 and SDK RC2 implies that still holds true.)

I also ask because ideally I'd prefer to have projects using the experiment be self-contained if at all possible. (IE: building csproj automatically does whatever is needed to acquire the Utf8String-enabled runtime and build a self-contained app.) My understanding of dogfooding nightly is that's not a supported scenario though. (I can deal with this if necessary, I just want to make sure I'm not making things unnecessarily difficult.)

[GrabYourPitchforks]

@PathogenDavid That's an unresolved bug in the package. The intent is that it should work regardless of runtime, but at the moment it requires a nightly. I believe we tested consuming it from a netstandard2.0 library _as long as you're on a nightly runtime_ and things passed.

[PathogenDavid]

Got it. Installed the latest .NET 6.0 nightly and it works, thanks!

[mconnew]

@GrabYourPitchforks, adding support to the serializers should probably be done before this is released in a .NET version. Strings are handled as a primitive type and we have regular C# code implementations for primitive serialization in DCS and XmlSerializer. It might need a small modification in the ref emit code to emit the equivalent of || type == typeof(Utf8String) but I suspect that's not the case. If it is, it wouldn't be a big change.
Edit: Also need support in the XmlReader, XmlWriter, XmlDictionaryReader, XmlDictionaryWriter, XmlDictionary, XmlDictionaryString, XmlQName (I think that's the name?), XmlConvert etc which would be a bigger change.

[GrabYourPitchforks]

@mconnew That's a good point. An earlier version of this was marked [Serializable] but got scrapped. Treating it as a primitive would make more sense. Would need to look into what support (if any) was given for the new Half type in .NET 5.

[mconnew]

I don't think we should do anything special for the Half type for any Xml reading/writing or serialization as the XML standard data types doesn't have a 16 bit floating point number. So it's a really bad idea to use Half in a type which will be serialized as the most restrictive schema type would be to constrain the value in XML to a 32 bit floating point number (xsd:float). Which means when consuming an XML document with a floating point value, you have to accept that the value could require 32 bits. The alternative would be to invent our own custom Schema type, but that's a bad idea as it has interop issues.

[ceztko]

It's probably too late at this stage to enter the discussion about how to improve UTF-8 support in .NET but still I would like to ask if alternative approaches, other than adding a brand new string class, has been discussed. More specifically, has multiple storage backends in string been considered? The basic idea would be to have both char[] and byte[] arrays in the string class, initialize just one of them by default (not necessarily the UTF-16 char[] array first, it could be also the UTF8 byte[] one depending on the platform/language of the system) and having the other to be initialized lazily in a thread safe manner, when and if needed. Specific accessors the byte[] array one other encoding agnostic APIs should then be added to string. This approach seems to be not new and actually adopted in an important programming framework: NSString in Apple Foundation library is described in the documentation as a sequence of UTF-16 code units, but allows to access an UTF-8 encoded const char* array that has the same lifetime of the string[1]. My main concern/fear about Utf8String is that by design that it will not (and can't) be a full drop-in replacement of string, making it no so interesting to create a full ecosystem around it. Instead having multiple storage backends in string could have the potential of allowing UTF-8 to be the main encoding in the runtime (again, this could be configured to be platform/language dependent). One cost of this would be a (slightly?) less efficient implementation that still allows for many optimizations to avoid both array initializations, especially in the case of ASCII strings where storage array length is same in both UTF-8/UTF-16 encodings and random access to the char array actually makes sense (in Unicode aware programs getting the length of the underlying array or indexing characters is often/probably a programming error). Of course the implementation of string class would be also much more complex.

[1] https://developer.apple.com/documentation/foundation/nsstring/1411189-utf8string?language=objc, "This C string is a pointer to a structure inside the string object"

[GrabYourPitchforks]

@ceztko Dynamically switching between byte-backed and char-backed strings was considered but largely rejected. Managed strings are defined in .NET as having string data (as UTF-16) inline with the string object itself. If we were to have string dynamically backed by bytes or chars, it would break every single _fixed_ statement in every single C# app or library that has ever been written. Pretty much any application beyond the simplest "Hello world!" program would fail at runtime.

[ceztko]

@GrabYourPitchforks thanks for the prompt comment. I would like to ensure you understood my proposal, since I have the feeling that you misunderstood it: I'm not suggesting to choose either char[] or byte[], statically or dynamically, and just have a single backing storage during the whole execution of the program. I'm suggesting to truly have two backing storage arrays, char[] and byte[], initialize one by default, and have the other initialized lazily, when and if required, during the same execution. string behavior would not change and no program would break: string will still have the same API to thread it like an UTF-16 string, but could instead be initialized by default with a UTF-8 byte[] depending on platform and have a new API to access specifically the UTF8 storage and more APIs to access it a encoding agnostic way (for example a string.UtfLength that return the size of the default backing array, either the char[] or byte[]), to avoid initialize both arrays in case the default backing array is the UTF8 byte[]. Many optimizations should be possible to avoid initialization of both arrays (especially in the case of ASCII strings).

This approach of course requires a lot of considerations on performance, also to guarantee string immutability (lazy initialization of the other backing array should be thread safe). Also minimum space occupation for a string grows a few bytes for the additional backing array pointer and possibly a flag with the state of the string. Also of course both arrays can be initialized having the string occupy more space during the whole program execution. Still, I don't expect average application that will rely on Utf8String to occupy any less space in RAM.

Have you ever discussed such proposal in these terms? As said NSString in Apple Foundation library allows to store multiple encodings at the same time, and I have the feeling they are using this strategy to gradually move all operating system internals to UTF-8, including syscalls, especially now that Swift language strings moved internal representation of string from UTF-16 to UTF-8[1].

[1] https://swift.org/blog/utf8-string/

[GrabYourPitchforks]

Yes, I understood your comment. The problem is that in .NET, it's assumed that strings have a specific format in memory. That is, if you have a string "Hello!", the raw contents in memory are:

xx xx xx xx 06 00 00 00 48 00 65 00 6C 00 6C 00 6F 00 21 00 00 00
|           |           |                                   +-- null terminator
|           |           +-- "Hello!" UTF-16 data
|           +-- length
+-- object header

Of interest here is that a string object doesn't _reference_ a byte[] or a char[]. (This is typical of runtimes like Java.) Instead, in .NET, a string object _is_ the raw data. The .NET compilers have made this assumption since the dawn of time. If you ever used the _fixed_ keyword in any code you've ever written, the compiler has baked knowledge of this layout in to the resulting EXE or DLL.

That's the risk of making a change to how string is laid out in memory. If we move anything here, or if we change it so that the data is not contained inline, but rather that it wraps a byte[] or a char[], we can't go back in time and update the entire ecosystem of DLLs which may have been created between 1999 and 2020. When those existing DLLs are brought forward onto the new runtime, they'll misbehave because the new runtime layout doesn't match the expected layout that has been hardcoded into the DLLs.

We've experimented with having the runtime identify these DLLs and patch them on-the-fly when they're loaded. But this is very fragile and isn't a viable long-term solution, and good luck debugging the app when the heuristic gets it wrong. Another thing we could do is block loading these DLLs entirely. But then we would have bifurcated the ecosystem into "compiled before 2020" vs. "compiled after 2020" - and you can't bring your existing libraries forward without recompiling them. This would cause too much pain.

[ceztko]

@GrabYourPitchforks thank you very much for the clarification. I understood that string had optimized memory layout but I could not exepect it to be the RAW object itself. Indeed a very space efficient solution: I rember I acutally tried to use a similar memory model (length and string data in the same contigous memory) in an UTF-8 string interop test I did some time ago. I'm happy to know you also evaluated low level solutions. If it still makes sense to talk about this now, I would love to still encourage you to pursue low level solutions: there's no sense in considering DLL and EXEs written for .NET framework (1996-2016) because they should be probably ported anyway to .NET Core/5.0/6.0.... I guess also recompilation of DLLs in this case would not force an upgrade of the .NET SDK but an upgrade of the compiler only, which should stop producing CIL that assumes strings memory layout. Unless you tell me all string marshaling scenarios are using these hardcoded assumption, and not just a limited set of DLLs that do custom string marshaling with fixed, you totally have my support in modifying the memory layout of string to make it more UTF-8 friendly, if it is still possible to do it.

[GrabYourPitchforks]

For what it's worth, if your app / library targets _netcoreapp3.1_ or _net50_ or later, the compiler does not bake knowledge of the shape of string into the resulting module. It instead uses an abstraction where the runtime can determine the shape of the string.

If your module targets anything else - including _netstandard2.0_ - the compiler bakes knowledge of the shape of string into the resulting module because the abstraction does not exist.

[ceztko]

@GrabYourPitchforks that's great! It's just my dream, ok, but let's for a second suppose that you want to change the _shape_ of the string in .NET 6.0 runtime to introduce a first citizen support to UTF-8 in a way that doesn't require adding a new Utf8String. Then your problem would just be netstandard2.0 and earlier modules, right? I think these could also be recompiled in such a way that you put trampolines in each fixed statement to check first if abstraction is available or not, and rely on legacy string shape in the latter case. Performance penalty of trampolines can be (mostly?) cut away by JIT. Then on .NET 6.0 you could require _netstandard 2.0_ and earlier modules to be recompiled with trampolines to be used (just recompiled, not upgraded to another framework), correct? I understand that is not an easy choice but I would really love to see that happen instead of a Utf8String class, for which the success is hard to predict. And I already know that I will hate to see improper mix of Utf8String and string, when it happens (because it will happen...).

[GrabYourPitchforks]

@ceztko I appreciate the enthusiasm around this, but honestly requiring the entire world to recompile their apps is just not a realistic proposition. One of the big pieces of feedback we got in the .NET Core 1.0 timeframe was that people wanted to take _their existing DLLs_ and pull them onto later runtimes. Often these were third-party NuGet packages. On occasion they were assemblies made by a vendor under contract, for which the contract had since expired and the source code was no longer available. These customers were blocked from migrating to .NET Core due to these issues. Given how painful that experience was for all involved, I can't fathom pushing an _even larger_ breaking experience onto customers.

[ceztko]

but honestly requiring the entire world to recompile their apps is just not a realistic proposition

But this already happened when moving to .NET framework 4.0, where (all?) .NET 2.0 compiled assemblies required _useLegacyV2RuntimeActivationPolicy_ flag to be loaded. Today similarly you would introduce a new flag that basically tell runtime to either try to patch DLLs (like you experimented) or run with the old string layout. The point is also that if today's compilers were already able to produce netstandard2.0 modules that take advantage of the string abstraction, if present, then the number of _unloadable_ modules would become very low with time, so maybe this could be a topic that may be worth some investment. I have the feeling that you discussed extensively about these topcis offline and maybe in other github issues, so sorry if these discussions tend to be repetitive for you.

[GrabYourPitchforks]

But this already happened when moving to .NET framework 4.0, where (all?) .NET 2.0 compiled assemblies required useLegacyV2RuntimeActivationPolicy flag to be loaded.

Not quite. Only mixed-mode C++/CLI assemblies were affected, and that's a _very_ small minority of assemblies. And even then, upgrading didn't require recompiling the existing assemblies. The flag only needed to be set at the app's entry point. Think of it as akin to an environment variable.

Contrast this against the _fixed_ keyword. Our analysis shows that today, approx. 2/3 of DLLs in the .NET ecosystem pin strings. As of this writing, NuGet hosts 2,835,410 unique packages. If 2/3 of them pin strings and our heuristic is 99% accurate, this means that our heuristic will miss __18,903__ packages. Those packages will experience undefined behavior at runtime. This undefined behavior might manifest itself only in production, not in testing. (These numbers account only for public NuGet packages, not the myriad private DLLs that exist within enterprises.)

Your point about the switch being opt-in rather than opt-out is well taken. But once the switch becomes opt-in, then at this point we may as well just take the break and say, "Look, friendly app dev, you've taken an explicit action to set this switch. Since you've opted in, all DLLs must be compiled using a recent compiler. We'll fail to load DLLs compiled with an older compiler." (There was some discussion on this earlier in this thread.) Granted, it means very few people actually opt in, but at least it solves the undefined behavior problem. :)

[Symbai]

Granted, it means very few people actually opt in

If the pros overwhelms the cons then most people will accept it and switch over time. And most DLLs used in many applications are actually open source now. Things have heavily changed here comparing to 10 years ago. It wouldn't be difficult to download and recompile if a pull request is not possible. In the end... its just a switch. You don't have to re-write lots of code.. at least in 90% of the DLLs we speak about. It appears to be the same kind of effort for library authors than to upgrade your project to .NET Core and .NET 5. And look how many popular libraries already got updated.

Speaking so, I have absolutely no idea how much benefit such a change would be. Do we get a lot of performance boost in all kind of situations? Or does this change only matters for a few handful of people doing some low level stuff?

[GrabYourPitchforks]

Speaking so, I have absolutely no idea how much benefit such a change would be. Do we get a lot of performance boost in all kind of situations? Or does this change only matters for a few handful of people doing some low level stuff?

Changing the internals of _string_ would improve perf for people performing rudimentary operations. Some scenarios: comparing two strings for equality, performing simple case conversion, maybe using them as keys in a dictionary. If you instead call a more complex API that's just going to turn around and project the string instance as a ReadOnlySpan<char>, we're now paying _both_ the CPU overhead of the conversion (fast but measurable) _and_ the memory overhead of bookkeeping, which on 64-bit platforms would add around 28 bytes per string instance.

[mconnew]

I think making Utf8String a first class citizen in .NET would be a better option. Anything which has special handling for strings should have special handling added for Utf8String. For example, adding methods/overloads on XmlReader/XmlWriter and TextReader/TextWriter to handle a Utf8String. Or maybe a Span and have an implicit cast for Utf8String. You don't want to be jumping back and forth between a String and Utf8String as it's expensive. Making a single String act like both will mean you are regular converting from one to the other and then you lose your performance gain. Anything which erases your perf gains in a hidden implicit way is a bad idea because use in a hot code path can happen accidentally. You really want to avoid the situation where adopting Utf8String actually slows your app down.

[ceztko]

@mconnew

I think making Utf8String a first class citizen in .NET would be a better option. Anything which has special handling for strings should have special handling added for Utf8String

The design proposal is saying that this will not happen. If introduced Utf8String will have its uses in networking/io and interop scenarios.

you don't want to be jumping back and forth between a String and Utf8String as it's expensive

In my pessimistic view, if Utf8String is introduced, the more it is used the more "jumping back and forth" will happen. That's why I would rather prefer low level solutions that do not involve introducing a new class.

[jkotas]

This is enlightening discussion. Let me summarize the options:

1. Introduce Utf8String as a new type
2. Pros: Benefits code that was converted to use Utf8String.
3. Cons: May require a lot of conversions that wipe out any benefits in real apps

4. Introduces tensition in the library design forever: Do all methods that take String have to have an overload that take Utf8String as well now? Where to draw the boundary?

5. Transparent dual representation

6. No code changes required for functional correctness

7. The overhead from dual representation is likely to wipe out or even outweigh any benefits in real apps. Unclear whether we would be able to find any workload that matters and that would benefit from this.

8. Change System.String internals to use UTF8 representation, unconditionally.

9. Most invasive change, it would need to be opt-in, but best for the long term.
10. It is what we would do if we were designing .NET today.
11. We would likely need to change "char" to be 1 byte in this mode as well.
12. Requires recompilation of all code involved and code changes.
13. No tension between String and Utf8String in the API design.

We have dismissed option 3 earlier as too hard to pull off, but we may give it a second though. It may not be as bad as it sounds. It would certainly be the best option for the long term if we can pull it off.

[mconnew]

There's a subtle detail which is probably important for option 1. Some things would need to take a Utf8String, others would be fine with a Span\ There's also the option of separating the api's up into two worlds. The old way of Streams, Strings, ArraySegments, and (buffer, offset, count), and the new way of PipeReaders/PipeWriters with Span. We could recommend if you are using Utf8String, then you need to be using the newer set of api's for IO.

[Symbai]

My vote is option number 3. And introduce this on a version which contains some breaking changes as well. So developers are clearly notified about the version they upgrade to, needs some attention. .NET 6 is considered as LTS version, so maybe .NET 7.0 would be a better choice, give people a breath and make it a standard on .NET 8.0

This might block certain apps from upgrading but in the long run, most apps and libraries will adopt. Performance has always been a killer argument for introducing breaking changes or doing the extra work. Most devs including myself have switched to .NET (Core) whenever it was doable because it felt like the better solution. Most devs switched from Newtonsoft to System.Text.JSON and this definitely needed some code changes but it was worth it.

For sure such as a deep change needs some serious and long discussion. But I'm absolutely sure that whatever brings some real benefit into performance defeats all doubts. It always have and always will. The only question is: If this benefit big enough to justify the change? Using strings in .NET has always been costly. And strings are used a lot.

The reason why the other options aren't as good as option 3 is because I feel like it would rarely be used. And even more painful, there are plenty of situations where it simply CANNOT be used, even if you want. Consider desktop applications that uses Localization. Resources are handled just like they are. Devs don't have any influence on this. Existing methods and overloads which accepts a string, might not accept an Utf8String. And even if they do, do they really benefit or will the Utf8String be converted into a .NET string?

The more we split in .NET, the more we split how devs are coding. Having a choice is great but only having a choice because of compatibility is sometimes unnecessary. .NET (Core) is designed to cut off old behavior from time to time. It is not meant to be compatible to stuff coded in 1990. If you want compatibility, you stay on .NET Framework. That's how I remember .NET (Core) was introduced.

[jkotas]

This might block certain apps from upgrading

That won't be acceptable for us. The Utf8 string representation would have to be opt-in.

[Symbai]

The Utf8 string representation would have to be opt-in.

Then you misunderstood. Opt-in is what I meant. Opt-in within .NET X and make it default in .NET X+1. What I meant with block certain apps from upgrading, that some people might not appreciate a breaking change. Like it has been on all breaking changes so far. Just because some people don't appreciate a breaking change, it doesn't mean a breaking change is wrong.

[jkotas]

.NET X and make it default in .NET X+1

It would need to be more like opt-in within .NET X and make it default in .NET X+7. 1 year would not be enough for the ecosystem to adopt a change of this magnitude.

[ceztko]

Change System.String internals to use UTF8 representation, unconditionally.

In the long term this would be definitely a smart move. What about being a introduced as a breaking change asap in linux/mac platforms with an opt-out? Also there should be some pressure on Windows system/kernel devs so they'll eventually offer an UTF8 API on their internals (with UTF8 possibly being the internal string representation as well).

[jkotas]

The playbook for any of the ideas in this space would be:

1. Start an experiment in dotnet/runtimelab.
2. Build enough to prove that the idea works for some real customer workload that matters.
3. Propose to bring the idea to mainline. This is where we can start talking about timing of opt-ins and defaults.

[Serentty]

@ceztko

Also there should be some pressure on Windows system/kernel devs so they'll eventually offer an UTF8 API on their internals (with UTF8 possibly being the internal string representation as well).

If you set a certain field in the manifest of an executable, “ANSI” strings will be treated as UTF-8. This makes non-wide string handling in C and C++ similar to Unix-like platforms. However, this was only added recently (last few years), and I'm not sure if the last version of Windows 10 not to have it is end-of-life yet. As for changing the internal representation, I imagine it would have far reaching implications that would be hard to debug. Making big changes in .NET is a lot easier than making big changes in Windows. Thankfully, if option #3 goes through (and I hope it does), .NET will be able abstract that stuff away.

[tannergooding]

We would likely need to change "char" to be 1 byte in this mode as well.

@jkotas, I'm pretty sure this is a non-starter. The C# compiler hardcodes sizeof(char) to be 2 and treats it as a C# compile time constant.

[jkotas]

Right, we would need to change C# compiler to make this work. I do not see why it would be a non-starter.

[tannergooding]

Right, we would need to change C# compiler to make this work. I do not see why it would be a non-starter.

It's the same avenue as language "dialects" which has been the type of change the compiler team has tried to keep out of the language. Changing the size of char basically means the entire .NET ecosystem needs to be audited, rewritten, and has to be assumed is no longer compatible with any existing code once the UTF8 switch is toggled. Anything that used sizeof(char) is now invalid and incompatible. The same goes with anything using a number of the APIs such as char.IsSurrogate, which use sequences such as `'u202F', etc. This also would impact other languages such as F# and VB and likely all parsers written in these languages, including Roslyn itself.

At least to me, it seems like a bifurcation of the ecosystem that is worse than just exposing UTF8String and Char8 types that are essentially identical to String and Char, but being 8-bits. In such a world, just like with Span, many APIs could just be a thin wrapper that forwards to a shared implementation dealing with an enumeration of Rune. This could be made efficient with value enumerators and generic specialization ( Method<T>(T enumerable) where T : struct, IEnumerator<Rune>):

• We already have such an enumerator for string, just no APIs that specialize on it and string itself is not IEnumerable<Rune> so it isn't a simple passthrough, but a method call instead
• This "solves" the issue of needing to expose one, the other, or both.
• The same "issue" as with Span may exist that causes us to want to expose both T[] and Span<T> overloads
• Such APIs could handle other encodings, UTF32 data or "code pages", as they simply operate on Rune and rely on the enumerator to decode correctly
• For ASCII text or for UTF16 text with no surrogates, this is just simple iteration
• For any surrogates or multi-byte sequences, this would involve the same logic as I'd expect most parsers to already handle to construct something usable.
• This wouldn't necessarily handle custom handling of invalid sequences, but I believe that's something we could come up with a solution for

Likewise, there are downsides in that:

• This will likely incur a penalty initially due to the majority of the ecosystem using String/Char, as that's all that exists today
• The surface area will explode as APIs are updated to handle both UTF8 and UTF16.
• Not all scenarios may work well with rune based sequences

However, I'd think that overall we are no worse off then we are today.

• Codebases that don't care don't need to make any changes and continue to work, with the same penalties or lack thereof
• Codebases that need to switch for perf reasons can and will only have penalties where a dependency requires the other and can decide whether it is acceptable or not
• Codebases that want or need to support both (such as via multi-targeting) can continue doing so have a clear path to do so without needing to #ifdef

You'll always have potential issues with interop where a native API expects one vs the other. However:

• Unix is almost exclusively UTF8, as is the web
• Windows supports UTF8 today in most of its *A APIs, provided you enable the UTF8 codepage: https://docs.microsoft.com/en-us/windows/uwp/design/globalizing/use-utf8-code-page

I think this would put us in a much more sustainable location, without needing to make drastically breaking changes (and if we were going to be making drastically breaking changes like this, there are likely a number other scenarios that would be good/desirable to "break" simultaneously).

[mconnew]

Personally I'm opposed to a Char8 type. My reasoning is the name "char" is short for "character" and implies it represents a character, but it might not. When you have multi-byte characters then a Char8 wouldn't represent a character. We already have byte which represents 8 bits in memory, and many low level api's already accept a byte[] (such as streams) which would reduce the api explosion. It's very rare that someone actually cares about individual entities in a string except as a sequence of bytes to read or write and then they don't care about character boundaries, or they want something to represent a single character. I think we have a chance here to simplify how UTF strings are handled. Having a Rune represent a single visual character simplifies things like case insensitive compares from a developers standpoint. Today developers need to think too much about multi-byte characters and introducing a Char8 doesn't help with that.

[tannergooding]

When you have multi-byte characters then a Char8 wouldn't represent a character

This is (essentially) already the case today. Most text doesn't contain surrogates, but the second you encounter one, any assumption that char represents a single processable thing goes out the window. That is, if you want to work with Rune or do anything other than treat it as invalid/check if it is a surrogate, you can't just have a low surrogate or just a high surrogate. Now, UTF-16 is slightly better because you can have at most two surrogates, a low and high, and you can detect the difference; while UTF8 on the other hand only lets you differentiate "first" (11xxxxxx) vs "trailing" (10xxxxxx) and so you might have to backtrack up to 3 to "know". I don't think this distinction is particularly useful most of the time, however.

Today developers need to think too much about multi-byte characters and introducing a Char8 doesn't help with that

I agree, you shouldn't, unless you really need to, need to think about UTF8 vs UTF16 vs UTF32 vs ASCII vs "some code page" vs "some future alternative". If your API instead deals with Rune then it doesn't matter what the underlying format is, everything "just works" as you don't need to worry about encoding or decoding. You get decoded characters and handle them as a complete unit and a user can give you whatever they had to start with. It's the same principle as Span where you shouldn't need to care if its GC tracked or native memory, you just want to be able to index into it and get the value.

I think we have a chance here to simplify how UTF strings are handled

I'm, personally, not convinced that this change is work completely breaking the ecosystem nor that things will ultimately be simpler. This is especially for libraries that need to multi-target because they need to continue supporting full framework or netstandard.

That all being said, if we were to take such a break, then there are also plenty of changes that have been previously rejected as "too breaking" but which are consistently brought up (several of which are also based on a year 2000 32-bit Windows centric view, and not saying these are all good or bad):

• Array.Length, Span<T>.Length, and ReadOnlySpan<T>.Length should be nint, not int (even if the GC maintained the internal int limit)
• ICollection<T> should implement IReadOnlyCollection<T>
• IEnumerable<T> should implement IEnumerable.GetEnumerator by default
• Array covariance
• Array shouldn't support non-zero based indexes
• Can't overload based on generic constraints
• Can't use pointers with generics
• Can't overload based on ref vs in
• The JIT can't optimize based on much of the language encoded data (such as readonly)
• Non primitive constants aren't supported
• Certain IL patterns are expensive to handle or represent which can hinder other optimizations
• etc

[jkotas]

It's the same avenue as language "dialects"

I do not see this as a language dialect. To me, it looks same as nullability (is nullability a language dialect?), or the portability and trimmability analyzers. Nobody is forcing it on you. You can opt-in, you will get a bunch of errors, and once you fix them, you get the benefits.

Changing the size of char basically means the entire .NET ecosystem needs to be audited, rewritten

I would not expect the entire .NET ecosystem needs to be rewriten, initially at least.

UTF8 encoding for strings is basically about up to 10% improvement in a typical app. Improvement in this range would not be noticed by 90% of the .NET apps out there. It is only the hyper-scale services and top-tier apps that really care. These services and apps will drive the changes in the components they use to maximimize the benefit of UTF8.

With 1, these changes would be about introducing Utf8String overloads in number of places.
With 3, these changes would be about fixing places that assume char is a 2 bytes. I believe you would need to touch a lot fewer places vs. Utf8String overloads (hypothesis to validate). The places that need behavior differences can use pattern like if (sizeof(char) == 2) { ... } else { ... } that the JIT optimizes depending on the actual runtime sizeof(char). Notice that this is fully backward compatible.

I'd think that overall we are no worse off then we are today.

Utf8String will be a second string exchange type. Even if one does not care about it, I am worried that it will start showing up in number of places, and one would not be able to escape it, 90% of users out there will see more complexity without much benefits. I despise the multiple string types that you often end up dealing with in C/C++, they produce very unplesant experience. I have people regularly complaining to me about how complex .NET became, sync vs. async in particular. I would not be looking forward to complains about string vs. Utf8String.

If we go with the Utf8String plan, I would like to see written down guidance for when to introduce Utf8String overloads, estimate the impact of this guidance on the platfom surface, and see whether we are happy with the amount surface that it will end up introducing to produce the desired performance benefits.

[jkotas]

previously rejected as "too breaking" but which are consistently brought up

FWIW, the problem with number of changes in your list is a poor cost/benefit ratio; not that they are too breaking.

[tannergooding]

is nullability a language dialect?

No, because you can freely mix nullable and non-nullable code, even within the same compilation. Ideally everything transitions, but it is not a strict requirement.
Likewise, a nullable-aware library can freely interact with a non-nullable-aware library or vice-versa. It only makes a difference on the diagnostics reported and was done this way because the alternative would have been a dialect and vastly breaking to the ecosystem (CC. @jaredpar).

On the other hand, with the proposed change where char becomes 1-byte and string is internally utf8, any existing .NET code is now incompatible.
While you could catch some things via static analysis, such as char c = '\u202F', you will not catch things likesizeof(char)because it is simply emitted into IL as2, likewise anything that just went and did2rather thansizeof(char)would need additional handling. Just scanningdotnet/runtimefor* 2` shows up quite a bit of native code, including in the GC, which would need to be updated because it assumes the size.

I believe this means that all code must be recompiled, at a minimum, to add some "utf8 string" flag and so the issue becomes that the entire .NET ecosystem must now recompile.

• For almost anything that deals with string indexing, comparing characters, or non-ASCII text, this means auditing the codebase and potentially adjusting it.
• For something like Roslyn, which multi-targets full framework and .NET Core (and likely must continue doing so for the foreseeable future), it likely means maintaining effectively two copies of the codebase.
• For apps which don't directly use string/char themselves, it still means that all dependencies must first be recompiled to support UTF8.
• For libraries where it is directly beneficial to use UTF8, no one will be able to use them without also retargeting.

Utf8String will be a second string exchange type. Even if one does not care about it, I am worried that it will start showing up in number of places, and one would not be able to escape it, 90% of users out there will see more complexity without much benefits.

The fundamental problem, that I understand, is that we have an entire ecosystem that takes System.String. For web apps, and many scenarios dealing with mostly ASCII data, this means we are using twice as much memory and therefore process data slower.
We can update a bunch of APIs to take ReadOnlySpan<byte> or UTF8String but this increases API surface and is a potential burden due to needing to maintain two code paths.

If we look at string, the underlying data format doesn't often matter. The places where it does is when you are indexing or when you are trying to interpret/handle individual characters, rather than runes. Now, if you are indexing and the underlying data format matches, it is a constant time lookup. If it doesn't match, the lookup becomes potentially O(n) and you must decide which is cheaper: "converting" or taking the "O(n)" hit. However, the lookup being O(n) only happens if the underlying data contains any non-ASCII data as when it is ASCII only, the lookup is still trivial.

For the scenario where you don't need to index, we could have a common abstraction that only allows you to enumerate the underlying runes. For apps and libraries that aren't concerned with the type, they can prefer taking/returning this type and provided they don't index, they don't need to be concerned if the data is ASCII, UTF8, UTF16, UTF32, or some future hypothetical type. This was the basis for the common Method<T>(T) where T : struct, IEnumerable<Rune> overload I suggested above.

The remaining problem is what to do with APIs that haven't been updated yet or which must index into the data in terms of characters and I don't necessarily have a good proposal for this. Ideally, over time, everyone would transition to not depending on indexing into the string and so it wouldn't matter what the format was. You'd return and deal with UTF8 by default unless you had mostly non-ASCII data, were calling legacy code, or had to interop with some native API that required UTF16 (most Windows, certain unix APIs, etc). BCL APIs would largely just work and take whatever you provided. If any API only takes string, then you can make a call on what is appropriate: "convert" or "pass in a special JIT wrapper an take the O(n)" hit, if one even exists.

[tannergooding]

The places that need behavior differences can use pattern like if (sizeof(char) == 2) { ... } else { ... } that the JIT optimizes depending on the actual runtime sizeof(char). Notice that this is fully backward compatible.

This, in particular, is also only compatible with some new C# compiler. Today, because sizeof(char) is constant, the else path is always dropped, even in debug builds: https://sharplab.io/#v2:EYLgtghgzgLgpgJwDQxASwDZICYgNQA+AAgEwCMAsAFBEDMABKfQML0De19XjDRALPQCyACgCU7TtyloAZvWFQ0ALzgB7GcOYALCAnEBeffRKjJUrhyrnrjMgE5hAIgBiq1Y9EBuM+YC+PqTgMKDgA7ksbcyJ7JwAhXQ9vKxt/ZK5U3yA===

[jkotas]

the issue becomes that the entire .NET ecosystem must now recompile.

Only the part of the .NET ecosystem that wants to be compatible with the UTF8 performance optimization. It is similar to compatibility with IL trimming (binary size performance optimization). Many libraries out there are incompatible with IL trimming and your app will break unpredictably if they use these libraries with IL trimming enabled today. We are building analyzers and annotations and expect libraries to audit and adjust their codebases to be IL trimmable with confidence. Many libraries out there will remain incompatible with IL trimming and that is fine.

For almost anything that deals with string indexing, comparing characters, or non-ASCII text, this means auditing the codebase and potentially adjusting it.

Yep, it would be useful to understand the cost of such auditing vs. creating Utf8String clones of many APIs in real world code.

For something like Roslyn, which multi-targets full framework and .NET Core (and likely must continue doing so for the foreseeable future), it likely means maintaining effectively two copies of the codebase.

I agree that there would be changes required in Roslyn if it wanted to be compatible with this optimization. I expect 99+% of the Roslyn code can stay intact and Roslyn should be still able to ship a single binary that runs everywhere and that is compatible with the UTF8 performance optimization.

For apps which don't directly use string/char themselves, it still means that all dependencies must first be recompiled to support UTF8.

Yep, only if the author of app wants to opt-in into the UTF8 performance optimization.

For libraries where it is directly beneficial to use UTF8, no one will be able to use them without also retargeting.

Libraries that squeeze last bit of performance by using UTF8 internally typically try to avoid allocations as much as possible, use buffer pooling, etc. Immutable Utf8String is unlikely to be very useful for them.

The fundamental problem, that I understand, is that we have an entire ecosystem that takes System.String. For web apps, and many scenarios dealing with mostly ASCII data, this means we are using twice as much memory and therefore process data slower.
We can update a bunch of APIs to take ReadOnlySpan or UTF8String but this increases API surface and is a potential burden due to needing to maintain two code paths.

Yes. And unless a significantly large percentage of API surface and libraries is updated to take and produce Utf8String directly in addition to string, mixing UtfString8 and String will require conversions between the two and can easily consume even more memory and have worse performance than using String alone today. It is the reason why Utf8String has been a contentious feature and why I think it may be worth exploring this alternative.

[tannergooding]

Only the part of the .NET ecosystem that wants to be compatible with the UTF8 performance optimization. It is similar to compatibility with IL trimming (binary size performance optimization). Many libraries out there are incompatible with IL trimming and your app will break unpredictably if they use these libraries with IL trimming enabled today. We are building analyzers and annotations and expect libraries to audit and adjust their codebases to be IL trimmable with confidence. Many libraries out there will remain incompatible with IL trimming and that is fine.

The main difference between this change and almost every other change made so far is that this is a break, rather than an additive change, and break in one of the most primitive layers. If I want to support IL trimming, it is largely an additive change on top of what I have. There may be some optimizations that need to happen, but for the most part my code stays as is and I'm likely just adding in annotations. There are external ways to provide limited annotations to make things work and in the general scenario, I may just end up with a larger set of binaries than I actually need if everything were correctly annotated. The same goes for most of the major changes we've made.

With this change, any string-as-utf8-compatible assembly becomes fundamentally incompatible with everything else. I have to move to a new compiler, with a new language version, which will be "unsupported" on anything other than .net x+.
Any interop code that did fixed (var pstr = "string") is now broken, any code that compares against a char or indexes into a string needs to be audited, all APIs that explicitly used the name utf16 to differentiate from the utf8 overload are now wrong and unnecessary. All of the code that is converting strings by using Encoding.UTF8.GetBytes is now suboptimal and needs to be adjusted, etc

It's entirely possible that I'm wrong and overthinking this. I think it is very likely that UTF8 is the better long term solution, particularly with the latest Windows versions now supporting it in their API surface, with it being the 'de-facto' standard for the web, and the general default for most file encodings, etc. It just seems like a massive break (unparalleled with anything else we have done) with a huge tail end of bugs, particularly in important, performance oriented, lowlevel code.

[jeremyVignelles]

It's entirely possible that I'm wrong and overthinking this.
I don't think your point of view is wrong. I find both insights very interesting and both have pros and cons. I don't know the real implications of both solutions.

I'd like to give you my perspective, as someone who've worked with .net for the last 6 years.

To me, there's no question that, in 2021, UTF-8 is way more common than UTF-16, except for the Windows world, but .net has a legacy to deal with, and I'm quite torn between the two point of views.

The Utf8 type is easier to introduce, but needs to introduce (a lot of) overloads in the CLR and utf-agnostic libraries. Having several type of strings is not easy to deal with, and I agree with the point that it might introduce too many conversions when dealing with multiple libraries.
For a library like LibVLCSharp, we get UTF8 strings from libvlc, and we would need to choose between:

• Keep things as-is and do the conversion to UTF-16, potentially doing the conversion twice if the string needs to be handled in UTF-8 later (web api...)
• Introduce a breaking change for platforms that have the Utf8String type. Upgrading for, say, .net core 5 to .net core 6 is now a breaking change
• Duplicate the APIs (have a Log event (where you register your log callback), and a LogUtf8 event). That's quite complex to implement, and the weight of having to deal with two different types starts to leak on library maintainers.

There are also other kind of costs incurred by having two types of string:

• While developing, we need to make a conscious decision about which type we need to use : What do I need to do with this type? (I had some experience with C/C++ and the char_t/wchar_t has always been a pain to work with).
• What would be the type of var s = "Hello";? Still a string? Would I need to write var s = u8"Hello"; or something ?
• When coming from a different language and learning C#, you'll be frustrated that you can't just use string because "Damn, that API needed a Utf8String". It's already difficult to understand the fundamental difference between int? and string? and why you can't just use T? in a generic overload seamlessly.

That said, breaking everything is far from easy, as described earlier.

fixed (var pstr = "string")

Not sure there is many code outside of the runtime that does that, because I'm not sure there are a lot of APIs that use UTF-16 strings other than the Windows APIs. On the other hand, being able to use UTF8 natively would be very helpful for interop with C apis like libvlc.

all APIs that explicitly used the name utf16 to differentiate from the utf8 overload are now wrong and unnecessary.

I think that would be an edge case, but the naming would indeed be wrong. I'm not sure it was really a best practice to call some overload taking a string utf16 assuming how it was encoded behind (whether it breaks or not depends on the implementation used), and the utf8 overload would take a byte[] which will likely have no breaking change.

̀Encoding.UTF8.GetBytes` is now suboptimal and needs to be adjusted, etc

If string is replaced, that method can be rewritten to just copy the bytes, or am I missing something?

Now what would happen with existing code if we break ? Say we have project A that depends on B, would both need to be updated ? Updated to what?
Would it be a flag that says "this assembly is string-length agnostic" or a flag that says "this assembly has a string length of 1".
In the former case, that would let assembly B be consumed in any assembly, whether or not it has been updated. But I fear that it's the latter option, and that both assemblies would be incompatible.

Can I suggest something ? What if existing assemblies (with UTF-16 strings) were run in a "compatibility layer" if they are not updated ? That way, the updated assemblies (the A in my example) could still consume old assemblies by automatically exposing "old-style strings" as an OldString class ? A consuming code could look like:

OldString result = B.DoSomething("Hello".ToOldString());
string res = result.ToNewString();

I don't really know the implications of that, but that might help the transition. Does that sound reasonable from a CLR point of view ?

Incompatible assemblies seems hard to deal with in real-world applications. Would libraries need to be built twice for those who can't upgrade yet ?
What if my app depends on a new library that only supports utf8 strings and still depends on an obsolete library because the maintainer has left ?

[jkotas]

For a library like LibVLCSharp, we get UTF8 strings from libvlc,

Thank you for sharing this example.

Encoding.UTF8.GetBytes` is now suboptimal and needs to be adjusted, etc

If string is replaced, that method can be rewritten to just copy the bytes

It would need to validate that the bytes are valid utf8 encoding in addition to copying the bytes to maintain the System.Text.Encoding contract. I agree with you that it does not look particularly concerning.

Would it be a flag that says "this assembly is string-length agnostic"

I think so.

or a flag that says "this assembly has a string length of 1".

I do not think we would encourage, or even introduce this option, initially at least. We would have existing assemblies that assume sizeof(char) is 2, and agnostic assemblies that do dynamic checks for sizeof(char) as necessary. Both of these would be compatible with the standard runtime mode where sizeof(char) is 2, but only the latter would be compatible with the utf8 optimized runtime mode where sizeof(char) is 1.

What if existing assemblies (with UTF-16 strings) were run in a "compatibility layer" if they are not updated ?

The runtime would have to convert the strings on the boundary between the compatible and incompatible assemblies. It gets hard and inefficient pretty fast. For example, imagine that you have a method that takes List<string> argument on this boundary. The runtime would have to insert code to convert the List<string> from one string type to the other string type to make things work...

What if my app depends on a new library that only supports utf8 strings and still depends on an obsolete library because the maintainer has left ?

You will always have an option to run your app in the standard runtime mode where sizeof(char) is 2 and give up the benefit of the optimization.

[jeremyVignelles]

Thanks for your reply, I really appreciate the open discussion here 🙂

If you say that a "char-size agnostic" option could be implemented, that would be an awesome option as it could be progressive through the ecosystem.
My fear was that it would change the size of some structs (that contain a char, for example), thus potentially breaking the ABI (if that term means anything in the C# world).
It would also change the range of values of the char type, but I don't know if that's a big deal, I wouldn't represent integer values in a char in C# (while I might be tempted to do so in C).

Thanks for the discussion, i'll keep following the thread here, but apart from a blocker, I'd go for the bold idea of letting thing break and get updated incrementally until we get a better .net.

My estimate is that 99% of the time, upgrading a library or an application to the new version will be as easy as enabling the flag (as opposed to the nullable story where you had to break many things). I'd say that most .net devs don't care about how a string is represented in memory, the same way they don't need to worry about the size of an int and other internal implementations, until they reach the limit or work with low-level code, as you probably do on a daily basis.

[tannergooding]

agnostic assemblies that do dynamic checks for sizeof(char) as necessary.

How would this handle the user string table (that is C# string literals)? Today, it is UTF16 encoded. I would have assumed that if we switched to UTF8, then it would become UTF8 encoded rather than carrying both or having a conversion cost on startup by the runtime.

[jkotas]

How would this handle the user string table (that is C# string literals)? Today, it is UTF16 encoded.

I think it would stay UTF16 encoded. Utf8 optimized runtime mode would pay for the conversion, but that should not be a problem. For JITed cases, the conversion cost is miniscule compared to cost of JITing and type loading. For AOT cases, we can store the converted string in the AOT binary if it makes a difference. Also, keeping the user string blob UTF16 encoded allows the agnostic binaries to work with existing tools and on older runtime in the "I know what I doing mode.".

There are number of encoding inefficiencies in the IL format. If we believe that it is important to do something about them and rev the format, it should be a separate exercise.

[jaredpar]

Bit of a long thread so is there a quick summary of why we feel like we need char to change to be one byte in length vs. the idea of exposing byte based operations on string?

My mental model of this approach, based on discussions we had back in the Midori days, is that the path forward here would be to change string to define its operations in terms of byte not char. Then we effectively deprecate char at that point and all the methods on string that expose char. Essentially let it become an artifact of history and move on with byte as the new standard.

Is the issue that we feel like we'd need to update to much code that is already written in terms of char today?

Let's leave issues of how C# uses string in the new world aside. If we make the jump at the runtime level to have utf8 be an opt-in story then we can likely do the same for the compiler. Imagine at a high level a /utf8string option where we effectively ban / deprecate all the old members on string and move to the new utf8 ones. Essentially foreach suddenly becomes byte based, not char based.

[ceztko]

Bit of a long thread so is there a quick summary of why we feel like we need char to change to be one byte in length vs. the idea of exposing byte based operations on string?

I think the whole discussion, especially since @jkotas intervened, moved to discuss a lower level approach that is not aiming to solve the problem through an additive API, but through changing the lower lever storage for the regular string type. This has the advantage of not jeopardizing the .NET API surface, which is remaining the same, covers interop scenarios and also should possibly introduce performance improvements in common workloads. The cost for this solution is handling ABI incompatibilities, and some situations where size/indexing of the storage array matters. I don't know if MS runtime team is going to have a resolution of this soon: they may stand by and wait for some experimenting on the lower level approach before taking a decision in which approach to follow. If MS is offering a remote short term contract to work on this I would love to apply :)

[jkotas]

Is the issue that we feel like we'd need to update to much code that is already written in terms of char today?

Yes, it is the crux of the problem. We have a lot of APIs and code written in term of string and char today. We are trying to find the best way to move the code to Utf8, while maximizing the performance benefits of Utf8 and minimizing additional cognitive load of the platform.

the path forward here would be to change string to define its operations in terms of byte not char.

I agree that this design would be an option. I think the downsite of this approch is that we would need to add byte clones of (at least some) APIs that take char, ReadOnlySpan<char> or Span<char>; and in turn change all code using these APIs to use the byte version instead when running in the Utf8 optimized runtime mode.

Then we effectively deprecate char at that point

I do not think that we would be ever able to deprecate char. There will always be a lot of components that just do not care about the performance benefit of Utf8, no matter how we expose it, and we need to make sure that they will continue to work. Any solution in this space needs to be designed as opt-in. The key question for me is whether it is better to opt-in via adding UTF8 clones of many APIs; or whether it is better to opt-in via compiler/runtime mode. I am leaning towards the latter as you can tell.

[mconnew]

Could this be solved with some JIT level marshalling? Whenever we make a native call which involves a string, we have to marshal the string with a full copy anyway. Could the JIT be used so that it ensures that a string passed to a method in a "legacy" assembly is converted to a 16-bit per character string and any callbacks to a newer assembly is converted to utf8. A mechanism could be added to treat an assembly as utf8 safe in the cases where an unmaintained assembly only uses string in a way which is safe (e.g. passes it through to a .NET assembly such as calling a File api with the string). This would ensure everything is safe, although with a potential performance cost. It would be up to the application developer to decide that their own code is safe and that the performance hit is worth it. Some helper classes could help alleviate a lot of the performance issues in targeted places, e.g. a class for newer apps to use that represents a cached UTF16 converted string which marshalling code recognizes and can use for the cached utf16 representation for hot code paths with a lot of reuse.

[jkotas]

Could this be solved with some JIT level marshalling?

I have commented on it in https://github.com/dotnet/corefxlab/issues/2350#issuecomment-761688346 . I do not think it is feasible to solve it via JIT level marshaling. I believe the marshaling would be too expensive and it would have to be done for too many situations.

[Serentty]

What if from the runtime's perspective, the UTF-8 string type were a completely separate type, and the choice between encodings would lie not with the CLR, but with the compiler? For example, the string keyword in C# would instead refer to Utf8String, string literals would default to UTF-8, and so on. Assemblies using either encoding could still communicate, as it would just be a matter of defaults. I know this goes against Roslyn's policy of not having “dialects” of C#, but it seems simpler than a change like this, which would probably create a rift that would last for years, if it ever closes at all.

[jeremyVignelles]

@Serentty I don't know how it works internally, but I'm under the impression that the runtime is a very small part to change compared to the BCL, which would need to be recompiled and re-checked, because there are assumptions that char represents an UTF-16 code point. (16bytes)

As jaredpar objected, that would create a branch in the ecosystem, and you would need to choose one or the other of the versions of .net. If you need an old library that won't get compiled for the new .net, but have other "new libraries" that use the new string representation, you'll get stuck.

jkotas' proposal seems like a more reasonable approach, where the BCL would become "string representation agnostic" over time, and encourage other libraries to do the same. you would only be able to "upgrade" if all your libs are agnostic, but with a less performant but still working fallback.

[whoisj]

@mconnew what do you think about a sizeof(char) == 4 setup?

Asking this specifically because only a 4-byte WORD has sufficient space to fit any Unicode value. Of course, we'd have to give up on constant time access to string[int] { get -> char } indexers, unless they started returning byte or sbyte values; in which case, we're back to the same problem of "I expect string indexer to return a 'char' which is short for 'character'". Honestly, there's no real winning in this situation.

[jeremyVignelles]

@whoisj That would waste a lot of memory for most of the characters, and would require a conversion from UTF8 and UTF16 from everywhere outside the .net world.

What you're suggesting is basically an UTF32 encoding, but utf8 is the most compatible with other pieces of software in my opinion

[mconnew]

@whoisj, in addition to what Jeremy said, it still wouldn't be sufficient. Unicode can use composed characters. For example é can be represented by the two character pair of the letter e (U+0065) and the acute accent (U+0301). If using the composed (as opposed to the single (U+00E9) form) version of the character, you still need to use two char's to represent this. This is a simple version, but things like emoji's use composition. For example, all the people and hand emoji's where you can choose the skin tone are a composed character with a skin tone specifying character followed by the image representing character. There is no single char representation.
The char data type represents a UTF code point. The Unicode table has 1,114,112 entries which can take up to 4 bytes to represent. So UTF-8 and UTF-16 have encoding mechanisms to specify there's a following value to complete the Unicode table entry (and UTF-32 has some leading zero's as 4 bytes is bigger than needed). So you have 3 different entities here. A UTF value which could represent a code point, or be part of a multi-value code point. A code point, which is made up of 1 or more UTF values. And a Rune, which represents a single thing you would see on your screen.
The problem trying to be solved here is that most of the internet uses UTF-8, but .NET uses UTF-16. This means when reading from the internet, everything needs to be converted from UTF-8 to UTF-16. This isn't just a matter of copying the values to every over byte. A two value UTF-8 code point will need to be converted to it's UTF-16 representation, which will include some bit shifting. A 3 value UTF-8 code point will need to be converted to 2 UTF-16 values. And then when displaying on the screen, you might have to combine multiple UTF code points into a single on screen character (or Rune).
My preference would be to make Rune[] or Span be the primitive any new api's use. When reading or writing to a stream, you want to do it one Rune at a time. With ASCII text, that would be one byte, but with other characters that could be 8 or more bytes. I don't really care how a character is represented under the hood. There's only a few things I care about.

1. Easy consumption and writing of a series of Rune's from a byte stream, which is basically a string.
2. Easy buffer math, i.e. do I have enough space left in my buffer to write the next Rune.
3. Easy manipulation of a series of Rune's, i.e. Find/Replace, ToUpper/ToLower, concatenation, substring etc

I wonder how much mileage you could get out of implicit conversion operators to convert ReadOnlySpan\

Char is the least useful representation of Unicode. It's neither the raw representation of what gets sent over a stream (file or network), nor a representation of single entities that you see on the screen. It's a halfway representation which has no useful purpose in isolation, unlike byte[] or Rune[].

[jeremyVignelles]

Iterating over bytes is a O(1) operation, iterating over UTF-8 values is doable in a reasonable time, but I don't know what it takes to iterate over runes? How do one know a composition is possible? does it need to use a lookup table?
If so, that wouldn't seem reasonable to iterate rune by rune to write in a stream performance-wise. For such cases, as byte[] copy should be preferred.

However, i'm OK for the inclusion of a .AsRuneSpan() or being able to cast to an explicit IEnumerable<Rune> implementation

[LokiMidgard]

I think its a property of the code point. Similar to upper and lower case.

If so, that wouldn't seem reasonable to iterate rune by rune to write in a stream performance-wise. For such cases, as byte[] copy should be preferred.

I'm not sure someone would iterate over a UTF-8 string to write every rune to a stream. I think I'd never seen someone iterate over a string and write single char's to a stream. After all a stream does not take Runes as input but bytes.

[GrabYourPitchforks]

I think I'd never seen someone iterate over a string and write single char's to a stream.

This would be lossy. As @mconnew mentioned earlier, sometimes you need to look at a _pair_ of chars in a sequence in order to generate the correct UTF-8 output. If you're operating on chars in isolation rather than keeping some kind of state, you could lose data. The System.Text.Encoding.GetEncoder API is a stateful conversion class meant to help with this scenario. (If you're working on the Rune level, you don't need to worry about lossy conversions, as each Rune is absolutely guaranteed to be a standalone well-formed Unicode scalar value.)

As Jan said, we're looking into whether it would make sense to back string with UTF-8 instead of UTF-16 as is done today. This would have some interesting behavioral and runtime consequences, and it'd result in a trade-off where memory utilization might go down, but CPU utilization might go up. In other words, common operations like string.IndexOf, string.Equals, and string.ToUpper might become slower, but each string instance would be smaller. Still need to weigh things here.

[mconnew]

I think I'd never seen someone iterate over a string and write single char's to a stream. After all a stream does not take Runes as input but bytes.

Have a look at how many classes use one of the overloads of Encoding.GetCharCount or Encoding.GetByteCount to see how many places care about how many bytes a char[] needs to write to a buffer, or how big a char[] is needed to store the sequence of bytes. Here's some code which writes a char[] to a buffer. If iterates through the char[] looking at each char in turn to see if it can be encoded in a single byte or if it needs multiple bytes. It then copies the value one at a time in that loop. If it finds a multiple byte char, it then searches for multiple in a row and then handles all of the consecutive ones using Encoding.GetBytes().
You just haven't needed to write any code which does the actual conversion between UTF-16 and UTF-8. Anyone writing a library which needs to be able to write UTF-8 from a char[] has had to worry about this if they are performance sensitive. If you write apps which are used by people using languages which use multi-byte characters such as Japanese, there's a good chance you've had to fix bugs in code where single character bytes were incorrectly presumed. English isn't the only language in existence. There's a reasonable argument to be made that UTF-16 should remain because switching to UTF-8 is optimizing for English and other languages frequently need 2 bytes so are easier to work with in memory that UTF-8. For example, switching string to use UTF-8 will cause many to have to switch from string.IndexOf(char value) to string.IndexOf(string value) because what they are searching for no longer fits in a single char.

[whoisj]

@mconnew , I think you're missing the point.

There's no reason a string type needs to be a fancy wrapper around char[]. Additionally, the char type is misleading because most of the world has agreed on Unicode and Unicode defines characters using 20 bits of information.

My suggestion was sizeof(char) == 4 (effectively UTF-32), but I was advocating for a string type backed by UTF-8. A UTF-8 backed string type wastes less memory than the current UTF-16 (UCS-2 on Windows) variant does. I was also suggesting that the indexer should return byte to enable quick indexing/parsing of values, but it should also provide a method (.CharAt(index)) which returns a 32-bit char value type.

Seems like I might also need to remind some of you that there are platforms besides Windows, and those platforms forcing all string data into UTF-16 requires a re-encoding of that data and an increase in its memory foot print.

[ceztko]

NOTE: Sorry for a wrong comment with no content from me, if you received it.

@whoisj

My suggestion was sizeof(char) == 4 (effectively UTF-32), but I was advocating for a string type backed by UTF-8. [...] but it should also provide a method (.CharAt(index)) which returns a 32-bit char value type.

This proposal has for sure some drawbacks, for example all char[] arrays used so far suddenly doubling the storage space. Also the string class already has a char indexer[1], which would become O(n) complexity suddenly from O(1). You'd really want to introduce a brand new type to define an Unicode code point, IMO.

[1] https://docs.microsoft.com/en-us/dotnet/api/system.string.chars?view=net-5.0#System_String_Chars_System_Int32_

[whoisj]

@ceztko

the string class already has a char indexer[1], which would become O(n) complexity suddenly from O(1). You'd really want to introduce a brand new type to define an Unicode code point, IMO.

I'm implemented utf-8 based string types in C++ and C# many times. In every case the optimal path is to keep the indexer returning the internal type value.

example (in the case where the underlying data type is byte[] for string):
```C#
public byte this[int index] { get; }

Example usage would be seeking the next new line character, which absolutely doesn't require a character-by-character search, but merely a byte-by-byte seek.

int index = myString.IndexOf('\n');
```

Even when looking for seeking for a character like '美', one can compose the utf-32 value into a utf-8 encoded value using the same 32-bit value space, and seek for the first series of bytes that match. Via unsafe, this can be amazingly quick.

In 99% of string parsing cases (the most likely reason code is using the indexer), reading a byte from the string is more than sufficient and there's generally no reason to read the entire Unicode value.

When code needs to pull each Unicode character out of a string, then this can be expensive and should be done via a utility like an enumerator. In which case, the enumerator handles the character composition for the caller.

[danmoseley]

@jkotas shall we transfer this to runtime or runtimelab as we are archiving this repo?

[jkotas]

The discussion in this issue is too long and github has troubles rendering it.

I think we should close this issue and start a new one in dotnet/runtime.