Lwjgl3: MemoryUtil.memUTF8 throws exception for trailling high surrogate

[Marcono1234]

Environment

• LWJGL version: 3.2.3
• LWJGL build #: 13 (?, Maven dependency)
• Java version: AdoptOpenJDK jdk-13.0.0.33
• Platform: Windows
• Module: core

Description

The MemoryUtil.memUTF8 methods throw an exception for trailing high surrogates. This could be "intended" given that some (but not all) of the methods calling encodeUTF8 require a valid string:

The specified text is assumed to be a valid UTF-16 string.

However, if this is not intended, maybe the trailing high surrogate could be encoded as regular 3 byte UTF-8 char.

Cause

https://github.com/LWJGL/lwjgl3/blob/35e5f80d7c5beb2a06af5fa0a29854a03c739b1f/modules/lwjgl/core/src/main/java/org/lwjgl/system/MemoryUtil.java#L2124

Assumes there is always a low surrogate after a high one.

Reproduction code

// Trailing high surrogate char
String str = Character.toString(Character.MIN_HIGH_SURROGATE);
MemoryUtil.memUTF8(str);

Throws StringIndexOutOfBoundsException.

[Spasi]

Hey @Marcono1234,

Yes, it is intended. I have added a note to the MemoryUtil class javadoc.

I did some work on it though and it was trivial to make LWJGL compatible with String.getBytes, for all kinds of malformed input. However, when a malformed/unmappable character sequence is detected, there's the question of how to handle it exactly:

1. ignore and skip?
2. replace with special byte sequence?
3. throw an exception?

String.getBytes does the second, but that may not be what the user wants. So, when dealing with trusted text, use the LWJGL codecs. When not, the JDK's CharsetEncoder/CharsetDecoder can be used instead, configured with the appropriate behavior on invalid input.

[Marcono1234]

@Spasi, thank you for the clarification.

I have sent an e-mail to the Gmail address you have set on GitHub a few weeks ago. Have you had a look at it already? Or is there another way to report issues which should potentially not be public?

[Spasi]

Reporting sensitive issues by e-mail is fine. See 272243c, I believe the issues you raised are addressed.