Rapidjson: Difference in parse and write validation

[lklein53]

Hello,

I encountered some differences between the kParseValidateEncodingFlag and the kWriteValidateEncodingFlag with regrads to surrogates.

In the following example:

int main () {
   const auto input = R"EOS(["\udc4d"])EOS";
   rapidjson::Document doc;
   rapidjson::ParseResult parseResult = doc.Parse<rapidjson::kParseValidateEncodingFlag>(input);
   if(parseResult.IsError()) {
     throw std::runtime_error("Parse Error");
   }   
   OStreamWrapper wrapper{std::cout};
   auto writer = Writer<rapidjson::OStreamWrapper, rapidjson::UTF8<>, rapidjson::UTF8<>, rapidjson::CrtAllocator, rapidjson::kWriteValidateEncodingFlag>(wrapper);
   if(!doc.Accept(writer)) {
     throw std::runtime_error("Write Error");
   }   
}

the input (an array with a single low surrogate) is parsed without an error, but later when writing the document to the stream an error is thrown.
is this a known issue or am I missing something in the example ?
If this is an issue and should be fixed I could provide a pull request that modifies the handling of surrogates in reader.h

Thanks for your work and kind regards

[miloyip]

I think kParseValidateEncodingFlag is not related in this problem, and can be omitted to reproduce this situation.
I checked the currently Reader handled unpaired surrogate with single high surrogate, but not single low surrogate:

https://github.com/Tencent/rapidjson/blob/88bd956d66d348f478bceebfdadb8e26c6844695/include/rapidjson/reader.h#L1026-L1035

I think it should generate kParseErrorStringUnicodeSurrogateInvalid as well for leading low surrogate.
I have not research much on this. May need to dig more on the standards and try on other implementations.
Open for discussion.

[lklein53]

Hi,

thanks for your reply. I think having a single low surrogate is invalid (Source: https://unicode.org/faq/utf_bom.html#utf8-4):

A different issue arises if an unpaired surrogate is encountered when converting ill-formed UTF-16 data. By representing such an unpaired surrogate on its own as a 3-byte sequence, the resulting UTF-8 data stream would become ill-formed. While it faithfully reflects the nature of the input, Unicode conformance requires that encoding form conversion always results in a valid data stream. Therefore a converter must treat this as an error. [AF]

I also did some research on the behavior of other JSON parsers:

• Boost JSON: Fails to parse the JSON with the following message Error illegal leading surrogate

• Nlohmann JSON: Fails to parse the JSON with the following message: syntax error while parsing value - invalid string: surrogate U+DC00..U+DFFF must follow U+D800..U+DBFF;

• SIMDJson: Parses the input and is able to print the JSON afterwards. Similar to rapidjson without the kWriteValidateEncodingFlag

• TaoJSON: Fails to parse the JSON with the following message: invalid escaped unicode code point

I created a merge request that changes the surrogate handling to also report kParseErrorStringUnicodeSurrogateInvalid for a single low surrogate (https://github.com/Tencent/rapidjson/pull/1744).