Nokogiri: XML parsing fails at vertical tab characters in attributes

[logiclrd]

Nokogiri has difficulties if the XML input contains vertical tab characters ("\v") within attribute values. It appears to abort element parsing and treat all following text, up to the next element start, as a text node.

The following script reproduces this:

require 'nokogiri'
require 'pp'

print Nokogiri::VersionInfo.new.to_markdown
print "\n"

def test(input)
  print "\n"
  print "input: "
  pp input

  print "output: "
  pp Nokogiri::XML(input).to_xml
end

test "<test attr=\"before\vafter\" />"
test "<test attr=\"before\vafter\" attr2=\"second attribute\" />"
test "<root><test attr=\"before\vafter\" attr2=\"second attribute\" /></root>"
test "<root><test attr=\"before\vafter\" attr2=\"second attribute\" /><test2 attr=\"value\" /></root>"

The output from this script on my system is as follows:

# Nokogiri (1.7.0.1)
    ---
    warnings: []
    nokogiri: 1.7.0.1
    ruby:
      version: 2.2.3
      platform: i386-mingw32
      description: ruby 2.2.3p173 (2015-08-18 revision 51636) [i386-mingw32]
      engine: ruby
    libxml:
      binding: extension
      source: packaged
      libxml2_path: "/home/flavorjones/code/oss/nokogiri/ports/i686-w64-mingw32/libxml2/2.9.4"
      libxslt_path: "/home/flavorjones/code/oss/nokogiri/ports/i686-w64-mingw32/libxslt/1.1.29"
      libxml2_patches: []
      libxslt_patches: []
      compiled: 2.9.4
      loaded: 2.9.4


input: "<test attr=\"before\vafter\" />"
output: "<?xml version=\"1.0\"?>\n<test attr=\"before\"/>\n"

input: "<test attr=\"before\vafter\" attr2=\"second attribute\" />"
output: "<?xml version=\"1.0\"?>\n<test attr=\"before\"/>\n"

input: "<root><test attr=\"before\vafter\" attr2=\"second attribute\" /></root>"
output: "<?xml version=\"1.0\"?>\n<root><test attr=\"before\"/>after\" attr2=\"second attribute\" /&gt;</root>\n"

input: "<root><test attr=\"before\vafter\" attr2=\"second attribute\" /><test2 attr=\"value\" /></root>"
output: "<?xml version=\"1.0\"?>\n<root><test attr=\"before\"/>after\" attr2=\"second attribute\" /&gt;<test2 attr=\"value\"/></root>\n"

Note that when the element being parsed is the root element, the text node that Nokogiri is interpreting as being after the end of the root element disappears. I assume this is because there is nowhere to attach it to the DOM, since it isn't contained within the document element. When there is a place to hook it up, that content becomes text following the element that originally contained it. When another element is encountered, the parser recovers and parses it correctly as another element.

The correct behaviour in this instance seems to be governed by sections 2.3 and 3.3.3 from the XML specification, namely this grammar rule under "Literals" in section 2.3 "Common Syntactic Constructs":

[10] AttValue ::= '"' ([^<&"] | Reference)* '"'
                | "'" ([^<&'] | Reference)* "'"

..and this section describing the interpretation of the characters captured by the grammar rule:

3.3.3 Attribute-Value Normalization

Before the value of an attribute is passed to the application or checked for validity, the XML processor must normalize the attribute value by applying the algorithm below, or by using some other method such that the value passed to the application is the same as that produced by the algorithm.

1. All line breaks must have been normalized on input to #xA as described in 2.11 End-of-Line Handling, so the rest of this algorithm operates on text normalized in this way.

2. Begin with a normalized value consisting of the empty string.

3. For each character, entity reference, or character reference in the unnormalized attribute value, beginning with the first and continuing to the last, do the following:

   • For a character reference, append the referenced character to the normalized value.

   • For an entity reference, recursively apply step 3 of this algorithm to the replacement text of the entity.

   • For a white space character (#x20, #xD, #xA, #x9), append a space character (#x20) to the normalized value.

   • For another character, append the character to the normalized value.

The vertical tab character, #xB, receives no special attention in either of these; the character class rule that defines the content of AttValue includes any character other than left angle bracket, ampersand and whichever delimeter character is used in the attribute (' or "), and then characters with value 11 should pass through the "Attribute-Value Normalization" algorithm unchanged, as they would match the final point ("For another character, ...").

[flavorjones]

Hi! Thanks for asking this question.

When Nokogiri parsing isn't doing what I expect, I always remember to look at the document errors. When I do this in your example:

def test(input)
  print "\n"
  print "input: "
  puts input.inspect

  print "output: "
  doc = Nokogiri::XML(input)
  puts doc.to_xml.inspect
  puts doc.errors
end

I see some enlightening errors:

invalid character in attribute value
attributes construct error
Couldn't find end of Start Tag test line 1
Extra content at the end of the document

Notably, the first two errors tell us that libxml2 thinks "\v" is an invalid character in an attribute value. I can't find exactly where in the XML spec this is stated (because STANDARDIZATION COMMITTEES, am I right?), but the libxml2 code that's parsing attributes (and generating this error) is here:

https://github.com/GNOME/libxml2/blob/master/parser.c#L4217-L4218

Interestingly, Xerces (the parser Nokogiri uses when running under JRuby) also thinks this is an invalid character, though it behaves slightly differently in how it recovers from this error.

In any case, my point is that Nokogiri's not actually responsible for parsing XML -- Nokogiri wraps libxml2 or xerces in a loving embrace, and has them do the parsing. You may want to ask upstream why this behavior is the way it is.

Sorry I couldn't be more helpful here.

[flavorjones]

Ah, actually I found where in the spec this is stated. The section you refer to above is "Normalization", which happens before "Validation".

The "Validation" part for an attribute is explained, in a recursive style, starting here:

https://www.w3.org/TR/xml11/#sec-starttags

the key bits being AttValue → Reference → CharRef → EntityRef → Name.

Name is made up of NameStartChar followed by NameChars:

NameStartChar      ::=      ":" | [A-Z] | "_" | [a-z] | [#xC0-#xD6] | [#xD8-#xF6] | [#xF8-#x2FF] | [#x370-#x37D] | [#x37F-#x1FFF] | [#x200C-#x200D] | [#x2070-#x218F] | [#x2C00-#x2FEF] | [#x3001-#xD7FF] | [#xF900-#xFDCF] | [#xFDF0-#xFFFD] | [#x10000-#xEFFFF]
NameChar       ::=      NameStartChar | "-" | "." | [0-9] | #xB7 | [#x0300-#x036F] | [#x203F-#x2040]

and you can see through careful study, reflection, and meditation that #xB is not in the set of valid NameChars.

Remember, kids: only YOU can prevent ridiculous standardization committee documents.

[logiclrd]

Ah, I wasn't aware of the underlying parser of Nokogiri. I may pursue this upstream. :-)

I don't believe the series of productions you've listed actually show "\v" being an invalid character for AttValue, because AttValue also has a production directly to the character class [^<&"] (or [^<&'] when the delimeter is a single quote). This will obviously accept any character other than the delimeter, open angle bracket and ampersand. Ampersand is excluded here, of course, because it is the character that introduces a Reference -- the other fork in the production. The name that forms an entity reference (the characters after the '&') must be a valid name, but attribute values themselves do not need to be made up only of valid XML names.

In any case, since it is Xerces rejecting the character and not Nokogiri itself, clearly that is where I need to pursue this. :-)

Thanks for your reply!

[logiclrd]

Ahh, I have uncovered why the character is actually rejected. I'm still not convinced it's a correct interpretation of the specification, but I think I see how they might have come to that decision. I'm including it here for the interest of anyone who reads the preceding discussion.

My investigation proceeded with reproducing the issue with libxml2, and libxml2 output the following errors during the parsing phase:

Entity: line 1: parser error : invalid character in attribute value
<root><test attr="before
after" attr2="value" /><test2 attr="value" /></root>
                        ^
Entity: line 1: parser error : attributes construct error
<root><test attr="before
after" attr2="value" /><test2 attr="value" /></root>
                        ^
Entity: line 1: parser error : Couldn't find end of Start Tag test line 1
<root><test attr="before
after" attr2="value" /><test2 attr="value" /></root>
                        ^
Entity: line 1: parser error : PCDATA invalid Char value 11
<root><test attr="before
after" attr2="value" /><test2 attr="value" /></root>
                        ^

This gave me a key word to search for in the XML specification: PCDATA. This eventually lead me to the following sequence of declarations:

1. 3.3.1 AttributeTypes

In this section, it is indicated that the value of an attribute can have one of a few different types. In particular, this construction is given:

[54]    AttType    ::=      StringType | TokenizedType | EnumeratedType
[55]    StringType     ::=      'CDATA'
[56]    TokenizedType      ::=  (an alternation of a variety of tokens)

I don't see anywhere that explicitly states this, but it seems to be implied that user attributes that aren't defined in the DTD are automatically StringType.

Note that this doesn't state that the attribute value is defined by the same part of the specification that governs <![CDATA[...]]> constructs. Rather, it states that when you are declaring an attribute, one option for the attribute's type is the literal string CDATA. This is where I believe the underlying parser is misinterpreting the spec.

But, continuing on...

1. 2.7 CDATA Sections

Here, the <![CDATA[..]]> section is laid out:

[18]    CDSect     ::=      CDStart CData CDEnd
[19]    CDStart    ::=      '<![CDATA['
[20]    CData      ::=      (Char* - (Char* ']]>' Char*))
[21]    CDEnd      ::=      ']]>'

Now, production [10] earlier showed an explicit definition for AttValue, and that definition did not reference the CData symbol. Because of following this branch, which I don't think is actually a correct interpretation of the XML specification, the underlying parser is in error: It is using CData instead of the character class [^<&"] to parse the attribute value data.

But, continuing on...

1. 2.2 Characters

The CData symbol is defined in terms of Char, and Char is defined here as:

[2]     Char       ::=      #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]  /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */

This is where control characters other than \t, \n and \r are excluded. However, per my reading of the spec, this only applies to the contents of <![CDATA[..]]> sections (and other parts of the XML spec that depend on Char). As AttValue is not defined in terms of CData or Char, I believe character \xB should be permitted in attribute values.

I'll pass this information upstream, and as I mentioned earlier I am including it in a comment here purely for the curious reader. This is not a bug in Nokogiri itself per se. :-)

[logiclrd]

Okay, final comment: I found the piece that clinches it. The character productions in the BNF form are not standard character sets. The notation [^<&"] does not mean "any Unicode code point other than <, & or "". The specification redefines character classes:

6 Notation

The formal grammar of XML is given in this specification using a simple Extended Backus-Naur Form (EBNF) notation. Each rule in the grammar defines one symbol, in the form

symbol ::= expression
[..]
Within the expression on the right-hand side of a rule, the following expressions are used to match strings of one or more characters:
[..]
[^a-z], [^#xN-#xN]
matches any Char with a value outside the range indicated.

Note this last line: the character set is intersected with that of the Char production. So, legitimately, vertical formfeed characters are excluded from attribute values (and every other point in an XML document) by the specification.

The more you know! :-)

[flavorjones]

Thanks for following up!