Nokogiri: SAX doesn't fire the `characters` method when a node is empty

[viprasant]

I'm trying to parse a large xml file using SAX parser. When the parser reaches a node which is empty, the characters method doesn't fire. Here is an example...

Here is the sample xml document.

<?xml version="1.0" encoding="UTF-8"?>
<root>
    <ISA type="array">
        <ISA>
            <I02>
                <name>Information1</name>
                <value>
                    <raw>00</raw>
                    <description></description>
                </value>
            </I02>
            <I02>
                <name>Information2</name>
                <value>
                    <raw></raw>
                    <description nil="true"/>
                </value>
            </I02>
        </ISA>
    </ISA>
</root>

Here is the sample script

require 'nokogiri'

class Parser < Nokogiri::XML::SAX::Document
  def initialize
    @count=1
  end
  def start_element(name, attrs = [])
    puts name
  end
  def characters(string)
    string.strip!
    puts "#{@count} #{string}"
    @count += 1
  end
  def end_element(name)
    puts name
  end
end

Nokogiri::XML::SAX::Parser.new(Parser.new).parse(File.open('sax_example3.xml'))

I had to use SAX because the file has 6.5 million lines.

What I'm trying to do is gather all the name values and then raw values into separate arrays and later I can zip both arrays to get key value pairs.

Am I approaching this the correct way? Is there any other way to do this?

Edit:

What I expected

array1 = ["Information1","Information2"]
array2 = ["00", ""]

All name values are assigned to array1 and raw values to array2 like shown above.

What I'm getting

array1 = ["Information1","Information2"]
array2 = ["00"]

array2 doesn't have the same number of elements as array1, which means there is no way to map names to array. The reason for this I think is that the characters method is not called if the node is empty.

Here is the output of the above program (Edited the above script and added line numbers)

root           
1              
ISA            
2              
ISA            
3              
I02            
4              
name           
5 Information1 
name           
6              
value          
7              
raw            
8 00           
raw            
9              
description    
description    
10             
value          
11             
I02            
12             
I02            
13             
name           
14 Information2
name           
15             
value          
16             
raw            
raw            
17             
description    
description    
18             
value          
19             
I02            
20             
ISA            
21             
ISA            
22             
root

As you can see between line 9 & 10, 16 & 17, 17 & 18 the start_element & end_element is executed but the characters method isn't.

[flavorjones]

@Vizkrig Thank you for submitting this! I'll try to take a look in the next day or so -- apologies for the slow response, it's been a nutty week.

[YegorZdanovich]

+1

[flavorjones]

Sorry for the slow response. I'm working through the v1.11.0 milestone so I may not get to it soon, unfortunately. If anyone wants to investigate I'd be very grateful.

[YegorZdanovich]

thanks for feedback. i see same issue using jruby.
maybe you can help me with starting point, what place should i check first for investigation?

[flavorjones]

@Vizkrig @YegorZdanovich got a few minutes to look into this this morning, and here's what I found.

__TL;DR__

The behavior you're describing, which is that characters callbacks aren't invoked for empty tags, is expected. Fortunately we can work around this using start_element and end_element.

__Behavior of the underlying library__

Nokogiri is a wrapper around a lower-level parser (libxml2 on CRuby, Xerces on JRuby). Let's step outside Nokogiri for a minute and check what libxml2's behavior is.

Helpfully, libxml2 provides a utility called testSAX that emits all the callbacks (here are some docs).

Here's the full output from testSAX when run against the XML provided by the OP:

SAX.setDocumentLocator()
SAX.startDocument()
SAX.startElement(root)
SAX.characters(
    , 2)
SAX.startElement(ISA, type='array')
SAX.characters(
        , 3)
SAX.startElement(ISA)
SAX.characters(
            , 4)
SAX.startElement(I02)
SAX.characters(
                , 5)
SAX.startElement(name)
SAX.characters(Information1, 12)
SAX.endElement(name)
SAX.characters(
                , 5)
SAX.startElement(value)
SAX.characters(
                    , 6)
SAX.startElement(raw)
SAX.characters(00, 2)
SAX.endElement(raw)
SAX.characters(
                    , 6)
SAX.startElement(description)
SAX.endElement(description)
SAX.characters(
                , 5)
SAX.endElement(value)
SAX.characters(
            , 4)
SAX.endElement(I02)
SAX.characters(
            , 4)
SAX.startElement(I02)
SAX.characters(
                , 5)
SAX.startElement(name)
SAX.characters(Information2, 12)
SAX.endElement(name)
SAX.characters(
                , 5)
SAX.startElement(value)
SAX.characters(
                    , 6)
SAX.startElement(raw)
SAX.endElement(raw)
SAX.characters(
                    , 6)
SAX.startElement(description, nil='true')
SAX.endElement(description)
SAX.characters(
                , 5)
SAX.endElement(value)
SAX.characters(
            , 4)
SAX.endElement(I02)
SAX.characters(
        , 3)
SAX.endElement(ISA)
SAX.characters(
    , 2)
SAX.endElement(ISA)
SAX.characters(
, 1)
SAX.endElement(root)
SAX.endDocument()

The important bits, which are the raw elements:

...
SAX.startElement(raw)
SAX.characters(00, 2)
SAX.endElement(raw)
...
SAX.startElement(raw)
SAX.endElement(raw)

You can hopefully see that libxml2's SAX parser itself doesn't invoke the characters callback for empty nodes. This intuitively makes sense to me, since there _aren't_ any characters.

__What does the API spec say?__

I then went to the SAX API docs and, although it doesn't explicitly say that "this callback won't be invoked if there isn't any character data", it seems pretty straightforward to me that the callback is intended to only be called when there is character data: http://www.saxproject.org/apidoc/org/xml/sax/ContentHandler.html#characters(char[],%20int,%20int)

__How can the OP solve the original problem?__

Something like this, where we use start_element and end_element to indicate presence of the <raw> tag, and either collect characters data if it's there, or else default to the empty string.

#! /usr/bin/env ruby

require "nokogiri"

xml = <<~EOF
  <?xml version="1.0" encoding="UTF-8"?>
  <root>
    <ISA type="array">
        <ISA>
            <I02>
                <name>Information1</name>
                <value>
                    <raw>00</raw>
                    <description></description>
                </value>
            </I02>
            <I02>
                <name>Information2</name>
                <value>
                    <raw></raw>
                    <description nil="true"/>
                </value>
            </I02>
        </ISA>
    </ISA>
  </root>
EOF

# parser to collect tuples of `name` and `raw` contents
class Parser < Nokogiri::XML::SAX::Document
  attr :tuples
  COLLECT_NOTHING = 1, COLLECT_NAME = 2, COLLECT_RAW = 3

  def initialize
    @tuples = []
    reset_state
  end

  def reset_state
    @state = COLLECT_NOTHING
    @current_name = ""
    @current_raw = ""
  end

  def start_element(name, attrs = [])
    @state = case name
      when "name" then COLLECT_NAME
      when "raw" then COLLECT_RAW
      else COLLECT_NOTHING
      end
  end

  def characters(string)
    case @state
    when COLLECT_NAME
      @current_name = string
    when COLLECT_RAW
      @current_raw = string
    end
  end

  def end_element(name)
    case name
    when "name"
      @state = COLLECT_NOTHING
    when "raw"
      @tuples << [@current_name, @current_raw]
      reset_state
    end
  end
end

p = Parser.new
Nokogiri::XML::SAX::Parser.new(p).parse(xml)
p.tuples # => [["Information1", "00"], ["Information2", ""]]

I hope this helps?

[YegorZdanovich]

@flavorjones thanks for quick response.