Huginn: Uneven match error in Website Agent

[HyShai]

The uneven match error really causes issues when the data has 1 or more matches. E.g. trying to extract from this xml:

<item>
  <link>http://example.com</link>
  <enclosure url="http://example.com/foo.mp3" length="9032127" type="audio/mpeg"/>
  <enclosure url="http://example.com/bar.mp3" length="9032131" type="audio/mpeg"/>
</item>

using xpath 1.0 there doesn't seem to be a way to extract both enclosures.

Is it possible to instead do a check inside the loop for nil? e.g.

# I don't know if this is valid ruby....but
# get the largest unique length
num_unique_lengths.max.times do |index|
  result = {}
  interpolated['extract'].keys.each do |name|

    #pseudo code - if that index doesn't exist then just set to empty string
    # or perhaps set that value to the whatever value is in the `0` index
    result[name] = output[name][index] || ""
    if name.to_s == 'url'
      result[name] = (response.env[:url] + result[name]).to_s
    end
  end

....

end

[cantino]

What selector are you trying to use @HyShai?

[HyShai]

@cantino doing something like this:

  "extract": {
    "url": {
      "xpath": "//*/item/link",
      "value": ".//text()"
    },
    "mp3": {
      "xpath": "//*/item/enclosure",
      "value": "@url"
    }
  }

mp3 returns multiple results but url only returns one result. In such a case, ideally I would like Huginn to not throw an error, instead it would create either a) multiple events or b) an array of values for mp3.

This has been a common issue for me when using the Website Agent to extract information from a source that can have a 1 to many relationship. E.g. 1 webpage (url) has multiple images (img).

[cantino]

@HyShai: I agree. I think we should have a "group" option or something that could combine the multiple mp3 values into an array.

[knu]

I don't think we need a new feature. If you want to create an event per enclosure, just do so:

  "url": {
      "xpath": "//*/item/enclosure",
      "value": "./parent::li/link/text()"
    },

[knu]

I didn't read the follow-ups carefully. Yeah, currently there is no way to emit an array value for a key in an event.

[level20peon]

Try it this way: https://github.com/cantino/huginn/issues/1033#issuecomment-140639795

[HyShai]

@level20peon unfortunately that workaround doesn't work in a scenario where there is an arbitrary/unknown number of elements to be matched (the EventFormatting Agent doesn't use String#scan so it would only return the first match.)

I still think we need some way to allow the WebsiteAgent to extract uneven matches. This has been a real annoyance for me :(

[Toucan-Sam]

Has anyone been able to figure this out?

[dsander]

@Toucan-Sam If there is only one item using array: true will work.

  "extract": {
    "url": {
      "xpath": "//*/item/link",
      "value": ".//text()"
    },
    "mp3": {
      "xpath": "//*/item/enclosure",
      "value": "@url",
      "array": "true",
    }
  }