Huginn: Adding "next" page navigation to WebsiteAgent

[knu]

I've been thinking of adding a key named "next" to WebsiteAgent so you can tell an agent to navigate to the next page(s) if existent.

For instance, you could specify the following:

"next": {
  "css": "a[rel~='next']",
  "value": "@href"
}

And if the fetched page had a link with a rel attribute value including "next", then the agent would navigate to the link URL. This would be repeated until there was no "next" link.

There should also be a way to stop an iteration, like when it reaches the point where it has already seen, or when it is to emit an event that is identical to any existing (recent) event. I'm yet to decide how to let user specify the conditions. Should it provide a way to save and access memory for ultimate programmability or is that an overkill?

I'll soon be working on it, so please share your ideas if you are interested.

[cantino]

Cool! Would just a selector and a max paginations be enough, when combined
with the normal event deduplication?

[cantino]

Still want this @knu?

[knu]

I have a local WIP branch for this but I am still unsure as to how the configuration options should be like. I'll come up with a PR when I get the time to make up my mind and complete the work!

[cantino]

@santoshgistto requested this too. Maybe we should re-open?

[somepaulo]

Pagination would definitely be a nice feature to have. As well as deep fetching. Say you parse a blog with excerpts, but you want full text - there's a case for both features. You want to be able to go through paged lists of posts getting urls, titles, images and excerpts, and then go into each listed post and grab its contents.
Currently, getting full content in a scenario like this is possible by adding a second agent, feeding it links from the first one, and merging its output. But I don't know how to achieve pagination.

[santoshgistto]

how about adding the menu link also ?

[knu]

This is now kind of feasible by combining the features I've added in the past half year or so, namely #1769, #1772, and #1816.

Here's the steps to make a site scraper with paging support:

1. Create a WebsiteAgent that scrapes a site and emits items with a URL to the next page.
2. Create a DeDuplicationAgent that receives events from the WebsiteAgent to dedupe by some unique key (like the URL) of each item.
3. Create a TiggerAgent that receives events from the DeDuplicationAgent and feeds the URL of the next page to the first agent only if a received event is for an item at the bottom of the page, and the page number is not too big (to avoid excessive paging).

Then you can listen on the second agent for new items.

Below is a sample scenario for scraping questions from Stackoverflow. Beware, it is going to be a high volume feed if you actually run it.

{
  "schema_version": 1,
  "name": "Stackoverflow Questions scraper",
  "description": "This is meant to be a scenario for scraping items from a paginated list site.",
  "source_url": false,
  "guid": "4ab16ebeeb041eafac431adf3cdf0db2",
  "tag_fg_color": "#ffffff",
  "tag_bg_color": "#5bc0de",
  "icon": "globe",
  "exported_at": "2017-05-19T09:56:12Z",
  "agents": [
    {
      "type": "Agents::WebsiteAgent",
      "name": "Stackoverflow Questions scraper",
      "disabled": false,
      "guid": "1ba1cdf342a02a527494f9f943cec867",
      "options": {
        "expected_update_period_in_days": "2",
        "url": "http://stackoverflow.com/questions",
        "url_from_event": "{{ next_url }}",
        "type": "html",
        "mode": "on_change",
        "include_sort_info": true,
        "extract": {
          "url": {
            "css": ".question-summary .summary h3 a.question-hyperlink",
            "value": "@href"
          },
          "title": {
            "css": ".question-summary .summary h3 a.question-hyperlink",
            "value": "normalize-space(.)"
          },
          "excerpt": {
            "css": ".question-summary .summary .excerpt",
            "value": "normalize-space(.)"
          },
          "taglist": {
            "css": ".question-summary .summary .tags",
            "value": "normalize-space(.)"
          },
          "page": {
            "css": ".pager .page-numbers.current",
            "value": "string(.)",
            "repeat": true
          },
          "next_url": {
            "css": ".pager .page-numbers.current + a[href]",
            "value": "@href",
            "repeat": true
          }
        },
        "template": {
          "url": "{{ url | to_uri: _response_.url }}",
          "next_url": "{{ next_url | to_uri: _response_.url}}"
        },
        "events_order": [
          [
            "{{_index_}}",
            "number",
            true
          ]
        ]
      },
      "schedule": "never",
      "keep_events_for": 3600,
      "propagate_immediately": true
    },
    {
      "type": "Agents::DeDuplicationAgent",
      "name": "Stackoverflow Questions",
      "disabled": false,
      "guid": "6e24d30164239f2d693d553700fa1ba7",
      "options": {
        "property": "{{url}}",
        "lookback": "100",
        "expected_update_period_in_days": "1"
      },
      "keep_events_for": 0,
      "propagate_immediately": true
    },
    {
      "type": "Agents::TriggerAgent",
      "name": "Stackoverflow Questions next page",
      "disabled": false,
      "guid": "1c8ecf437b9449fd882bfd85be8364f4",
      "options": {
        "keep_event": "true",
        "rules": [
          {
            "type": "field<=value",
            "path": "page",
            "value": "3"
          },
          {
            "type": "field==value",
            "path": "sort_info.position",
            "value": "1"
          },
          {
            "type": "field>=value",
            "path": "sort_info.count",
            "value": "15"
          }
        ],
        "expected_receive_period_in_days": "2"
      },
      "keep_events_for": 3600,
      "propagate_immediately": true
    }
  ],
  "links": [
    {
      "source": 0,
      "receiver": 1
    },
    {
      "source": 1,
      "receiver": 2
    },
    {
      "source": 2,
      "receiver": 0
    }
  ],
  "control_links": [

  ]
}

[laginha87]

Hi, should this issue still be open? is the solution posted by @knu enough?

[knu]

@laginha87 To me, yes!