Peertube: Support for WebSub

[Alkarex]

Hello,
This is a suggestion to consider implementing WebSub / PubSubHubbub for the existing RSS/Atom feeds.

(Already mentioned in https://github.com/Chocobozzz/PeerTube/issues/493#issuecomment-386458200 )

---

Roadmap

• [x] launch a notification task every time the ressource is updated with new content/modifications
• [x] add the Hub to the feed (it's already supported by pfeed as a parameter but we might want to add multiple hub support)

[rigelk]

@Alkarex do you know of any potential clients that would effectively use our WebSub feeds? (apart from the internal notification system)

[Alkarex]

My use-case is for RSS/Atom feed readers. For instance, I am one of the developers of FreshRSS, but some larger actors include Feedly, The Old Reader, Inoreader, NewsBlur, IFTTT, etc.

P.S.: While RSS/Atom works fine with pull, WebSub / PubSubHubbub enables push, which is faster and often times more resource efficient. For instance, FreshRSS automatically reduces the pull frequency to at max one per day when a feed supports WebSub / PubSubHubbub.

[Alkarex]

Maybe @julien51 could have a tip there

[rigelk]

@Alkarex @julien51 do you think we should setup a WebSub hub ourselves, as common infrastructure for PeerTube instances?

[rigelk]

ping @Alkarex @julien51 - before I go forward with the PR, I need to know if pubsubhubbub.superfeedr.com is the only hub I should care about for know. Being new to WebSub, I don't see a lot of hubs and this one seems like the default to stand out.

In your experience, is it fine if we default to that hub, or should we use/setup another one?

[Alkarex]

Hello @rigelk
Here are some statistics from my FreshRSS instance:

$ find -name '!hub.json' | xargs jq -r .hub | sort | uniq -c | sort -nr
     38 http://pubsubhubbub.appspot.com/
      2 https://pubsubhubbub.appspot.com
      2 https://mamot.fr/api/push
      1 https://websub.rocks/blog/102/DZ0RbPbGlPdQVR6xPX8U/hub
      1 https://techcrunch.com/?pushpress=hub
      1 https://pubsubhubbub.superfeedr.com/
      1 https://kristinahobbs.com/?pushpress=hub
      1 https://humblr.social/api/push
      1 https://henningjust.wordpress.com/?pushpress=hub
      1 https://hackaday.com/?pushpress=hub
      1 https://f.a80.fr/pubsubhubbub/alkarex
      1 https://blog.ouseful.info/?pushpress=hub
      1 http://pubsubhubbub.superfeedr.com/
      1 http://medium.superfeedr.com/
      1 http://hackaday.com/?pushpress=hub
      1 http://feedpress.superfeedr.com/

When using a third-party hub, https://pubsubhubbub.appspot.com (Google) or some variants is the most used (especially because it is used by https://feedburner.google.com and other Google services), but https://pubsubhubbub.superfeedr.com is also a good option (used by e.g. Medium http://medium.superfeedr.com).

But instead of a third-party hub, some systems are using a built-in mechanism. This is the case for WordPress (when using a plug-in https://wordpress.org/plugins/pubsubhubbub/ or when hosted on WordPress.com), Friendica, GNU Social...

Using a third-party hub is easier (it is really just sending a very simple HTTP request when something has changed) but less decentralised and less powerful. In particular, it is not practical to use a third-party hub if you would like to support custom filters (e.g. having clients interested only in some specific news but not all).

Using a shared hub as a common infrastructure for PeerTube instances would be half way between those two approaches.

I can gladly help with testing / architecture / and other feedback.

For testing, see https://websub.rocks/publisher and https://websub.rocks/subscriber (the subscription part might not be relevant for you).

Here are some examples of feeds using WebSub with various mechanisms:

• https://push-pub.appspot.com/ (Best one for testing)
• http://rss.slashdot.org/slashdot/slashdotMain
• http://feeds.arstechnica.com/arstechnica/index
• https://hackaday.com/feed/
• https://feeds.feedburner.com/techcrunch and other feeds served via Feedburner (many)
• https://www.blogger.com/feeds/2471378914199150966/posts/default and other Blogger sites
• WordPress sites hosted on https://wordpress.com or using https://wordpress.org/plugins/pubsubhubbub/
• Friendica instances such as https://f.a80.fr/feed/alkarex/
• Etc.

Let me know whether that answers your questions so far :-)

[Alkarex]

P.S.: Maybe https://github.com/hemerajs/websub-hub could be inspiring (not tried)