Rss-bridge: Behaviour of FeedExpander when unable to fetch the remote content

[teromene]

When FeedExpander fetches a webpage of a feed in order to parse it and fails, there are two possible behaviours:

• RSS-Bridge returns a server error, resulting in the feed not being available
• RSS-Bridge goes into failsafe and prints the original feed, but inserts a warning in the content.

Both methods are already implemented into RSS-Bridge. While I am in favor of the first one, I understand the advantages that the second one brings.

What do you think @LogMANOriginal @ORelio ?

[ORelio]

To summarize my point from here, I think users would benefit from the failsafe behavior as it allows them to stay up to date with the latest content, even if the feed cannot be expanded anymore and the associated bridge needs maintainance. From a user point of view, it is harder to notice that a feed is dead as feed readers do not always report that properly.

[LogMANOriginal]

This is quite a tricky question. Taking your arguments into consideration, it took me a while to think about it. Here is my conclusion:

Let's start with a general definition for all bridges, based on BridgeAbstract.

If a bridge is unable to download or extract data (i.e. source is no longer available or DOM changed), it should issue a server error. This is how we generally get notified on broken bridges anyway.

Notice, however, that a bridge doesn't _have to_ issue a server error if it can at least download and extract __some__ of the data. This could be the case if, for example, a bridge suddenly can't extract author information due to DOM changes (but everything else works). I would call such a bridge "partially broken but still functional". It is up to the bridges' maintainer to choose the right action for those cases (issue a server error or fail silently).

Now to the FeedExpander problem at hand.

The bridge is able to download and extract the original feed data, but for whatever reason it is unable to download or extract additional data. In this case we have two options, which were mentioned before:

• issue a server error, or
• fail silently (and add a notice to the feed)

So far, so good. But...

The entire purpose of a FeedExpander-bridge is to collect and add (expand) contents to an existing feed in order to make it more valuable to the end user. A bridge that only returns the "original" feed or just adds "Cannot fetch additional contents", doesn't serve any purpose whatsoever. You could just subscribe to the original feed or make use of the FilterBridge in that case.

So, the bridge should, in fact, issue a server error.

There is but one exception to this: There is no need to issue a server error, as long as a bridge adds important (valuable) information to an existing feed (i.e. remove ads, disable auto-play on videos, etc...). This is another case of "partially broken but still functional".

I hope all of this makes sense to you :smile:

[ORelio]

How broken bridges are handled in your feed reader? In my case, it can take several days or weeks before I notice that a feed is no longer functional. Until that happens, I'm not aware of new content being published and may miss something important.

The fallback mode, even with no additional content, would help, because instead of getting no articles, the user would get truncated articles, which both help to:

1. Notice that something is going wrong (feed is not silent/broken) and take actions
2. Stay up to date with the latest content while the bridge is being fixed

Not sure if it's clearer said this way.

[LogMANOriginal]

How broken bridges are handled in your feed reader?

My feed reader (tt-rss) shows a "warning" sign for broken feeds (and you can get a summary view as well):

But it is generally not very visible (only if the feed is selected or if you enter the settings), so I understand your problem.

Not sure if it's clearer said this way.

Absolutely, thanks for explaining your situation :+1:
Technically this is an issue of the feed reader though.

That being said, I actually have an idea on how to combine both requirements (return a valid feed & let the bridge issue errors), but I'm not entirely sure if it's possible:

Let's assume a bridge fails. When this happens, RSS-Bridge generally returns error 500 (server error) or error 400 (client error). Error 400 is unlikely to happen once a feed is subscribed to, so we can ignore it for now (there are circumstances where it can happen, for example if the bridge has been changed and parameters are suddenly incompatible).

Currently, when error 500 happens, RSS-Bridge displays a generic HTML error page. I think it is feasible to return different types of contents depending on the requested format. For example:

• If the requested format is HTML, JSON or Plaintext: Return a regular HTML error page, just like we do now (no changes)
• If the requested format is ATOM or MRSS: Return a valid feed with the error 500 page embedded (feed with one item, which is the HTML error page)

This would make feed readers show the error message as if it was a regular item.

What is your opinion on this?

[ORelio]

Agreed. That's a nice way to solve the issue.

Thinking about how that would be implemented, we'd need to show the issue only once. If generating a random URL, the feed reader would get one item per fetch, which would get very noisy. With a fixed one, the feed reader would show the error once, but may ignore errors in the future... maybe by generating a random URL _and_ caching the feed for 24 hours instead of the regular timeout? :thinking:

[em92]

maybe by generating a random URL and caching the feed for 24 hours instead of the regular timeout?

I suggest to generate url with hash of input params and (int)(time()/86400). 86400 = number of seconds in 24 hours.

[LogMANOriginal]

You can actually create static URL/IDs based on the query and error code. For example:

```PHP
...
} catch(HttpException $e) {
/* This is just an example, don't access $_SERVER directly :) */
$url = $_SERVER['QUERY_STRING'] . '&error_code=' . http_response_code($e->getCode());

/* Use an ID to obfuscate the URL (because it's not a valid URL) */
$id  = hash('md5', $url);

...
// Either use static code to return feed, or generate feed with "pseudo" items

}

[teromene]

Another way than using the format of the feed we are requesting would be by using, if present, the Accept header.
When 'text/html' is the first one, we should return an HTML page and so on.

[LogMANOriginal]

Good point. We can actually combine those and use different types as fall back. For example:

switch($format) {
    case 'Atom':
        if($header === 'text/xml') {
            returnAtomError();
        }
    case 'Mrss':
        if($header === 'text/xml') {
            returnMrssError();
        }
}

returnHtmlError(); // In case no match or Html format

[ORelio]

Creating static URL/IDs based on query and error code might cause the following issue:

• Error code 0 occurs for URL A
• Error appears in feed
• Bridge is fixed and updated
• After some time, error code 0 occurs again for URL A
• Error does not appear in feed reader because it has the same ID as before

@em92's suggestion seems a simple and straightforward way of making sure the error will only appear once per day, without having to tinker with caching :+1:

Apart from that, header sniffing seems promising, but I'm not sure all feed readers will properly set the header because a feed URL is supposed to give some XML even when header is not set. Why not simply putting the error as an item and letting the formats do their job?

[LogMANOriginal]

Error does not appear in feed reader because it has the same ID as before

You are right, I didn't think about that. We could try to detect if a bridge was updated by checking the RSS-Bridge version, but that only works if changes were committed or the version number updated. With this in mind I think the 24 h solution is our best option.

Why not simply putting the error as an item and letting the formats do their job?

True, the receiver expects that format anyway. It also works out of the box for any of the currently supported or future formats.