News: feeds not updating

[jknockaert]

A number of issues have been filed regarding streams no longer updating since the upgrade to 13.1.0: #443, #412 (which was closed without a fix), and probably also #408 (where I misunderstood what was happening.
Some observations that have been made:

• some feeds do update, others don't
• feeds do update when running the updater on individual feeds

I think I identified what is causing the issue. At some point in the code (function isValid in file ParserAbstract.php) it is checked for each item (of a feed) whether it should be added by evaluating a number of filters. There should actually be (at most) one filter there, which is set (addFilter) in function read in FeedIo.php that adds it to the reader. However, this filterlist is not reset between feeds. That's where it goes wrong. It could be fixed by adding $this->getReader()->resetFilters(); to the read function in FeedIo.php, but I am not sure that is the right place for this to happen. Ideally it should be handled at a higher level where the loop over the feeds is defined.

[etam]

As a temporary workaround I created this script, that can be used in cron:

php ./occ news:updater:all-feeds | jq --raw-output '.feeds | map("\(.id) \(.userId)") | join("\n")' | xargs -L 1 php ./occ news:updater:update-feed

[Grotax]

Check #416. The current release was packed with a old feed-io version. And we had #432 to fix that.

[jknockaert]

@Grotax As far as I can see the issue is not with feed-io. The issue is with how feed-io is used in the news app. As long as feed-io is called for a single feed everything is fine. But when looping over feeds within the same php call, things go wrong. I can't see how #416 and #432 are related here.

[SMillerDev]

Please make a pull request so we can test your improvement. I, as the author of the FeedIO integration, think the mentioned PRs are the solution as we don't ever set any filters. I'd be happy to be proven wrong though.

[jknockaert]

@SMillerDev The filter is set here: https://github.com/alexdebril/feed-io/blob/release/3.0/src/FeedIo/FeedIo.php#L282
During the update when looping over feeds the filterlist isn't reset, so when I was checking the filterlist used at https://github.com/alexdebril/feed-io/blob/release/3.0/src/FeedIo/ParserAbstract.php#L120 I found out it contained 102 FeedIo\Filter\ModifiedSince objects. That couldn't be right.
After adding $this->getReader()->resetFilters(); at https://github.com/alexdebril/feed-io/blob/release/3.0/src/FeedIo/FeedIo.php#L280 everything returned to normal. However, I think this is a proper solution only in the context of the news app. So it would be better to fix it where the loop over the feeds is managed; however, I have no idea how to do tht.

[SMillerDev]

That'd be best in the FeedFetcher class.

[jknockaert]

@SMillerDev Can you take this up? I will review.

[jknockaert]

@Grotax Wouldn't it be more appropriate to label this as a bug, rather than undecided?

[v1r0x]

Feeds are still broken in latest release. Weird thing is: it's also broken in 13.0.1 after a downgrade 😕

Edit: This fixed the problem on 13.0.1

[adeptg]

For me script from @etam helped. Now in web UI I see new articles. But Nextcloud News Android app doesn't see any new article

[jknockaert]

Fix proposed in #448.

[v1r0x]

Shouldn't this be closed when #448 is merged and confirmed it's working?

[jknockaert]

Alright

[metcheverry23]

As a temporary workaround I created this script, that can be used in cron:

php ./occ news:updater:all-feeds | jq --raw-output '.feeds | map("\(.id) \(.userId)") | join("\n")' | xargs -L 1 php ./occ news:updater:update-feed

this workaround is working for me

[Sinkmanu]

I have installed the version 13.1.3 and I have applied the patch #448 and when I perform an update with cron.php it still without update the feeds.

sudo -u www-data php -f cron.php

[jknockaert]

@Sinkmanu Did you check #366?

[Sinkmanu]

@jknockaert , I have checked and now it works correctly. It was my problem. thanks :+1: