News: Feed not updated, without any errors [$15]

[v1r0x]

IMPORTANT

Read and tick the following checkbox after you have created the issue or place an x inside the brackets ;)

• [x] I have read the CONTRIBUTING.md and followed the provided tips

Explain the Problem

After update to news 14.1.0 a couple of feeds are no longer updated

Steps to Reproduce

Explain what you did to encounter the issue

1. add http://blog.fefe.de/rss.xml?html as feed
2. Either wait for the cron to update feeds or run php ./occ news:updater:update-feed <feedid> <userid>
3. no new entries are shown in the news app, although new entries should be available

System Information

• News app version: 14.1.0
• Nextcloud version: 17.0.2
• Cron type: system cron
• PHP version: 7.3.x
• Database and version: 10.4.11 mariadb
• Browser and version:
• Distribution and version: ubuntu 18.04 lts

Contents of nextcloud/data/nextcloud.log

{"reqId":"QgkF79bFne0ZG2suHbeb","level":0,"time":"2020-01-02T14:43:28+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"new parser added : FeedIo\\Standard\\Json","userAgent":"--","version":"17.0.2.1"}
{"reqId":"QgkF79bFne0ZG2suHbeb","level":0,"time":"2020-01-02T14:43:28+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"new parser added : FeedIo\\Standard\\Atom","userAgent":"--","version":"17.0.2.1"}
{"reqId":"QgkF79bFne0ZG2suHbeb","level":0,"time":"2020-01-02T14:43:28+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"new parser added : FeedIo\\Standard\\Rss","userAgent":"--","version":"17.0.2.1"}
{"reqId":"QgkF79bFne0ZG2suHbeb","level":0,"time":"2020-01-02T14:43:28+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"new parser added : FeedIo\\Standard\\Rdf","userAgent":"--","version":"17.0.2.1"}
{"reqId":"QgkF79bFne0ZG2suHbeb","level":0,"time":"2020-01-02T14:43:28+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"read access : http:\/\/blog.fefe.de\/rss.xml?html into a feed instance (feed class : FeedIo\\Feed)","userAgent":"--","version":"17.0.2.1"}
{"reqId":"QgkF79bFne0ZG2suHbeb","level":0,"time":"2020-01-02T14:43:28+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"start reading http:\/\/blog.fefe.de\/rss.xml?html","userAgent":"--","version":"17.0.2.1"}
{"reqId":"QgkF79bFne0ZG2suHbeb","level":1,"time":"2020-01-02T14:43:28+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"hitting http:\/\/blog.fefe.de\/rss.xml?html","userAgent":"--","version":"17.0.2.1"}
{"reqId":"QgkF79bFne0ZG2suHbeb","level":0,"time":"2020-01-02T14:43:28+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"response ok, now turning it into a document","userAgent":"--","version":"17.0.2.1"}
{"reqId":"QgkF79bFne0ZG2suHbeb","level":1,"time":"2020-01-02T14:43:28+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"the stream is modified, parsing it","userAgent":"--","version":"17.0.2.1"}
{"reqId":"QgkF79bFne0ZG2suHbeb","level":0,"time":"2020-01-02T14:43:28+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"accurate parser : FeedIo\\Parser\\XmlParser","userAgent":"--","version":"17.0.2.1"}
{"reqId":"QgkF79bFne0ZG2suHbeb","level":1,"time":"2020-01-02T14:43:28+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"correct last modified date for feed Fefes Blog","userAgent":"--","version":"17.0.2.1"}
{"reqId":"QgkF79bFne0ZG2suHbeb","level":1,"time":"2020-01-02T14:43:28+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"correct public id for node Fefes Blog","userAgent":"--","version":"17.0.2.1"}
{"reqId":"QgkF79bFne0ZG2suHbeb","level":0,"time":"2020-01-02T14:43:28+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"Feed http:\/\/blog.fefe.de\/rss.xml?html was modified since last fetch. #0 items","userAgent":"--","version":"17.0.2.1"}

Contents of Browser Error Console

Read http://ggnome.com/wiki/Using_The_Browser_Error_Console if you are unsure what to put here

---

There is a $15 open bounty on this issue. Add to the bounty at Bountysource.

[Grotax]

Yea Just checked my instance same issue.

My guess is that it's related to #594 and the feed not having a lastBuildDate nor pubDate for the items.

Do your other feeds also lack that element?

[v1r0x]

I've checked another feed and there is no lastBuildDate or pubDate.

I reverted the changes from #594 and only reverting these lines didn't work. But also reverting this line fixed the problem (at least for manual fetching).
Do you know why there's a difference between DateTime and \DateTime?

[Grotax]

@kesselb could you take a look at this?

[SMillerDev]

The difference is the need to include the class first. If there's no include \Class; at the top new Class(); won't work. Would be a simple fix but probably means that the class isn't tested properly either.

[v1r0x]

Yeah, I know that difference :)

My question was why the feeds won't work with new DateTime() although it is included, but using new \DateTime() works. Sorry for the bad wording in my last comment.

[kesselb]

Index: lib/Fetcher/FeedFetcher.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- lib/Fetcher/FeedFetcher.php (revision 7d02d59d671f2d63737bc49510693c1cf3df711b)
+++ lib/Fetcher/FeedFetcher.php (date 1578148278618)
@@ -288,7 +288,7 @@
         $newFeed->setUrl($url);  // the url used to add the feed
         $newFeed->setLocation($location);  // the url where the feed was found
         $newFeed->setLink($feed->getLink());  // <link> attribute in the feed
-        if ($feed->getLastModified() instanceof DateTime) {
+        if ($feed->getLastModified() instanceof DateTime && $feed->getLastModified()->getTimestamp() > 0) {
             $newFeed->setHttpLastModified($feed->getLastModified()->format(DateTime::RSS));
         }
         $newFeed->setAdded($this->time->getTime());

Updating http://blog.fefe.de/rss.xml?html works for me with above patch. But this seems to be more a workaround than a fix.

curl -I http://blog.fefe.de/rss.xml?html
HTTP/1.1 200 Here you go
Server: Gatling/0.16
Content-Type: text/xml; charset=utf-8
Content-Length: 11276
Last-Modified: Sat, 04 Jan 2020 10:49:42 GMT
Date: Sat, 04 Jan 2020 14:38:42 GMT

curl -I --header "If-Modified-Since: Sat, 04 Jan 2020 10:49:42 GMT" http://blog.fefe.de/rss.xml?html
HTTP/1.1 304 Nix Neues

This seems to be an issue with feed-io because the feed support if-modified-since and is also sending a last modified value. Pushed a fix upstream: https://github.com/alexdebril/feed-io/pull/259

This is probably still a issue with news because somewhere the items are lost / something is done to not add the items. I'm not sure where :see_no_evil:

[e-alfred]

I am also facing this issue with numerous feeds (but not all, not even all from the same provider like Reddit) and it seems to try to update, but finds zero new items:

{"reqId":"l8ztdRUWNhsughzIuUyH","level":0,"time":"2020-01-09T14:45:12+01:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"read access : https:\/\/www.reddit.com\/r\/cats\/.rss into a feed instance (feed class : FeedIo\\Feed)","userAgent":"--","version":"17.0.2.1"}
{"reqId":"l8ztdRUWNhsughzIuUyH","level":0,"time":"2020-01-09T14:45:12+01:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"start reading https:\/\/www.reddit.com\/r\/cats\/.rss","userAgent":"--","version":"17.0.2.1"}
{"reqId":"l8ztdRUWNhsughzIuUyH","level":1,"time":"2020-01-09T14:45:12+01:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"hitting https:\/\/www.reddit.com\/r\/cats\/.rss","userAgent":"--","version":"17.0.2.1"}
{"reqId":"l8ztdRUWNhsughzIuUyH","level":0,"time":"2020-01-09T14:45:12+01:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"Feed https:\/\/www.reddit.com\/r\/cats\/.rss was modified since last fetch. #0 items","userAgent":"--","version":"17.0.2.1"}
{"reqId":"O7hrq78DYjFwoSNbsjb0","level":0,"time":"2020-01-09T15:00:11+01:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"read access : https:\/\/www.reddit.com\/r\/cats\/.rss into a feed instance (feed class : FeedIo\\Feed)","userAgent":"--","version":"17.0.2.1"}
{"reqId":"O7hrq78DYjFwoSNbsjb0","level":0,"time":"2020-01-09T15:00:11+01:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"start reading https:\/\/www.reddit.com\/r\/cats\/.rss","userAgent":"--","version":"17.0.2.1"}
{"reqId":"O7hrq78DYjFwoSNbsjb0","level":1,"time":"2020-01-09T15:00:11+01:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"hitting https:\/\/www.reddit.com\/r\/cats\/.rss","userAgent":"--","version":"17.0.2.1"}
{"reqId":"O7hrq78DYjFwoSNbsjb0","level":0,"time":"2020-01-09T15:00:12+01:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"Feed https:\/\/www.reddit.com\/r\/cats\/.rss was modified since last fetch. #0 items","userAgent":"--","version":"17.0.2.1"}

[Grotax]

Yea so the issue persists with news 14.1.1 which includes the fixed feed-io.
Seems like the fetching is working but then the items are not added to the feed.
Deleting and re-adding the feed will update the feed but probably just once.

[kesselb]

Hmm. Probably revert https://github.com/nextcloud/news/pull/594 then? Unfortunately, I'm running short on time next weeks.

[Disciplus86]

Hello, i using nextcloud 17.0.2 et news 14.1.2 and i have the same problem. My only solution is to put this command :
sudo -u www-data php ./occ news:updater:all-feeds | jq --raw-output '.feeds | map("\(.id) \(.userId)") | join("\n")' | sudo -u www-data xargs -L 1 php ./occ news:updater:update-feed
i have no error...

I have remove and reinstall the app but i have no change...

[Disciplus86]

Hmm. Probably revert #594 then? Unfortunately, I'm running short on time next weeks.

Grrr 👎, i have try to modify this #594... now, it s not working a message : httpLastModified is not a valid attribute

and now i have :
Could not update feed with id 13 and user userx : DateTime::__construct(): Failed to parse time string (1578923602365093) at position 15 (3): Unexpected character

For this problem : the solution is to activate/deactivate full text on all feed manually.
but the autoupdate not working...

[gameman733]

I think i got hit by this. It seemed to affect my reddit /r/news feed pretty badly (https://www.reddit.com/r/news/.rss), which is weird because I have other reddit feeds that occasionally would show a few entries. I ended up downgrading back to 14.0.2, but my feeds still weren't updating. I had to update the last_modified column on the oc_news_feeds table before it started finding any of the missing entries. I don't remember what they were set to at the time, but they seemed to be set to the current time in UTC, so I updated to current date at midnight. A bit of a hack job, but maybe this info will be useful?

[Grotax]

Ok I guess reverting this change is our best option right now.

[Disciplus86]

Hi, i have resolve my problem with 2 actions:

• I have remove all reddit rss list
• i have disable full text in all news list.

can you confirme this solution?

[gameman733]

I started another nextcloud instance with the news app, for testing purposes. Added the reddit news rss feed I mentioned before, and after the initial population, it stopped adding anything new. Full text indexing was disabled. I enabled it, and it populated more, but then stopped again. I just now disabled it, and more feed items were imported.

[SMillerDev]

Yeah toggling full text will fetch the data again forcefully. It seems it just doesn't import this feed and the change will have to be reverted.

[TobiasDev]

Hello, I'm also seeing a behavior similar to this on several feeds. It doesn't seem to fetch data when I have "enable full text" on except for when I'm turning it on or off. So it force updates them

[SMillerDev]

Unless you can offer some new information, please don't comment that you have the same problem. It only hides the useful information

[kesselb]

I'm able to reproduce it :tada:

Example for http://blog.fefe.de/rss.xml:

1) Subscribe to feed via ui
2) First import is successful. 20 items importeded, Last Modified is Fri, 24 Jan 2020 17:22:00 +0000 for feed and every item (that is expected because the items have no date itself).
3) Delete latest 4 items and run update-feed command. No new items are added because the server responded with 304 (thats's expected).
4) Change last modified to Fri, 24 Jan 2020 17:20:00 +0000. Now the latest item should be added again but feed-io will not return it.

https://github.com/alexdebril/feed-io/blob/c651b6fc8952ebf0121bc0cc06ceeb3a69d482cf/src/FeedIo/Filter/ModifiedSince.php#L37-L44

If we call feed-io with modifiedSince the above filter is enabled. Any news item without a pubDate is not valid then. I think that's wrong and pushed a fix upstream.

We are not able to change this behaviour from news (it all happens within feed-io). So the right workaround until this is fixed is to not use modified since.

[DudleyDursley]

Another feed that isn't updated anymore: https://www.klimareporter.de/feed/rss/

It has a <lastBuildDate> for the feed and <pubDate> for the items.

[kesselb]

I don't see any issues with https://www.klimareporter.de/feed/rss/ :thinking: They seem to ignore If-Modified-Since header and always return a response. Feed-io will return ever item newer than last modified, news will add non existing items and last modified is set to now (because that's returned as lastBuildDate).

What values do you see for http_last_modified in oc_news_feeds table and what is the latest item for the feed in oc_news_items table (please post the full row)?

[kesselb]

Hmm. Looks good to me. Also the update is working for me. @DudleyDursley why do you removed the column names again? It was much easier to read with ;)

https://github.com/nextcloud/news/commit/a3246a927de542e1b3ab403359bfd3c08705b6a7#diff-8f9069b2f49b315e39e562bd9309ac5f

Since this commit updatedDate is no longer set https://github.com/nextcloud/news/blob/6673cbc3d940745a0ecddb93b32805a0fbe79eb1/lib/Service/FeedService.php#L269-L271

but used here as comparison. Probably something to review @SMillerDev.

[DudleyDursley]

http_last_modified in oc_news_feeds: Mon, 27 Jan 2020 09:14:21 +0000

The latest item in oc_news_items. I hope the csv isn't a problem, normal SELECT produced a multi-line output that destroyed the formating:
21669;6f0311beb5db4c12fadb54c9b41a6475;236a034eb93856659ece902297a99bc6;236a034eb93856659ece902297a99bc6;f;"weniger schnee führt zu weniger unfällen, längere vegetationsperioden können die ernten verbessern, heiße sommer steigern den umsatz von eisläden: durch den klimawandel wird manches auch besser. trotzdem muss man sich mit den negativen folgen der erderhitzung beschäftigen – damit etwas dagegen getan wird. hier kommt das positivehttps://www.klimareporter.de/erdsystem/hier-kommt-das-positive";https://www.klimareporter.de/erdsystem/hier-kommt-das-positive;https://www.klimareporter.de/erdsystem/hier-kommt-das-positive;Hier kommt das Positive;;1577872800;;"<p>Weniger Schnee führt zu weniger Unfällen, längere Vegetationsperioden können die Ernten verbessern, heiße Sommer steigern den Umsatz von Eisläden: Durch den Klimawandel wird manches auch besser. Trotzdem muss man sich mit den negativen Folgen der Erderhitzung beschäftigen – damit etwas dagegen getan wird.</p> ";;;46;0;f;f;1577890646918943

[kesselb]

What are you doing? The first version including the column names are ok. Ask yourself is the CSV output above easy to read? :smiley:

[DudleyDursley]

I'm sorry, I was just confused by the formating:

id | guid_hash | fingerprint | content_hash | rtl | search_index | guid | url | title | author | pub_date | updated_date | body | enclosure_mime | enclosure_link | feed_id | status | unread | starred | last_modified

21669 | 6f0311beb5db4c12fadb54c9b41a6475 | 236a034eb93856659ece902297a99bc6 | 236a034eb93856659ece902297a99bc6 | f | weniger schnee führt zu weniger unfällen, längere vegetationsperioden können die ernten verbessern, heiße sommer steigern den umsatz von eisläden: durch den klimawandel wird manches auch besser. trotzdem muss man sich mit den negativen folgen der erderhitzung beschäftigen – damit etwas dagegen getan wird.+| https://www.klimareporter.de/erdsystem/hier-kommt-das-positive | https://www.klimareporter.de/erdsystem/hier-kommt-das-positive | Hier kommt das Positive | | 1577872800 | | <p>Weniger Schnee führt zu weniger Unfällen, längere Vegetationsperioden können die Ernten verbessern, heiße Sommer steigern den Umsatz von Eisläden: Durch den Klimawandel wird manches auch besser. Trotzdem muss man sich mit den negativen Folgen der Erderhitzung beschäftigen – damit etwas dagegen getan wird.</p>+| | | 46 | 0 | f | f | 1577890646918943 | | | | | hier kommt das positivehttps://www.klimareporter.de/erdsystem/hier-kommt-das-positive | | | | | | | | | | | | | |

I created a new instance in a Docker container, added the feed an it is not updating.

[e-alfred]

@kesselb Which changes/PR/commit are necessary to test this? I would like to apply the changes and see if it works.

[kesselb]

@e-alfred https://github.com/alexdebril/feed-io/pull/260

@DudleyDursley All good :smile: Probably someone else can reproduce.

[e-alfred]

@kesselb Okay, then I already applied the right patch over the weekend. It seems to work, but I am a bit wary because some feeds are still not updated but it is just because there is no new content from those sites. :-)

[DudleyDursley]

Not sure if it matters. The item I posted is the latest one when sorted by pub_date and the latest in the web interface. The latest item by last_modified is this:

21666 | 1b09f0fd895480c655b6c891616e4953 | 68aebc1d4770296566e06707576201ad | 68aebc1d4770296566e06707576201ad | f | die seit juni regierende fünf-parteien-koalition will bis 2029 aus der kohle aussteigen und bis 2035 die co2-emissionen auf null bringen. dafür muss finnland mehrere sektoren umbauen.+| https://www.klimareporter.de/europaische-union/finnland-vorwaerts-zur-klimaneutralitaet | https://www.klimareporter.de/europaische-union/finnland-vorwaerts-zur-klimaneutralitaet | Finnland: Vorwärts zur Klimaneutralität | | 1577713490 | | <p>Die seit Juni regierende Fünf-Parteien-Koalition will bis 2029 aus der Kohle aussteigen und bis 2035 die CO<sub>2</sub>-Emissionen auf null bringen. Dafür muss Finnland mehrere Sektoren umbauen.</p>+| | | 46 | 0 | f | f | 1577892448931120 | | | | | finnland: vorwärts zur klimaneutralitäthttps://www.klimareporter.de/europaische-union/finnland-vorwaerts-zur-klimaneutralitaet | | | | | | | | | | | | | |

[alexdebril]

Hi
I've just released v4.5.2 of feed-io featuring @kesselb 's hotfix, please let me know if it fixes your issue.

[DudleyDursley]

I set up a small webserver and copied the xml from https://www.klimareporter.de/feed/rss/ to a textfile I then updated manually. Updates are working, so the problem is not the feed itself.

Is it possible that feed.io or News aren't updating this feed because of this header Expires: Wed, 17 Aug 2005 00:00:00 GMT ?

[alexdebril]

I don't think so, feed-io only looks at the last-modified header

[alexdebril]

Last-modified header gives Sat, 01 Feb 2020 09:00:00 GMT whereas the last item was published on Fri, 31 Jan 2020 19:04:18 +0100.

Could you try with feed-io version 4.5.0 in your application please ? @kesselb : I'm pretty sure the http-last-modified fixer is the root-cause of this.

[DudleyDursley]

Their Last-modified header is always the current date/time.

4.5.0 works.

[kesselb]

Hmm. But the feed has a lastPubDate. Why would the fixer run?

[DudleyDursley]

Whatever the problem is, master seems to work as well.

[alexdebril]

The feed is really buggy: now there's a lastBuildDate that always gives the current date and so does the HTTP Last-Modified header ... Actually it's impossible to trust the feed's lastBuildDate / lastPubDate or whatever node specified by the RSS protocol. The only way to provide an accurate value for \FeedIo\FeedInterface::getLastModified() is to iterate through all items and pick the latest value (when items have a publication date).

[kesselb]

Tricky :man_shrugging:

https://github.com/alexdebril/feed-io/blob/3f53b1105a4dcc0af878410739582b7ed3530edd/src/FeedIo/Standard/Rss.php#L126

We could remove lastBuildDate (still not sure after reading the RFC what's the purpose) and put the http-last-modified header behind the last-modified fixer. That should work for fefe (only http last modified header) but also for this klimareporter feed. I'm not a fan to be honest. I think klimareporter is broken and see no reason to fix (or add to workaround for) that.

[tobip]

I have the same problem with blog.fefe.de, and this would be another feed url that doesn't update:
https://node2.feed43.com/5058372758686815.xml

[kesselb]

https://node2.feed43.com/5058372758686815.xml

lastBuildDate is also the issue for that feed.

[alexdebril]

I think this should fix the problem: https://github.com/alexdebril/feed-io/pull/262

to test it, set the dependency as follow in your composer.json:

"debril/feed-io": "dev-issue/261 as 4.5.3",

then run composer update debril/feed-io

@DudleyDursley please let me know if it solves the issue for you.
@kesselb I agree, we should prefer clean solutions over hacking the RSS protocol. But at the end we have users and sometimes they want to consume crappy feeds ...

[DudleyDursley]

Instead of using composer I replaced the default feedio whith this. It didn't work.
If composer does anything else, I can look into it, but I have no experience with all this php stuff and News releases don't have a composer.json.

[alexdebril]

fixed with feed-io v4.5.3. The way it manages lastModified complies now with fefe.de, klimareporter.de and normal feeds

[tetzlav]

I think this should fix the problem: alexdebril/feed-io#262

to test it, set the dependency as follow in your composer.json:

"debril/feed-io": "dev-issue/261 as 4.5.3",

then run composer update debril/feed-io

doesnt work for me...

[alexdebril]

You don't need the alias anymore, just install version 4.5.3 and you're set.

"debril/feed-io": "^4.5.3",

I made an update to the code since yesterday, maybe that's why the issue isn't fixed yet for you

[alexdebril]

Also @tetzlav , what is the feed you have troubles with?

[tetzlav]

You don't need the alias anymore, just install version 4.5.3 and you're set.

"debril/feed-io": "^4.5.3",

I made an update to the code since yesterday, maybe that's why the issue isn't fixed yet for you

i checked out news git master, edited composer.json, updated feed-io, make -> but https://blog.fefe.de/rss.xml?html wont be updated anymore since my update to nextcloud 17.0.3 yesterday...

[tetzlav]

Oh, it works again! But only for really new feeds, not for allready syncronized (and not as new marked) articles.

[Grotax]

I released news 14.1.3 which contains feed-io 4.5.3 yesterday.

[richjsouth]

I released news 14.1.3 which contains feed-io 4.5.3 yesterday.

I have News version 14.1.3 but for about 6 days now I've not had any feed updates but can cofirm that the "enable full text" then "disable full text" will force an update but they aren't updating automaticaly.

[Grotax]

Yea news doesn't fix that itself when a feed runs into like a frozen status it will stay there toggelig full text helps because it kicks off the whole sync thing from 0

[Grotax]

Don't know if there is an easy solution for that :unamused:

[kesselb]

It depends from feed to feed. It's also good to check the nextcloud log. Probably other issues are blocking the cron from executing.

[richjsouth]

Interestingly, it seems to be working again after toggling full text to not full text even though I had previously restarted the server and restarted cron etc.. The app on my phone ownCloud News (which I think is the 'official' one) seems a little glitchy (sometimes won't refresh) but eventually sorts itself out.

[DudleyDursley]

I'm still having problems with the Klimareporter-feed but I might have found the problem:

If new feed items are fetched, http_last_modified in oc_news_feeds is set to the <pubDate> of the latest item. If there are no new items, it is set to the current date/time. This is probably happening because the <lastBuildDate> of the feed and the Last-Modified-header are always set to the current date/time.

Another peculiarity of this feed is that the <pubDate> of the feed items isn't the time they appear on the website or in the feed, it is a time several hours before that. Maybe it's the last time they touched it in their CMS or whatever.

This means that http_last_modified will usually be "bigger" than the <pubDate> of new feed items and in turn (I think) that these feed items will be ignored. Manually setting http_last_modified to a time directly before the <pubDate> of missing feed items fixes the problem temporarily.

[SMillerDev]

Yeah, that's a feed problem. Last-Modified should contain the date it was last modified, not the current date.

[gameman733]

I updated my test instance with the new version of the news app a few days ago and still am not getting anything with the reddit rss feed. Just as reference, it still works fine with the version prior to 14.1.x (don't have the exact version on hand right at this second)

[BOFH90]

The following feed doesnt update on my news app: http://feeds.feedburner.com/blogspot/rkEL
all other are working flawlessly

[alexdebril]

Bug is still there, here is the explanation https://github.com/alexdebril/feed-io/issues/266

[alexdebril]

I've pushed a fix here: https://github.com/alexdebril/feed-io/pull/267 . It's being tested through https://github.com/alexdebril/feed-io-impl and it seems to be good, I just want to wait a bit to be sure it doesn't introduce a regression for blog.fefe.de

[alexdebril]

bugfix published through version 4.5.4 of feed-io

[DudleyDursley]

This appears to be working for me.

[gameman733]

I installed news 14.1.4-RC1 on a test nextcloud 18.0.3 instance and threw the reddit RSS feed in there and I'm still not getting updates. Url is https://www.reddit.com/r/news/.rss. Is there something else I can try or more information needed to help with this?

[alexdebril]

I've found that feed-io's state can lead to this. The easy solution is to create a new instance of \FeedIo\FeedIo for each call.

When the same instance is used for every calls:
https://travis-ci.com/github/alexdebril/feed-io-sanity-checker/jobs/313429992

When using a new instance for each call:
https://travis-ci.com/github/alexdebril/feed-io-sanity-checker/jobs/313416589

[kesselb]

@alexdebril what do you suggest? Just update FeedIo or create a new instance for each feed?

[alexdebril]

@kesselb sorry I forgot to get back here after releasing the fix... The issue is fixed with v4.5.6 so a simple update of feed-io is enough

[spackmat]

Hi, I have another feed not updating: https://www.shopblogger.de/blog/feeds/index.rss2

But I think it has its own problem (as I discussed in 2018 in an FreshRSS issue: https://github.com/FreshRSS/FreshRSS/issues/1020): Its Etag- and Last-Modified-headers have a date in 2041 set and so any feedreader caching the feed will always get a 304 response, when they send a request with this date in a if-modified-since request header. FreshRSS solved such problems (I had three feeds of those) with a feed option disabling the cache at all for a problematic feed. Maybe News also stumbles upon that.

I just wrote to the author informing hin about his broken feed, but did that also in 2018 with no answer. Hoping the best. But there always will be feeds with similar problems that work in Feedly or other readers well (as they have workarounds for them).

[alexdebril]

@spackmat this feed is not so wrong, I've checked it with feed-io and it should work:

./bin/feedio check https://www.shopblogger.de/blog/feeds/index.rss2

reading https://www.shopblogger.de/blog/feeds/index.rss2
--------------------------------------------------------

 the feed has items (15):                [OK]
 the date flow is normal:                [OK]
 the feed has items on second call (7):      [OK]
 the feed is updateable:                 [OK]
 a call in the future is empty as expected:      [OK]
 a call at Jan 1970 is filled as expected:   [OK]
 a call with modifiedSince = 1yr old is filled:      [OK]
+--------------------------------------------------+------------+-----------+---------------------------+---------+------------+-----------+----------+------------+--------+
| URL                                              | Accessible | readSince | Last modified             | # items | unique IDs | Date Flow | Jan 1970 | 1 year old | Future |
+--------------------------------------------------+------------+-----------+---------------------------+---------+------------+-----------+----------+------------+--------+
| https://www.shopblogger.de/blog/feeds/index.rss2 | OK         | OK        | 2020-04-29T10:25:00+02:00 | 15      | OK         | OK        | OK       | OK         | OK     |
+--------------------------------------------------+------------+-----------+---------------------------+---------+------------+-----------+----------+------------+--------+

The wrong Last-Modified header is not a problem here, as feed-io falls back on it only if it absolutely no date was found inside the feed.

[spackmat]

I see. But it hasn't updated since I added the feed 15 days ago, the initial load went fine. Do you see another reason why the feed doesn't get updated? My other dozens of feeds seem to work well (hard to tell without having seen all of them updating in FreshRSS).

[dennmuel]

Sorry to just jump on the waggon, but I too still see this behaviour with News v.14.1.7 and this feed: https://www.deutschlandfunk.de/podcast-forschung-aktuell-komplette-sendung.417.de.podcast.xml

(Also: isn't #665 a duplicate?)

[alexdebril]

@e-alfred your comment is a duplicate of #691

[e-alfred]

Here are a few feeds that don't update since 14.1.7 was released two weeks ago, but worked perfectly before that:

http://feeds.feedburner.com/ServeTheHome
https://www.heise.de/security/rss/alert-news-atom.xml
seclists.org/rss/fulldisclosure.rss

Some Reddit feeds work, others don't which is also quite strange.

[DudleyDursley]

I'm also subscribed to ServeTheHome but the feed is working for me. I'm using the current https-url though: https://feeds.feedburner.com/servethehome

[e-alfred]

Almost all feeds work now once again with release 14.1.8, thanks for fixing issue.

[vinylmeister]

All my feeds stopped updating a few hours ago. Updated to the .9 release, but still not working.

[loxK]

Same here all feeds not updating anymore. Last week it also happened I had to run the cron manually multiple times until it worked. But looking at the logsn the cron is running fine as planned.

Today it happened again, nothing in the logs, not even an attempt to update the feeds. I ran the command as advised here https://github.com/nextcloud/news/issues/607#issuecomment-573680398, without error. All feeds updated and now they continue to update via cron like a charm.

[Vultwulf]

Reddit is a weird one since depending on how you are sorting Top, New, etc the RSS won't be in chronological order. So if a new item makes it to the list that has been published before any of the other items in the RSS it might not find the "new" one with the older published date. This sounds like it could break RSS standards, but it's still a feed that I want to track.

I've had to do various workarounds for the last few major versions for it to keep working the way that I expect it to. The latest one I have takes the $lastModified date and goes back a week to see if there are any "new" older items that might have gotten missed.

FeedFetcher.php

        if (empty($lastModified) || !is_string($lastModified)) {
            $resource = $this->reader->read($url);
        } else {
            $dt = new DateTime($lastModified);
            $dt->modify('-7 day');
            $resource = $this->reader->readSince($url, $dt);
        }

[loxK]

Personally I have no Reddit feed.

A wrong feed should not prevent ALL feeds from updating at all.

[adeptg]

I have the same (?) issue with this feed: https://habr.com/en/rss/feed/posts/all/a56ae7f180513779ab750cb8b752cc28/?fl=ru%2Cen
Full text search enable/disable triggers the update, but it doesn't update via cron or occ news:updater:update-feed 80 adept
The other feeds look fine.

news: 14.1.11, Nextcloud 17.0.8 installed via docker

It used to work before some update, but I'm not sure if it was Nextcloud 16 to 17 update, or some News app update.

[SMillerDev]

Did you try the solution in the FAQ?

[adeptg]

@SMillerDev thanks for the advise! I've checked FAQ, and, as far as I can see, there is no solution for my issue: only 1 feed from my list doesn't work. I use system cron. When I run docker exec -u www-data nextcloud php ./occ news:updater:update-feed 80 adept I see in logs

Info news hitting https://habr.com/ru/rss/feed/posts/a56ae7f180513779ab750cb8b752cc28/?with_hubs=true%3Fwith_hubs%3Dtrue
Debug news start reading https://habr.com/ru/rss/feed/posts/a56ae7f180513779ab750cb8b752cc28/?with_hubs=true%3Fwith_hubs%3Dtrue
Debug news read access : https://habr.com/ru/rss/feed/posts/a56ae7f180513779ab750cb8b752cc28/?with_hubs=true%3Fwith_hubs%3Dtrue into a feed instance (feed class : FeedIoFeed)
Debug news new parser added : FeedIoStandardRdf
Debug news new parser added : FeedIoStandardRss
Debug news new parser added : FeedIoStandardAtom
Debug news new parser added : FeedIoStandardJson

And nothing else happens - no any new article was added.
But I can see new articles in browser if I open https://habr.com/en/rss/feed/posts/all/a56ae7f180513779ab750cb8b752cc28/?fl=ru%2Cen
Thanks in advance for your help!

[adeptg]

I've tested 14.2.0 with https://habr.com/en/rss/feed/posts/all/a56ae7f180513779ab750cb8b752cc28/?fl=ru%2Cen - it updates only on manual "enabled/disable full text search" trigger. Other feeds work well

[v1r0x]

14.2.0 breaks all of my feeds. Toggling "full text" works as @adeptg already mentioned

[primalmotion]

Same here. None of my feeds are actually updating using cron. I have to toggle "full text" with 14.2.2

[adeptg]

@SMillerDev I still have the same issue on News 14.2.2 (latest for Nextcloud 18). What version of News I need to use to get rid of this bug?

[SMillerDev]

I don't really remember what version we fixed this bug in, but it's not going to be fixed in the 14.x branch. We don't really have enough maintainers to maintain one version, let alone back ports.

[adeptg]

Got it, thanks. Will update to the latest one.
Thanks for your work! You're doing a great job with very limited resources