FreshRSS 🚀 - Reeder not working with FreshRSS greader API anymore

Hello,
Could you please try this test https://github.com/FreshRSS/FreshRSS/issues/2620#issuecomment-548895740 and report your logs?

And also check that you have correctly updated your FreshRSS, by ensuring that the following line is identical on your side:
https://github.com/FreshRSS/FreshRSS/blob/03208771f7d38771f7b9c965b79877d97cce311a/p/api/greader.php#L699

Alkarex on 23 Nov 2019

Please provide some information about your system: platform, versions, installation...

Alkarex on 23 Nov 2019

Sure, it's an Ubuntu 18.04.3 LTS (x86_64), FreshRSS 1.15.3 installed via git. Webserver is an Apache 2, MySQL and PHP is also the newest version.

Line 699 in greader.php is indeed "'id' => '' . $id, //64-bit decimal"

I've uncommented the lines now, let's see if something is showing up in the logs.

KingKarlo on 23 Nov 2019

👍1

Okay, in the log a new part starting with "[Sat, 23 Nov 2019 13:00:00 +0100] [warning] --- badRequest() Array" was added. I'm not sure how good "bdRequest" sounds, but the rest looks normal :)

Should I post it in here (I would remove the server name and the IPs) or should I send it to you differently?

KingKarlo on 23 Nov 2019

Yes, you can post here (after editing any confidential information from the logs such as passwords and addresses) or mail me.

Alkarex on 23 Nov 2019

Okay, here's the part that got added...and I hope I've replaced the server name properly and masked the IPs.

KingKarlo on 23 Nov 2019

Hum, the client tries to log in but does not provide any credentials. You do not have removed any Email or Passwd parameter, have you?

Alkarex on 23 Nov 2019

No, but I've entered them while setting up the connection and that worked fine without an error.

Could it be also an error on the Reeder side?

I think the author changed something with one of the last versions regarding FreshRSS.

KingKarlo on 23 Nov 2019

For such a request, the Email and Passwd parameters are expected as POST (best) or GET data, but they do not seem to be there. It does look like a problem from the client. Try to log out and in again in the client.
Unfortunately, I do not have the possibility of trying that client myself.

Alkarex on 23 Nov 2019

Deleted the old account and set up a new one, it's the same behavior.

Looks like it could be indeed a problem on the Reeder side.

I'm wondering why it worked for others with Reeder and not for me. You only use the server name, user name and password in Reeder. So there's not much that can go wrong in theory.

KingKarlo on 23 Nov 2019

Have the same problem, greader does not work; fever run fine. I use Reeder 4.2.1

Tealk on 27 Nov 2019

I also tried to contact the Reeder developer, but haven't heard back so far.

Good to see that I'm not the only one :)

KingKarlo on 28 Nov 2019

👍2

I also tried to contact the Reeder developer, but haven't heard back so far.

Good to see that I'm not the only one :)

One more here with the same issue.

davidrei on 2 Dec 2019

Unfortunately no feedback from Reeder yet.

I wish Silvio Rizzi would be as fast as Alexandre 🥇

Seems to be indeed more Reeder related than FreshRSS, maybe something broke with the latest update to 4.2.1

KingKarlo on 3 Dec 2019

😄1

According to the logs I have seen, it indeed does not seem to be related to FreshRSS. More logs welcome ( https://github.com/FreshRSS/FreshRSS/issues/2684#issuecomment-557789803 ) from other users, as you might not hit the same issue.

Alkarex on 3 Dec 2019

Same exact issue here.
Super strange. I don't think I did anything, and it just stopped working. What I thought caused it was updating IOS, but Reeder on MacOS also doesn't work.
Reeder using grader api/FreshRSS on 3 devices all broke. Fever is fine.
I've tried reinstalling FreshRSS, logging the clients back in, deleting profiles etc.
I'll try get some logs as well.

bobharris on 10 Dec 2019

Super strange is exactly my feeling!

I'm also not sure what I did exactly, even though I'm sure that I've updated FreshRSS and Reeder (with new FreshRSS support) at almost the same time. So it's hard to say what went wrong.

Reeder with Fever API is working for me too and the strange thing is greader API is working with EasyRSS on Android (installed it for testing) but it doesn't work with Vienna on Mac. But I wouldn't rule out that that's a Vienna thing.

Really weird and I hope we can figure out what happened!

KingKarlo on 10 Dec 2019

I've setup the https://demo.freshrss.org instance in 1.14.3. Can you try Reeder on this instance so we can be sure the problem comes from Reeder and not from FRSS 1.15? Login: demo / password: demodemo (or you can create an account, but it will be dropped at the next update)

marienfressinaud on 11 Dec 2019

👍1

i never deleted the "greader" setting, i just look into the reader and it runs again. I don't understand that now.
But that doesn't work on any google server that might have had a problem?

Tealk on 11 Dec 2019

I've setup the https://demo.freshrss.org instance in 1.14.3. Can you try Reeder on this instance so we can be sure the problem comes from Reeder and not from FRSS 1.15? Login: demo / password: demodemo (or you can create an account, but it will be dropped at the next update)

Works great here. Reeder IOS. I might try roll back mine to 1.14.3.

bobharris on 11 Dec 2019

Indeed! Works great with my Reeder for iOS apps too.

These are the same devices where it isn‘t working with the 1.15.x

Will try with Vienna on Mac when I‘m at home.

KingKarlo on 11 Dec 2019

Okay, Vienna is too broken Itself to figure out if it's working or not 😁

But FreshRSS with the demo 1.14.3 instance is working on all devices via greader APi

KingKarlo on 12 Dec 2019

Something odd is going on with my FreshRSS via Reeder greader. I am sometimes pulling articles now. However the account never goes into a "synced" state like the fever api does.

I have logging enabled on the greader.php, so I can dump the logs from the last couple days and up them.

bobharris on 12 Dec 2019

Okay, I might have figured out something very weird by accident (I should have made a backup first g). And please don't start a discussion about the right number of unread articles :)

Greader API in 1.15.3 is working as soon as the number of unread articles is below 10.000

If it's above 10.000 it's not working.

In 1.14.x it seems to work all the time, no matter how many unread articles.

Could that be?

KingKarlo on 12 Dec 2019

The number 10000 was added in #2335, but I don't think that should be related to any API calls.

Frenzie on 12 Dec 2019

Okay, I might have figured out something very weird by accident (I should have made a backup first _g_). And please don't start a discussion about the right number of unread articles :)

Greader API in 1.15.3 is working as soon as the number of unread articles is below 10.000

If it's above 10.000 it's not working.

In 1.14.x it seems to work all the time, no matter how many unread articles.

Could that be?

It'd explain my problems. Right now I'm at ~9500. I've been hovering in and out of working the past couple days.....

bobharris on 12 Dec 2019

The number 10000 was added in #2335, but I don't think that should be related to any API calls.

But it looks at least from the behavior like it‘s related. Maybe not the change in #2335, but at least the number 10.000. I made multiple tests now, with 9.999 unread articles it‘s working, with 10.000 it isn‘t.

KingKarlo on 12 Dec 2019

If any of you could provide me some logs of a not-working call, that would help :-)

Alkarex on 15 Dec 2019

Strange, I still have line 900 & 901 uncommented in greader.php, but a refresh doesn't add anything in the log_api.txt. Not sure why...

Otherwise I would've sent logs, but maybe someone else having the problem could jump in.

KingKarlo on 15 Dec 2019

@KingKarlo Could you please check that you at least have some entries in your Web server log? If not, it means that the requests (if any) are not even reaching FreshRSS

Alkarex on 15 Dec 2019

Yes, I see the Reeder access in the Apache logs.

And I tried the following (after taking a mysql dump this time g):

Tried to sync -> not working
Marked enough articles read to go to 9987 unread articles -> tried sync -> works
Add another feed to get above 10.000 unread articles (10.008) -> tried to sync -> not working
Marked again to drop below 10.000 -> tried to sync -> works

Really weird, I know...and still not sure what it's proving...but still weird.

KingKarlo on 15 Dec 2019

I'm not seeing anything in log_api.txt after turning on in greader.php.

All I'm seeing is:
[Tue, 24 Dec 2019 20:35:31 +0200] [warning] --- Fever API: Refresh items - notImplemented()
[Tue, 24 Dec 2019 20:46:23 +0200] [warning] --- Fever API: Refresh items - notImplemented()
[Tue, 24 Dec 2019 20:57:40 +0200] [warning] --- Fever API: Refresh items - notImplemented()

Using the fever API for >10,000 articles.

bobharris on 24 Dec 2019

That's interesting, because it's the same with my installation at the moment and I thought it's a problem just on my side.

I guess my messages come from a different client on iOS I'm using, but I don't see any new log entries regarding greader API access.

Still the same behavior here, as soon as I get over 10.000 unread articles greader API stops working in Reeder.

KingKarlo on 26 Dec 2019

Hello. I think I have the same problem. I have installed FreshRSS version 1.15.3 (clean install) on macOS 10.14.6, configured to use SQLite, accessible via a nginx instance working with PHP-FPM.

The FreshRSS is configured to accept API requests, and have an API password set for the admin user I'm playing with.

When I try to add a FreshRSS account in Reeder 4.2.2 (macOS) with the URL https://<redacted-hostname>/freshrss/p/api/greader.php/ (the part after the freshrss is automatically set by Reeder), put my admin user, and my API password, then I have an error "An error occurred while trying to log in.".

The whole site UI works perfectly fines.

I have this access log (i.e. not considered as an error) in nginx when trying to login _via_ Reeder:

<redacted-remote-ip> - - [27/Dec/2019:23:57:59 +0100] "POST /freshrss/p/api/greader.php/accounts/ClientLogin HTTP/1.1" 400 22 "-" "Reeder/4020.29.01 CFNetwork/978.2 Darwin/18.7.0 (x86_64)"

And if I analyze the traffic with Charles (https://www.charlesproxy.com), I see that:

_Request_

[Headers]
POST /freshrss/p/api/greader.php/accounts/ClientLogin HTTP/1.1
Host    <redacted-hostname>
Content-Type    application/x-www-form-urlencoded
Connection  keep-alive
Accept  */*
User-Agent  Reeder/4020.29.01 CFNetwork/978.2 Darwin/18.7.0 (x86_64)
Content-Length  130
Accept-Language en-us
Accept-Encoding br, gzip, deflate

[Content]
client=Reeder&service=reader&accountType=HOSTED_OR_GOOGLE&Passwd=<redacted-api-password>&output=json&Email=<redacted-username-which-is-not-an-email>

_Reply_

[Headers]
HTTP/1.1 400 Bad Request
Server  nginx/1.17.6
Date    Fri, 27 Dec 2019 22:50:32 GMT
Content-Type    text/plain; charset=UTF-8
Transfer-Encoding   chunked
X-Powered-By    PHP/7.1.32
Connection  keep-alive

[Content]
Bad Request!

This is logs from log_api.txt:

There is no Logs in FreshRSS internal Logs (checked via the UI). There are no logs from PHP-FPM.

I hope it helps, and I hope I'm not off-topic.

javerous on 28 Dec 2019

👍1

@javerous Thanks for the debugging. nginx servers are very often with bad configurations, and in your case, I notice that you have no $_SERVER['PATH_INFO'].

Have you tried the API test on https://your-server.example.net/freshrss/p/api/ ?
Have you checked the nginx example on https://freshrss.github.io/FreshRSS/en/admins/10_ServerConfig.html and in particular the fastcgi_split_path_info line?

More debugging welcome.

If you can, it is better to avoid exposing your /freshrss/p/ in your URL, and instead use a sub-domain such as https://freshrss.example.net/ or a folder alias such as https://example.net/freshrss/ (instead of ~~https://example.net/freshrss/p/~~ ). Check https://freshrss.github.io/FreshRSS/en/admins/10_ServerConfig.html and https://freshrss.github.io/FreshRSS/en/admins/03_Installation.html

I personally recommend Apache and/or our official Docker images, potentially with a dedicated reverse proxy such as Traefik (see the examples in our Docker section).

Alkarex on 28 Dec 2019

👍1

@Alkarex Thanks for the answer.

Indeed, I don't have this PATH_INFO variable. I will take a look on why PHP don't set it… For now, everything else is hosted by nginx on my server, so I will try to stay on it and make it work.

About 1. — I wasn't aware of this summary page. It shows

FreshRSS API endpoints

Google Reader compatible API

Your API address:
https://<redacted-hostname>/freshrss/p/api/greader.php
Google Reader API configuration test:
❌ FAIL: HTTP error 400 Bad Request

Fever compatible API

Your API address:
https://<redacted-hostname>/freshrss/p/api/fever.php
Fever API configuration test:
✔️ PASS

About 2. — I wasn't aware of this doc 😅. I'm going to fix my config according to the indications.

I will make changes to prevent, at least, to expose the p directory. Thanks for pointing me that…

javerous on 28 Dec 2019

Okay, so the problem was indeed fastcgi_split_path_info. Next time I will read the doc…

So I think I'm a bit off-topic there. Feel free to remove my messages for clarity purpose.

Not to be completely useless, for my "contribution" to the original problem, here is a summary of what I have tested:

Server

macOS 10.14.6 (Server)
nginx 1.17.6 with OpenSSL1.1.1d (TLS SNI support) - using TLS to serve FreshRSS
PHP-FPM 7.1.32
FreshRSS 1.15.3 (clean install, from github release) with SQLite. I'm not using a subdomain.
SQLite 3.24.0

Client

Web (via Safari 13.0.4 / macOS 10.14.6): ✅
Web (via Chrome 79.0.3945.88 / macOS 10.14.6): ✅
Reeder 4.2.2 (macOS 10.14.6): ✅
Reeder 4.2.2 (iOS 13.3): ✅

Bye bye Feedly…

_Note 1: It means that everything work without any problem. My tests include receiving articles, marking them as read, etc.
Note 2: the iOS tests took some time, I had to fight with OpenSSL and iOS new certificate control (HT210176)…_

javerous on 29 Dec 2019

🎉1

@javerous Thanks for the feedback :-) If you happen to be able to reproduce the problem reported higher, maybe with this 10000-article bug, please let us know. I am not able to test this client on my side.

Alkarex on 29 Dec 2019

@Alkarex I will. I just need to find a feed which that much articles ^^'

javerous on 29 Dec 2019

@Alkarex So, I did some tests by forging a bunch of fake RSS file stored on my website.

I have weird behaviors, but not really what @KingKarlo is seeing.

If I try to add 4 RSS feed in Web UI with respectively 20, 5000, 6000, 6500 articles, then everything works fine (the articles appear in Web UI + Reeder) & Read-Unread works as expected.
If I try to add an RSS feed in Web UI with >= ~7000 articles then, after some time (the bigger the feed is, the more I have to wait to have my feed being added), the page https://<redacted-hostname>/freshrss/i/?c=feed&a=add just show a complete white page (the content show by web inspector is <html><head></head><body></body></html>, but I suspect my browser to build an empty HTML structure for me - I didn't check with Charles).
-> The subscription appears in Subscription management (Web UI).
-> It has an orange color, and the text This feed is empty. Please verify that it is still maintained. is shown in the settings.
-> None of the articles of this specific feed appears in Reeder or Web UI.
-> There is no errors in FreshRSS logs or php logs, but nginx print this error log when trying to add the feed (but never when trying to actualize it), which can explain the white page, I guess:

2019/12/30 01:05:36 [error] 51923#0: *2368 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /usr/local/nginx/html/freshrss/lib/SimplePie/SimplePie/Cache/File.php on line 126" while reading response header from upstream, client: <redacted-remote-ip>, server: localhost, request: "POST /freshrss/i/?c=feed&a=add HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php-fpm.sock:", host: "<redacted-hostname>"

I say ~7000 because it's not stable (sometimes it works at 6999, sometimes not, same for 7000, etc.). Which makes me think it's either a silent time-out somewhere, either a silent memory allocation error somewhere.

So, well, on my side I'm not even to the point to have distinction between read / unread or sync problem with Reeder: FreshRSS fail to load feed with >= ~7000 articles.

Note: I have the exact same result if I subscribe to those tests feed directly in Reeder (except for the white add page, of course).

[Edit-1]
I saw a bunch of similar errors in FreshRSS logs. Not sur if it's related to my tests, or something wrong with my others feeds…

SQL error listHashForFeedGuids: HY000: 1 too many SQL variables while querying feed 70

[Edit-2]
Just to give some more context, all articles have about the same size (i.e. almost the same text), and the 9000 articles feed file was ~2.4 MB (i.e. not really huge).

javerous on 30 Dec 2019

@javerous Thanks for the additional tests. Yes, so many articles at the same time when adding a single feed will indeed fail. We would need to use a streaming approach like I have done in the recent SQLite export/import. But that is a rare scenario. Feeds serve typically between 10 and 100 articles at a time. The 10k limit from above is when there are more than 10k unread articles in total in FreshRSS.

Alkarex on 30 Dec 2019

@Alkarex Yes, it was my thinking.

Anyway, I will try to enhance my test by adding articles 5000 by 5000 to reach the 10000 meet by @KingKarlo.

javerous on 30 Dec 2019

Okay, so I was able to have more than 10000 unread articles on my FreshRSS instance. It took some time (less than a minute, though), but it eventually synced without any problem in Reeder.

So: I'm not reproducing 🤷‍♂️

Screenshot 2019-12-30 at 23 06 53

javerous on 30 Dec 2019

👍1

@javerous Thanks for the tests. Interesting. Regarding the time it takes, many clients have a very inefficient synchronisation strategy (see my recommendation https://github.com/FreshRSS/FreshRSS/issues/2566#issuecomment-541317776 ) as it should take < 10s even with that many articles.

Alkarex on 31 Dec 2019

👍1

That's indeed very interesting! Still I can reproduce the problem and it seems to be at least related to 10.000 articles. Below sync is working, above not.

My best guess would be that a "synthetic" feed with 10.000 dummy articles might behave slightly different than 50 real feeds with a DB grown over 12+ months with ten thousands of articles, different read states, stared articles, etc.

So for at least @bobharris and me the behavior seems to be the same.

And we both don't see anything special in the logs even though we've enabled logging.

Since @Frenzie mentioned above the introduction of the number 10.000 in #2335, could that be related?

Would it be possible to provide a version I could test with that number removed or at least bumped to 30.000 or so?

Happy to test it if that helps!

KingKarlo on 31 Dec 2019

You can just edit that line? ;-)

Frenzie on 31 Dec 2019

I'm pretty sure I'm able to edit the line if I figure out what line :)

Can't find the right spot in the thread...and I'm still not sure how and if something related to archiving could break the sync.

KingKarlo on 31 Dec 2019

Well, it's either Ctrl+F on the linked PR or 10000 in the search bar up above. :-P

https://github.com/FreshRSS/FreshRSS/blob/90c7292326538522a5df97b3f0a847b8a28f759f/app/Models/Feed.php#L10

Frenzie on 1 Jan 2020

Looks like I looked at the wrong place.

Changed it and it didn't change anything. But as mentioned before I would've been surprised if an archiving thing broke the sync. But it was worth a try.

I didn't go through all the release notes and changes but were there any significant changes in the greader API between 1.14.x and 1.15.x? I'm still wondering why it works with the former but not the later.

KingKarlo on 1 Jan 2020

A look at https://github.com/FreshRSS/FreshRSS/commits/90c7292326538522a5df97b3f0a847b8a28f759f/p/api/greader.php doesn't suggest anything to me.

The aforementioned #2335 still looks like possibly the most significant all round behavioral change to me. That may not be too hard to quickly test as a hypothesis by checking out d7f888392678d711478ed64f4a52f43021d143af to see what happens.

Basically if nothing jumps out I guess you'd have to bisect it manually by checking out the commits in an orderly fashion.
https://github.com/FreshRSS/FreshRSS/compare/1.14.3...1.15.0

I.e., start around halfway, depending on behavior go halfway up or down, form new hypotheses or just keep going until you find the guilty commit (or at least a significantly reduced range of commits).

I just don't see how the number 10000 can be significant. :-/

Frenzie on 1 Jan 2020

Going through all the commits starting at 1.14.3 to figure out when it breaks and dive in from there sounds like the safest bet. I'm just not sure if I have a DB backup in the 1.14.3 state, I have to check that (but I'm afraid I have none :( )

I'm still tempted to suspect that it could be Reeder related since they introduced their "native" FreshRSS support around exactly at that time. But in the end from the look and fell of the implementation it does just look like an additional menu entry for FreshRSS with an autocomplete of the correct path. It works exactly the same like their previous and still exiting greader API option.

If I think about it...I guess the one thing we haven't tested so far is the following setup:

FreshRSS 1.14.x
Reeder (current version), i.e. with the "native" FreshRSS support
More than 10.000 unread articles (the demo instance had less)

If that is working it would point more into the direction of something FreshRSS related.
If that isn't working it would point more into the direction of something Reeder related.

Kind of unfortunate the Silvio from Reeder never responded to get a better understanding if something in Reeder has changed under the hood.

Maybe Reeder integration via greader API is now broken forever, but I would be really interested in what caused it...

KingKarlo on 1 Jan 2020

👍1

https://github.com/FreshRSS/FreshRSS/pull/2942

Alkarex on 2 May 2020

Seems to work again, at least with FreshRSS /master branch https://github.com/FreshRSS/FreshRSS/issues/2960
https://github.com/FreshRSS/FreshRSS/issues/2956

Alkarex on 9 May 2020

Freshrss: Reeder not working with FreshRSS greader API anymore

Most helpful comment

All 53 comments

Related issues