Rss-bridge: [Twitter] @<twitter_user> failed with error 429

[Tintwo]

Hello folks,

Since 12h, I've got an error 429 from rss-bridge (see below : @ failed with error 429).

Error message: The requested resource cannot be found!
Please make sure your input parameters are correct!
cUrl error: (0)
PHP error: Trying to access array offset on value of type null
Query string: action=display&bridge=Twitter&context=By+username&u=<twitter_user>&format=Mrss
Version: dev.2020-02-26

I run rss-bridge from the latest tag inside a docker container (I updated it before to try again) and running < 15 twitter account with it.
Is it related to a "too many request" classic http code ? Or an ip ban or something ?

Regards.

[gurify]

Seeing the same issue on two different rss-bridge instances - one on Digital Ocean, the second one at home.
I'm using the latest Twitter bridge (with yesterday's commit)

[friedpuppet]

Looks like the Bridge takes its API key from some script in twitter itself.
Shouldn't we use individual API keys? Or have the option.

[em92]

Got same 429 http code and following error on DEBUG mode.

Notice: Undefined offset: 0 in /app/bridges/TwitterBridge.php on line 423

Notice: Trying to access array offset on value of type null in /app/bridges/TwitterBridge.php on line 423

Instance is my laptop which I rarely use and only for myself. So the problem origin are latest changes to twitter bridge.

[em92]

So the problem origin are latest changes to twitter bridge.

Or twitter itself. Anyway, bump @triatic

[wheredideveryonego]

This was working on the latest for several days, and started without any code change. Something has changed in the response being returned by Twitter.

[nottavi]

I know this will be a kind of "bad answer" but why not using nitter to retrieve the RSS feed for an user ?
https://nitter.net/[username]/rss

It seems to work on my side. Could be a good bypass solution

[triatic]

@nottavi Nitter scrapes Twitter same as RSS-Bridge. If they have fewer problems we can probably learn from their code and implement here.

[triatic]

This doesn't look like an api key or guest key issue since it persists after clearing both.

Bump @teromene who rewrote the bridge.

[triatic]

Oops, it actually is, the regex is failing to pick up the guest token as @em92 rightly pointed out.

[Felix83000]

When will the update be available? (approximately)

Regards

[wheredideveryonego]

My feed refreshed some time in the last few hours, so what ever the issue is with parsing the guest token is intermittent. Should be "fun" to track down...

[triatic]

The issue has been identified and fixed by Nitter https://github.com/zedeus/nitter/issues/200 .

Maybe it will give us some clues.

[wheredideveryonego]

@triatic the changes in nitter look like they are to ensure that the token is used from the thread which originated the request. that does translate well to what i understand about the implementation of this bridge... admittedly not much.

[zedeus]

My fix for nitter is using the old https://api.twitter.com/1.1/guest/activate.json endpoint with the newest bearer token. This uses less bandwidth and thus memory, but if you prefer fetching twitter.com to get the token, you can now find it in the headers, eg:
set-cookie: gt=1275871187218968587; Max-Age=10200; Expires=Wed, 24 Jun 2020 22:09:30 GMT; Path=/; Domain=.twitter.com; Secure

[zedeus]

Your token management is more careful than it needs to be. Here's what I know:

• For the /2/ endpoints, there's a global rate limit of 187 requests
• Every 15 minutes this gets reset
• If you use the token after exceeding the rate limit, you get an IP-based timeout
• After exactly 3 hours from the moment you got the token, it expires
• If the token is not actively used, it expires after ~1 hour
• You can fetch 500 tokens every 30 minutes, if you exceed that you must wait till the next 30 minute window

[triatic]

Super helpful @zedeus , thanks for checking in! I'm sure we can use this information. Great work on Nitter, I'm using it at the moment for RSS.

[arnd-s]

I fixed it on my side with the diff below. The regex must parse the header and not the body as mentioned above. Now i have to find out how to make a pull request ;)

--- a/bridges/TwitterBridge.php
+++ b/bridges/TwitterBridge.php
@@ -416,10 +416,10 @@ EOD;
        // Get a guest token. This is different to an API key,
        // and it seems to change more regularly than the API key.
        private function getGuestToken() {
-               $pageContent = getContents('https://twitter.com');
+               $pageContent = getContents('https://twitter.com', array(), array(), true);

                $guestTokenRegex = '/gt=([0-9]*)/m';
-               preg_match_all($guestTokenRegex, $pageContent, $guestTokenMatches, PREG_SET_ORDER, 0);
+               preg_match_all($guestTokenRegex, $pageContent['header'], $guestTokenMatches, PREG_SET_ORDER, 0);
                $guestToken = $guestTokenMatches[0][1];
                return $guestToken;
        }

[triatic]

The PR by @sarnd works for me. People who are using rss-bridge under the cli may also need #1623 .

[martywd]

The PR by @sarnd works for me. People who are using rss-bridge under the cli may also need #1623 .

Same here. I'm runing @sarnd 's PR as well now, and so far, TwitterBridge is working with no more 'error 429's'.

I'll add that I am NOT running #1623 .
.

[chankalan]

It seems to be fixed with the last commit, great thanks !

[Felix83000]

Fix ok! Thanks 👍 👍

[triatic]

@martywd the vast majority will be using curl to fetch page contents. #1623 helps those using cli with no defined certificate bundle.