Rss-bridge: [Twitter] Legacy website design being discontinued on June 1, 2020

[VerifiedJoseph]

Twitter is discontinuing the 'legacy' website design, which currently powers this bridge, on June 1, 2020.

[em92]

Ping @pmaziere.

[triatic]

June 1 is here, about half of my twitter feeds are now returning no data.

[triatic]

All Twitter feeds showing zero results now. I guess the 50/50 results I saw earlier was due to some servers not having shut down legacy support.

[Nono-m0le]

@teromene do you know who was responsible for this bridge? If not, any idea who can propose a PR for that?

[triatic]

The maintainer is @pmaziere , but anyone is welcome to propose a PR!

[drego85]

Hi Team,
I've done several tests but currently I don't think there is an effective solution.

Let me explain...

Twitter from 1 June requires a browser with javascript to view the tweets (that is load the tweets with javascript).

If I disable JavaScript in my Firefox browser, I see the message:

We've detected that JavaScript is disabled in your browser. Would you like to proceed to legacy Twitter?

If I confirm and I'm directed to:

https://mobile.twitter.com/

Well, I thought.

We scrap the data from the mobile version, change the structure of the page (must change all the references in the code) but it works. I have a working version locally.

But, and here's the real problem, if I do a search I don't see recent results.

For example, if I search for "AndreaDraghetti" in the mobile version I see very old results. In the Desktop version of a few hours ago.

Try it too (remember to disable JavaScript in the desktop version):

https://mobile.twitter.com/search?q=AndreaDraghetti&s=typd&x=0&y=0
https://twitter.com/search?q=AndreaDraghetti&src=typed_query&f=live

I don't know how you can fix this :(

[Pofilo]

Hey @drego85, this URL seems to work with the mobile version:

https://mobile.twitter.com/search?q=AndreaDraghetti&src=typed_query&f=live
(just changed the end of the URL)

[drego85]

It absolutely works, but the results are not recent. The last result shown is from May 30th (in the desktop version it is today)..

[Pofilo]

Well, with this mobile URL (https://mobile.twitter.com/search?q=AndreaDraghetti&src=typed_query&f=live), I have the same results than on the other version:

[drego85]

You haven't disabled JavaScript :)

This is the graphical interface with JavaScript disabled:

[Pofilo]

Okay my bad ..

[drego85]

No problem, better to make multiple checks ;)

[triatic]

@drego85 We can simulate the actions of Javascript, we just have to work out what the Javascript is doing and write it in PHP.

[somini]

This is a bummer.

There's always Nitter for a quick hack, until this is solved: https://github.com/zedeus/nitter

[teromene]

I have pushed an update to TwitterBridge.
It required a major rewrite, and probably still has problems that needs to be quelled.

[drego85]

Thanks @teromene great workaround, I didn't know that "tokens" would be released to guest users.

[teromene]

Can you share your input parameters as well as eventually your log @Nono-m0le ?

[triatic]

Excellent work @teromene , I can confirm successful operation in cli mode.

[kinoushe]

Glad it's working again! Thanks @teromene

One problem is that display_url doesn't appear to be the correct usage for the img src - it results in broken images.

The media_url_https works correctly for the enclosures, and I think this should be used for $display_image also.

For the enclosures could we instead retain the 'orig' tag as before to retrieve the original sized images here?

The following works correctly for me:

$image = $media->media_url_https . '?name=orig';
$display_image = $media->media_url_https;

[doowruc]

It's not working in my heroku instance.

I did my usual process of pull from here, push to heroku, and heroku triggered a deploy.

I get an error: Bridge returned error 429! (18418)

with detail in HTML format:

Twitter @github was unable to receive or process the remote website's content!
Error message: `The requested resource cannot be found!
Please make sure your input parameters are correct!
cUrl error: (0)
PHP error: `
Query string: `action=display&bridge=Twitter&context=By+username&u=github&noimgscaling=on&format=Html`
Version: `dev.2020-02-26`

or in plaintext format:

<p class="exception-message"><strong>Twitter @github</strong> was unable to receive or process the
remote website's content!<br>
Error message: `The requested resource cannot be found!<br />
Please make sure your input parameters are correct!<br />
cUrl error:  (0)<br />
PHP error: `<br />
Query string: `action=display&bridge=Twitter&context=By+username&u=github&format=Mrss`<br />
Version: `dev.2020-02-26`</p>

All modes in all formats give the same error

[doowruc]

I just deployed to a new heroku app using the button on the repo front page and the new instance also has the same error

[teromene]

429 response is when twitter considers that you have made too many requests
I will look into it

[drego85]

@doowruc I think to avoid this limitation, you can customize the code by adding two valid API tokens generated by the Twitter development site.

[doowruc]

@drego85 add them where?

[drego85]

Line 346 and 347 and you can find more info of bearer tokens here.

However I haven't personally tried.

[Pofilo]

The last commit fixes the Bridge, thanks ! :)
But I have some remarks:

• https://github.com/RSS-Bridge/rss-bridge/commit/06891ae35f0947d8f2a8daa0094b25e2db862226#diff-f82ed36085a4ed36bb827ac4e6db7b25R234: the id of the user is now its id as a number : 54932564 against Canardpcredac -> an entry with the old version will have a different id than with this entry
• https://github.com/RSS-Bridge/rss-bridge/commit/06891ae35f0947d8f2a8daa0094b25e2db862226#diff-f82ed36085a4ed36bb827ac4e6db7b25R230: the author field is inversed: Canardpcredac (@Canard PC) against Canard PC (@Canardpcredac)

I may have the time this weekend for a MR, not sure.

[triatic]

It worked for a while and now I see failed to open stream: HTTP request failed! HTTP/1.0 403 Forbidden

How long are the authorization and x-guest-token headers cached for? Do they need to be periodically refreshed?

[danielsale]

Mine updated once, this morning. I am following lists and now get:

Bridge returned error 403! (18418)
Twitter list by danielleesale</strong> was unable to receive or process the
remote website's content!<br>
Error message: `The requested resource cannot be found!<br />
Please make sure your input parameters are correct!

[triatic]

@danielsale try deleting the TwitterBridge folder in the cache directory. Worked for me, but you'll need to keep doing it until the cache expiry time for the headers is reduced.

[markwaters]

Thank you @teromene for fixing this , I was surprised and disappointed by how much of my news comes in via Twitter these days.
Thanks @triatic for the cache clearing suggestion , I was using MemCacheD and getting the 4XX errors , switched back to the FileCache for now and seeing tweets again.

[doowruc]

ok, thats odd... 2 of my many bridge feeds in feedly just refreshed and pulled in more tweets. But if I try and add a new feed in my bridge UI, I;m still getting the 429 error

[akkiverma]

Getting error in twitter feed:

Bridge returned error 403!

Twitter search test was unable to receive or process the remote website's content!
\nError message: The requested resource cannot be found! \nPlease make sure your input parameters are correct! \ncUrl error: (0) \nPHP error:

[akkiverma]

aah I emptied the cache folder and it started working.

[Tintwo]

aah I emptied the cache folder and it started working.

Yes, it works for a time. I don't know yet what is the condition to have this problem, bug actually it cost me to flush cache files twice a day.

[Edit] : several times a day, actually :/

[akkiverma]

Yes the error is returning back again and again. It works once when I clear the cache but never updates. I have enabled debug too yet it makes the cache.

[nousernameavailableanymore]

Is there any way to automate this (clearing the cache) with the docker installation?

[triatic]

I don't know about docker, but a cron can be set to delete cache/TwitterBridge/* once an hour.

[nousernameavailableanymore]

Yes, that was my thought as well. But there is no cron in the docker container.

[akkiverma]

Why the need to create a cron to clear cache? I have enabled debug yet the cache gets formed. I think that need to be fixed.

[triatic]

It's a temporary fix until someone corrects the cache code in the bridge.

[nousernameavailableanymore]

The twitter rss-bridge suddenly started working again for me. Except for the still broken images. Is this a known issue as well?

[kinoushe]

Yes, the broken images is a known issue @nousernameavailableanymore, see https://github.com/RSS-Bridge/rss-bridge/issues/1562#issuecomment-639393175 for the fix.

[nousernameavailableanymore]

I have no clue what that means?
Assume that someone had no clue whatsoever about webdevelopment.
How would this person fix the issue?

[kinoushe]

@nousernameavailableanymore just wait for the bridge to be updated with the fix then.

[somini]

@teromene Many thanks for updating the bridge, I appreciated it very much.

I opened #1591 to include the fix @kinoushe mentioned above. Another regression is that t.co links are no longer resolved, and if they refer to another tweet, that tweet would be shown in the item contents.

[kinoushe]

Since we're now using the Twitter API it's possible to include videos:
$media->video_info->variants[0]->url

We would need to loop through the variants[] to find the highest bitrate:
https://developer.twitter.com/en/docs/tweets/data-dictionary/overview/extended-entities-object#tweet-video

And then render video HTML instead of the image HTML, with the url as src and media_url_https as the video poster.

[nousernameavailableanymore]

Will there be a new docker image with these changes?

[teromene]

I have pushed a new update.
This should fix images, profile names as well as introduce a new caching strategy for guest tokens.

[kinoushe]

@teromene are we able to keep the ?name=orig size on the enclosure images as before?

$image = $media->media_url_https . '?name=orig';

media_url_https and media_url bring in a downsized image from my testing.

[danielsale]

Getting a 403 on lists again.

[somini]

Since we're now using the Twitter API it's possible to include videos:
$media->video_info->variants[0]->url

When an advertiser chooses to limit video playback to just Twitter owned and operated platforms, the video_info object will be replaced with an additional_media_info object.

Note the video_info object might not exist.

[kinoushe]

@somini oops, I missed that disclaimer. But I think conditionally displaying most/non advertiser videos would be better than nothing at all as current.

From my limited testing the past few days video_info works well.

[somini]

@somini oops, I missed that disclaimer. But I think conditionally displaying most/non advertiser videos would be better than nothing at all as current.

From my limited testing the past few days video_info works well.

I'll include a first pass on this on #1595 . It also supports "animated GIF", exposed as videos.

Thanks for digging into the API.

[kinoushe]

Excellent, thanks @somini!

[triatic]

@teromene the caching strategy is better now but I think we need to expire tokens based on time as well as usage.

[nousernameavailableanymore]

I updated to the latest docker image yesterday. For some reason the version it still shows RSS-Bridge as version "dev.2020-02-26".
Unfortuantely I still get 403s for the most part. Sometimes it seems to work sporadically but I still got the broken images.

[teromene]

@triatic It should expire at the same time as the default cache expiration time
@nousernameavailableanymore The version thing is normal AFAIK, I have not had time to update it yet. The images and 403 should be fixed in the latest build

[triatic]

@teromene are you saying the token should last no longer than CACHE_TIMEOUT in the bridge (normally 5 minutes) when it is used under 100 times? Because I'm not seeing that behaviour here.

[ShaneZampire]

I use to have #Breaking tweets displayed on my web page till the new Twitter came out and it stopped working. I just replaced the old Twitter Bridge with the new one and now I get the image below. Do I need to replace other files or is this still being worked on? Thanks

[em92]

@ShaneZampire pull latest version of TwitterBridge. It is probably fixed already

[ShaneZampire]

I just tried the latest as of 4 days ago. I guess Twitter's new version has broke it for good for us :( I am thankful for all the hard work that went into this project. Before Twitter broke it, it worked awesome and I am super grateful for the months I did get to use it.

[em92]

@ShaneZampire, the screenshot you gave does not tell anything to me. You need to provide full text of error.
Also are you using Memcached?