Twint: Twitter Legacy Theme Deprecation - May impact Twint functionality from 01/06/2020

[chriscore]

Description of Issue

Twitter legacy UI deprecation - DigitalInformationWorld
As described in the article above, Twitter will be deprecating the legacy theme from 1st June 2020. It seems as though this may impact at least some functionality in Twint.

Potential affected areas seem to be restricted to code calling get.py, specifically the RandomUserAgent method:

• if self.config.TwitterSearch (which is the only case that sets the wa arg here to true), where the user agent is specifically set to Mozilla/5.0 (Windows NT 6.4; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2225.0 Safari/537.36"
• else, where the user agent is set using FakeUserAgent library, which may contain user agents which do no longer work after the deprecation
• if the above if/else fails, the user agent is set to one of a random list of hardcoded user agents, which may contain values that are affected by the deprecation.

I will continue looking into this, but raising now for visibility.

[pielco11]

604

[pielco11]

Currently down, still possible to run searches on mobile version

https://mobile.twitter.com/search?q=from%3Anoneprivacy&s=typd&x=0&y=0

[llunn]

I believe that this is working not because of the mobile url, but instead because the search endpoint returns a json payload, which is handled differently than most of the other types of data collected (e.g. user profile, follower/following list, profile based tweet collection).

[aksub99]

I suddenly started getting the error:

CRITICAL:root:twint.get:User:'NoneType' object is not subscriptable

when I try using twint.run.Lookup(c).
Installing with pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint doesn't solve the problem.

[spookyvert]

@aksub99 I get the same error on my machine. It working when testing offline doesn't when I deploy to Heroku