twint = broker after change of policy of twitter.com

Created on 17 Dec 2020  Â·  4Comments  Â·  Source: twintproject/twint

Twitter.com has changed its policy and only supports a limited set of browsers (https://help.twitter.com/en/using-twitter/twitter-supported-browsers).

This yields the following output/error for the python Requests library when requesting the twitter Token:

b'<!DOCTYPE html>\n<html dir="ltr" lang="en">\n<head>\n <meta charset="utf-8" />\n <meta name="viewport"\n content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=0,viewport-fit=cover" />\n\n <style>\n body {\n -ms-overflow-style: scrollbar;\n overflow-y: scroll;\n overscroll-behavior-y: none;\n }\n\n .errorContainer {\n background-color: #FFF;\n color: #0F1419;\n max-width: 600px;\n margin: 0 auto;\n padding: 10%;\n font-family: Helvetica, sans-serif;\n font-size: 16px;\n }\n\n .errorButton {\n margin: 3em 0;\n }\n\n .errorButton a {\n background: #1DA1F2;\n border-radius: 2.5em;\n color: white;\n padding: 1em 2em;\n text-decoration: none;\n }\n\n .errorButton a:hover,\n .errorButton a:focus {\n background: rgb(26, 145, 218);\n }\n\n .errorFooter {\n color: #657786;\n font-size: 80%;\n line-height: 1.5;\n padding: 1em 0;\n }\n\n .errorFooter a,\n .errorFooter a:visited {\n color: #657786;\n text-decoration: none;\n padding-right: 1em;\n }\n\n .errorFooter a:hover,\n .errorFooter a:active {\n text-decoration: underline;\n }\n </style>\n</head>\n<body>\n <div class="errorContainer">\n <img width="46" height="38"\n srcset="https://abs.twimg.com/errors/logo46x38.png 1x, https://abs.twimg.com/errors/[email protected] 2x"\n src="https://abs.twimg.com/errors/logo46x38.png" alt="Twitter" />\n <h1>**This browser is no longer supported**.</h1>\n <p>\n Please switch to a supported browser to continue using twitter.com. You can see a list of supported browsers in our Help Center.\n </p>\n <p class="errorButton"><a href="https://help.twitter.com/using-twitter/twitter-supported-browsers">Help Center</a>\n </p>\n <p class="errorFooter">\n <a href="https://twitter.com/tos">Terms of Service</a>\n <a href="https://twitter.com/privacy">Privacy Policy</a>\n <a href="https://support.twitter.com/articles/20170514">Cookie Policy</a>\n <a href="https://legal.twitter.com/imprint">Imprint</a>\n <a href="https://business.twitter.com/en/help/troubleshooting/how-twitter-ads-work.html">Ads info</a>\n \xc2\xa9 2020 Twitter, Inc.\n </p>\n </div>\n</body>\n</html>'

Most helpful comment

I am also finding this.
If I do a simple lookup(user) in twint, it produces a NoneType error, and checking where this arises leads me to module user.py at function User, where it attempts u.id = inf(ur, "id"). In other words it seems to me as borne out above that Twitter is not accepting access using any of the User_agents and hence is asserting the browser is not supported, hence no data returned, hence no id field (or any other fields) scraped, and thus a null return.

So at present it looks like the user-agent list is not enabling access. This has been ongoing since yesterday 16 December. Prior to that no issue. Sounds like Twitter have changed something again.

I have tried a simplistic solution which is to use user agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 which seems to be an up to date user agent for Chrome v 87 respected by Twitter but no joy. This is probably at the limit of my technical ability in terms of work arounds. I think what is needed is something better than a simple user-agent string approach, and instead implement a 'zombie' version of a recognised browser and channel requests via that.

All 4 comments

I am also finding this.
If I do a simple lookup(user) in twint, it produces a NoneType error, and checking where this arises leads me to module user.py at function User, where it attempts u.id = inf(ur, "id"). In other words it seems to me as borne out above that Twitter is not accepting access using any of the User_agents and hence is asserting the browser is not supported, hence no data returned, hence no id field (or any other fields) scraped, and thus a null return.

So at present it looks like the user-agent list is not enabling access. This has been ongoing since yesterday 16 December. Prior to that no issue. Sounds like Twitter have changed something again.

I have tried a simplistic solution which is to use user agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 which seems to be an up to date user agent for Chrome v 87 respected by Twitter but no joy. This is probably at the limit of my technical ability in terms of work arounds. I think what is needed is something better than a simple user-agent string approach, and instead implement a 'zombie' version of a recognised browser and channel requests via that.

Can confirm, been dead since yesterday..

Mine works fine after the last patch on token.py yesterday? See Issue #1061 and PR #1075

Ah... I missed that one because there is so little activity on this repo in general. Indeed, such a patch should fix it

Sent from my iPhone

On 18 Dec 2020, at 03:11, Innocentius notifications@github.com wrote:


Mine works fine after the last patch on token.py yesterday?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

edsu picture edsu  Â·  3Comments

dmuth picture dmuth  Â·  4Comments

adnanmig picture adnanmig  Â·  3Comments

Nestor75 picture Nestor75  Â·  3Comments

Nestor75 picture Nestor75  Â·  4Comments