Twint: Twint stopped working out of the blue

[fsiep]

Have been using python 3.6 scripts for the last 3 months, using twint. Yesterday, I received a 404 with one of the scripts and with the other I am seeing no progress, just a countdown that indicates that the execution has been stopped.

[robert-moore]

Same here. It stopped 17 Sept around 8pm EST for me.

twint.run:Twint:Feed:noDataExpecting value:

See this thread: https://github.com/twintproject/twint/issues/915

[dansar39]

Habra posibilidad de solucionarlo? Twiter cambio algo?

[dansar39]

Conocen otro parecido para reemplazarlo, para escritorio

[lukaspistelak]

same here :+1:

error search  Expecting value: line 1 column 1 (char 0) 0
Expecting value: line 1 column 1 (char 0) [x] run.Feed[!] if get this error but you know for sure that more tweets exist, please open an issue and we will investigate it!CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 1.0 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 8.0 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 27.0 secs

[osamaabdalla]

same here:

CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 1.0 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 8.0 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 27.0 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 64.0 secs

any solutions !

[dansar39]

Habra alguna solucion?

---

De: lukaspistelak [e-mail]
Enviado: domingo, 20 de septiembre de 2020 9:05
Para: twintproject/twint [e-mail]
Cc: dansar39 [e-mail]; Comment [e-mail]
Asunto: Re: [twintproject/twint] Twint stopped working out of the blue (#918)

same here 👍

error search Expecting value: line 1 column 1 (char 0) 0

Expecting value: line 1 column 1 (char 0) [x] run.Feed[!] if get this error but you know for sure that more tweets exist, please open an issue and we will investigate it!CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)

sleeping for 1.0 secs

CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)

sleeping for 8.0 secs

CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)

sleeping for 27.0 secs

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/twintproject/twint/issues/918#issuecomment-695763513, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APYWXU67CHXK3AIA5TGY6RDSGXAVJANCNFSM4RTB7HHA.

[adulau]

Twitter just killed the legacy without login and JS enabled.

[akuma527]

Twitter just killed the legacy without login and JS enabled.

Hi @adulau , can we get this working again?

[adulau]

For my understanding, the only option is to go for a complete instrumentation (Selenium) of the browser to support JS but it also means using logged accounts. It would be major rewrite of the project.

[zhouhanc]

same here!

[guidovranken]

For my understanding, the only option is to go for a complete instrumentation (Selenium) of the browser to support JS but it also means using logged accounts. It would be major rewrite of the project.

https://github.com/JustAnotherArchivist/snscrape still works just fine. It relies on HTML parsing using BeautifulSoup, but Selenium/JavaScript is not needed.

Use snscrape --jsonl twitter-user <user> to get very similar results to twint --json.

I'm not really familiar with the internals of either project, but implementing in twint whatever snscrape is doing should get us going again. @pielco11

[zhixideyu]

据我了解，唯一的选择是使用浏览器的完整工具（Selenium）以支持JS，但这也意味着使用登录帐户。这将是该项目的主要重写。

https://github.com/JustAnotherArchivist/snscrape仍然可以正常工作。它依赖于使用BeautifulSoup的HTML解析，但是不需要Selenium / JavaScript。

使用snscrape --jsonl twitter-user <user>变得非常相似的结果twint --json。

我对这两个项目的内部都不是很熟悉，但是无论snscrape正在做什么，都可以用twint来实现，这应该会让我们重新开始。@ pielco11

can get a tweet link, but how do you get the text content

[lukaspistelak]

https://github.com/mherrmann/selenium-python-helium

[akuma527]

https://github.com/mherrmann/selenium-python-helium

This becomes unresponsive when I login into twitter with my user and pass. Is this happening to anyone else?

[arghzero]

@guidovranken @zhixideyu and anyone else looking for an alternative, looks like the --jsonl flag was added to snscrape quite recently, it's present in the github source but the version on pypi doesn't have it yet.

To get tweets in json using snscrape (including their text content) do this:

pip install --upgrade git+https://github.com/JustAnotherArchivist/snscrape@master
snscrape --jsonl twitter-user <twitter-handle-without-@-sign>

You'll get back json like this:

{
  "url": "https://twitter.com/textfiles/status/1308158822922493952",
  "date": "2020-09-21T21:39:02+00:00",
  "content": "@Sierra_OffLine Just for the record, this person is now an \"Afterlife Coach\" who writes books about how your pets will enjoy the afterlife https://t.co/GK3qnk9Pf2",
  "id": 1308158822922494000,
  "username": "textfiles",
  "outlinks": [],
  "outlinksss": "",
  "tcooutlinks": [],
  "tcooutlinksss": "",
  "retweetedTweet": null
}

@adulau can whatever method snscrape uses to get tweets be ported into twint?

[zhixideyu]

snscrape --jsonl twitter-user <twitter-handle-without-@-sign>

Hello, there are still some mistakes after I install it according to what you said. I hope you can introduce it clearly

[zhixideyu]

@guidovranken @zhixideyu和其他正在寻找替代方法的人，似乎该--jsonl标志是最近才添加到snscrape的，它存在于github源码中，但是pypi上的版本还没有它。

要使用snscrape（包括其文本内容）在json中获取推文，请执行以下操作：

pip install --upgrade git+https://github.com/JustAnotherArchivist/snscrape@master
snscrape --jsonl twitter-user <twitter-handle-without-@-sign>

您将像这样返回json：

{
   “ url ”：“ https://twitter.com/textfiles/status/1308158822922493952 ”，
   “ date ”：“ 2020-09-21T21：39：02 + 00：00 ”，
   “ content ”：“ @Sierra_OffLine仅用于根据记录，此人现在是“来世教练”，他写有关您的宠物如何享受来世的书https://t.co/GK3qnk9Pf2 “，
   ” id “：1308158822922494000，
   ”用户名“：”textfiles “，
   ” outlinks “：[]，
   ” outlinksss “：” “，
   ” tcooutlinks “：[]，
   ” tcooutlinksss “：” “，
   ” retweetedTweet “：null 
}

@adulau可以将snscrape用于获取tweet的任何方法移植到twint吗？

AttributeError: module 'functools' has no attribute 'cached_property'

[arghzero]

that looks like an issue with your python install. Above works for me on python 3.8

[zhixideyu]

that looks like an issue with your python install. Above works for me on python 3.8

ok I'm going to install 3.8

[zhixideyu]

看起来像您的python安装有问题。以上对我适用于python 3.8

好的，我要安装3.8

It works, but how can you run a script instead of a command line

[fsiep]

The 404 error seems to be triggered by Twitter finger-printing the request. The URL works in various browsers. A non-Javascript version of twitter is available through https://mobile.twitter.com. The link works without login.

[rennanharo]

It seems to be a generalized problem with Twitter scrapers. The GetOldTweets3 package also stopped working after the changes Twitter made to the endpoint.

[dansar39]

python

y aqui el script

Saludos

---

De: 窒息得鱼 [e-mail]
Enviado: martes, 22 de septiembre de 2020 4:07
Para: twintproject/twint [e-mail]
Cc: dansar39 [e-mail]; Comment [e-mail]
Asunto: Re: [twintproject/twint] Twint stopped working out of the blue (#918)

看起来像您的python安装有问题。以上对我适用于python 3.8

好的，我要安装3.8

It works, but how can you run a script instead of a command line

―
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/twintproject/twint/issues/918#issuecomment-696497781, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APYWXU6CJAMJRSROG42DNTLSHAPIZANCNFSM4RTB7HHA.

[elamje]

Just to be absolutely clear - After reading the thread above, people seem to be saying everything but the project is now in the graveyard.

@adulau Is it safe to say this project is now in archive due to the change in Twitter's API?

[himanshudabas]

For my understanding, the only option is to go for a complete instrumentation (Selenium) of the browser to support JS but it also means using logged accounts. It would be major rewrite of the project.

I have made a PR #944, it uses new endpoints of twitter. There's no need to use _Selenium_ or a JS enabled browser/headless browser. A lot of stuff might still be broken though.

[irisdemented]

For my understanding, the only option is to go for a complete instrumentation (Selenium) of the browser to support JS but it also means using logged accounts. It would be major rewrite of the project.

I have made a PR #944, it uses new endpoints of twitter. There's no need to use _Selenium_ or a JS enabled browser/headless browser. A lot of stuff might still be broken though.

@himanshudabas firstly, thank you for your work on this! i started fresh and downloaded the twint with your fix added, but continue to getcan't find '__main__' module in 'twint' when I run searches.. I'm not sure where to start troubleshooting. have tried in both 3.6 and 3.6.8. Any ideas?

[himanshudabas]

@irisdemented did you install it locally using pip install -e .
I'll lay down the steps.
I recommend using virtual environment.

1. Create a virtual environment with python 3.6.x
2. Activate this virtual environment and cd to the _twint_ directory. (Which contains setup.py
3. Run pip install -e . (this will install the _twint_ locally in your virtual environment)
4. Now you can create a .py file in you system anywhere and can simply import twint (but remember, your virtual env must be active to import this _twint_ installation.)

Let me know if you get it working.
Note : there might be a missing _dataclasses_ error so simply pip install dataclasses in your virtual env to fix that.

[irisdemented]

@himanshudabas extremely happy to report this is working!

once i figured out i needed to use it as module and not at the command line, everything clicked into place.

thanks much for your work and help.

[milad14]

@himanshudabas thanks for your work
I can not retrieve tweets when I configure the user and search item concurrently for example this:

c = twint.Config()
c.Username = "realDonaldTrump"
c.Search = "great"
twint.run.Search(c)

but they work when they are apart

[himanshudabas]

@milad14 please check #973

[milad14]

@milad14 please check #973

thanks a lot . it is solved