Twint: trying to use python threads to get tweets from different languages

[Nestor75]

Issue Template

Please use this template!

Initial Check

If the issue is a request please specify that it is a request in the title (Example: [REQUEST] more features). If this is a question regarding 'twint' please specify that it's a question in the title (Example: [QUESTION] What is x?). Please only submit issues related to 'twint'. Thanks.

Make sure you've checked the following:

• [x] Python version is 3.5 or higher.
• [] Using the latest version of Twint.
• [x] I have searched the issues and there are no duplicates of this issue/question/request.

Command Ran

!/usr/bin/python

import threading
import twint
import threading

def launch_query(c):
twint.run.Search(c)

max_queries=3 #numero de consultas simultaneas
twint_array=[]
for i in range(max_queries):
twint_array.append(twint.Config())
twint_array[i].Search = "#tesla"
twint_array[i].Timedelta =2
twint_array[i].Count = True
twint_array[i].Limit = "20"

if  (i==0):
    twint_array[i].search_name ="test_ca"
    twint_array[i].Lang = "ca"
elif i==1:
    twint_array[i].search_name ="test_es"
    twint_array[i].Lang = "es"
elif i==2:
    twint_array[i].search_name ="test_en"
    twint_array[i].Lang = "en"
print(i)

print("$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$")
hilo1 = threading.Thread(target=launch_query, args=(twint_array[0],))
print("$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$")
hilo2 = threading.Thread(target=launch_query, args=(twint_array[1],))
print("$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$")
hilo3 = threading.Thread(target=launch_query, args=(twint_array[2],))
hilo1.start()
hilo2.start()

Description of Issue

I would like to use threads to run several twint instances to get tweets form the same topic but in different languages. Up to know I am doing it launching several twints instances in different terminals.

I have read a closed error about threads but as per the information , you don't say it's impossible.

I ran the code above and the outcome is the following:

0
1
2
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.5/threading.py", line 862, in run
self._target(self._args, *self._kwargs)
File "00-campanya-threading.py", line 19, in launch_query
twint.run.Search(c)
File "/media/sf_virtualboxshare/Code_repository/TwitterProject.v7/twint/run.py", line 29, in Search
run(search.Search(config).main())
File "/media/sf_virtualboxshare/Code_repository/TwitterProject.v7/twint/run.py", line 5, in run
get_event_loop().run_until_complete(x)
File "/usr/local/lib/python3.5/asyncio/events.py", line 678, in get_event_loop
return get_event_loop_policy().get_event_loop()
File "/usr/local/lib/python3.5/asyncio/events.py", line 584, in get_event_loop
% threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'Thread-1'.

/usr/local/lib/python3.5/threading.py:947: RuntimeWarning: coroutine 'Search.main' was never awaited
del exc_type, exc_value, exc_tb
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.5/threading.py", line 862, in run
self._target(self._args, *self._kwargs)
File "00-campanya-threading.py", line 19, in launch_query
twint.run.Search(c)
File "/media/sf_virtualboxshare/Code_repository/TwitterProject.v7/twint/run.py", line 29, in Search
run(search.Search(config).main())
File "/media/sf_virtualboxshare/Code_repository/TwitterProject.v7/twint/run.py", line 5, in run
get_event_loop().run_until_complete(x)
File "/usr/local/lib/python3.5/asyncio/events.py", line 678, in get_event_loop
return get_event_loop_policy().get_event_loop()
File "/usr/local/lib/python3.5/asyncio/events.py", line 584, in get_event_loop
% threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'Thread-2'.

Do you have any example of how to use threads?
I know is not the last twint code but it works great for me and I want to try threads before moving to the latest twint code

Environment Details

Using Windows, Linux? linux What OS version? ubuntu Running this in Anaconda? no Jupyter Notebook? no Terminal? yes

[Nestor75]

I have found the solution here https://github.com/tornadoweb/tornado/issues/2308

you only have to import asyncio and add this line to the function
asyncio.set_event_loop(asyncio.new_event_loop())
and it works :)

the updated code is:

!/usr/bin/python

import threading
import twint
import threading
import asyncio

def launch_query(c):
asyncio.set_event_loop(asyncio.new_event_loop())
twint.run.Search(c)

max_queries=3 #numero de consultas simultaneas
twint_array=[]
for i in range(max_queries):
twint_array.append(twint.Config())
twint_array[i].Search = "#tesla"
twint_array[i].Timedelta =2
twint_array[i].Count = True
twint_array[i].Limit = "20"

if (i==0):
twint_array[i].search_name ="test_ca"
twint_array[i].Lang = "ca"
elif i==1:
twint_array[i].search_name ="test_es"
twint_array[i].Lang = "es"
elif i==2:
twint_array[i].search_name ="test_en"
twint_array[i].Lang = "en"
print(i)
print("$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$")
hilo1 = threading.Thread(target=launch_query, args=(twint_array[0],))
print("$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$")
hilo2 = threading.Thread(target=launch_query, args=(twint_array[1],))
print("$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$")
hilo3 = threading.Thread(target=launch_query, args=(twint_array[2],))
hilo1.start()
hilo2.start()

[pielco11]

Hmm interesting, would you like to create a pull request? I'm quite busy right now, in case choose the dev branch and not master

[Nestor75]

I think it doesn't make sense in creating a pull request. it's not a change in your code but a way or example of how to use twint with threading.

What I would do is including an example of how to do it in the wiki or in the readme.

I will post here an example but a clean one, more easily to understand, without debug prints and some comments and it's up to you to include it wherever you consider