Twint: Timeout problem after API fix on Debian

Created on 12 Oct 2020  路  37Comments  路  Source: twintproject/twint

Issue Template

Please use this template!

Initial Check

If the issue is a request please specify that it is a request in the title (Example: [REQUEST] more features). If this is a question regarding 'twint' please specify that it's a question in the title (Example: [QUESTION] What is x?). Please only submit issues related to 'twint'. Thanks.

Make sure you've checked the following:

  • [] Python version is 3.6; => IS version 3.7
  • [] Updated Twint with pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint; => IS updated
  • [] I have searched the issues and there are no duplicates of this issue/question/request. => NO clear duplicates

Command Ran

For example twint -u realDonaldTrump but any command actually

Description of Issue

There is some kind of connection timeout (see errors below). I have this special errors since the Twitter API fix on twint side. Before it could connect but had the API errorCRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
Now it cannot connect at all.
On the Debian system I tested it with the twint version which you get if using pip3 install twint (apparently the version without the API fix) because the old error appears.
If installing latest version from github via pip3 install . -r requirements.txt there are the timeouts.

Interestingly when testing from a differnt system Linux version 5.4.0-48-generic (buildd@lcy01-amd64-010) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #52-Ubuntu SMP Thu Sep 10 10:58:49 UTC 2020 everything works as expected with the most recent version from github.

Environment Details

Linux version 4.19.0-10-amd64 ([email protected]) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.132-1 (2020-07-24)

# twint -u realDonaldTrump
WARNING:root:Error retrieving https://twitter.com/: ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f6847f2dc88>, 'Connection to twitter.com timed out. (connect timeout=10)'))")), retrying
WARNING:root:Error retrieving https://twitter.com/: ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f6847f2dbe0>, 'Connection to twitter.com timed out. (connect timeout=10)'))")), retrying
WARNING:root:Error retrieving https://twitter.com/: ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f6847f2d978>, 'Connection to twitter.com timed out. (connect timeout=10)'))")), retrying
WARNING:root:Error retrieving https://twitter.com/: ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f6847f2d7b8>, 'Connection to twitter.com timed out. (connect timeout=10)'))")), retrying
WARNING:root:Error retrieving https://twitter.com/: ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f6847f2d438>, 'Connection to twitter.com timed out. (connect timeout=10)'))")), retrying
ERROR:root:Error retrieving https://twitter.com/: ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f6847f28e80>, 'Connection to twitter.com timed out. (connect timeout=10)'))"))
CRITICAL:root:6 requests to https://twitter.com failed, giving up.
Traceback (most recent call last):
  File "/usr/local/bin/twint", line 8, in <module>
    sys.exit(run_as_command())
  File "/usr/local/lib/python3.7/dist-packages/twint/cli.py", line 311, in run_as_command
    main()
  File "/usr/local/lib/python3.7/dist-packages/twint/cli.py", line 303, in main
    run.Search(c)
  File "/usr/local/lib/python3.7/dist-packages/twint/run.py", line 427, in Search
    run(config, callback)
  File "/usr/local/lib/python3.7/dist-packages/twint/run.py", line 319, in run
    get_event_loop().run_until_complete(Twint(config).main(callback))
  File "/usr/local/lib/python3.7/dist-packages/twint/run.py", line 35, in __init__
    self.token.refresh()
  File "/usr/local/lib/python3.7/dist-packages/twint/token.py", line 61, in refresh
    res = self._request()
  File "/usr/local/lib/python3.7/dist-packages/twint/token.py", line 57, in _request
    raise RefreshTokenException(msg)
twint.token.RefreshTokenException: 6 requests to https://twitter.com failed, giving up.
picture certrik
👍1

Most helpful comment

@himanshudabas
Another info ... I am using this behind a proxy with environmental variables ($http[s]_proxy=http://proxy.domain.tld:8080) set. I tested the proxy setup. So that is not the problem.
A connection with the requests module is working fine, if not using requests.Session() to connect. If I use requests.Session() it seems that it doesn't care about the environmental variables.

It definitely is the problem. If I alter your script to read ...

import requests
from time import time
s = time()

_timeout = 25
url = 'https://twitter.com'
_session = requests.Session()
_session.proxies = {
  "http": "http://proxy.domain.tld:8080",
  "https": "http://proxy.domain.tld:8080",
}
req = _session.prepare_request(requests.Request('GET', url))
r = _session.send(req, allow_redirects=True, timeout=_timeout)

run_time = time() - s
print(f'Status Code : {r.status_code}')
print(f'Request Took {run_time} ms')

... it works.
So either the requests module has to be changed to honor environmental variables when using Session() (which would be good I think) or a workaround could be that your API improvement reads config settings from twint regarding proxy (Proxy_host, Proxy_port, Proxy_type).

picture certrik on 13 Oct 2020
🚀1 ❤1

All 37 comments

I have same problem.

aaryanwang picture aaryanwang on 12 Oct 2020

Can you try this on Python 3.6?
Let me know if it works.

himanshudabas picture himanshudabas on 12 Oct 2020

@himanshudabas I have no easy way to test with Python 3.6 at the moment but probably will figure something out. The Ubuntu system I referenced (Linux version 5.4.0-48-generic (buildd@lcy01-amd64-010) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #52-Ubuntu SMP Thu Sep 10 10:58:49 UTC 2020) where it works as expected has Python 3.8.5, if this info helps.
The exact version of Python on the Debian system where it seems NOT to work is 3.7.3 .

certrik picture certrik on 12 Oct 2020

The Ubuntu system I referenced (Linux version 5.4.0-48-generic (buildd@lcy01-amd64-010) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #52-Ubuntu SMP Thu Sep 10 10:58:49 UTC 2020) where it works as expected has Python 3.8.5

That's weird.
For me the aiohttp package was unable to fetch from 'twitter.com' on Python 3.8+ so I had to downgrade my Python to 3.6 to make it work correctly.
Even though aiohttp was able to fetch from other _urls_ just fine, for some reason it never returned anything for 'twitter.com'.

I'll look into it.

Edit : 13-10-2020
Correction : On further investigation I found that this has nothing to do with aiohttp as I previously suspected. Because in your case your requests time out without even getting the token. And token is fetched through requests and unlike aiohttp, requests works fine with https://twitter.com.
So I suspect there can be an issue with you Internet connection or outgoing ports on your system.
check the comment below for more on this

himanshudabas picture himanshudabas on 12 Oct 2020
👍1

Update :
I tested it on Python 3.8.5, Ubuntu 20.04 and It's working, just like you said.

Will update you after testing on 3.7.

himanshudabas picture himanshudabas on 12 Oct 2020

I'm running python 3.8.6 - same issues for me as original post

Qu4kE picture Qu4kE on 12 Oct 2020

@Qu4kE can you try to run it on Python 3.8.5 and see if that solves it?
I ran it on 3.8.5 and it works fine, so maybe something's changed in 3.8.6 which breaks the aiohttp.
Later today I'll try to replicate this on 3.8.6.

himanshudabas picture himanshudabas on 12 Oct 2020

I don't think so,I'm running python 3.8.5 ,but still have same issues.

aaryanwang picture aaryanwang on 13 Oct 2020

I have the same issue. Running with python 3.6.10 in Mac Catalina.

Gabriel0402 picture Gabriel0402 on 13 Oct 2020

@aaryanwang @Gabriel0402 @certrik can you tell me these details?

  1. Windows / Linux / Mac / WSL
  2. Python and pip version.
  3. Are you you using Python virtual environment? or conda virtual environment? or is _twint_ installed system-wide?
  4. Open your browser (If you are on a GUI system, and open https://twitter.com/explore in Incognito mode of your browser. and check if it loads anything.
  5. Now run the following script in the same python environment as you are running your _twint_ in :
import requests
from time import time
s = time()

_timeout = 25
url = 'https://twitter.com'
_session = requests.Session()
req = _session.prepare_request(requests.Request('GET', url))
r = _session.send(req, allow_redirects=True, timeout=_timeout)

run_time = time() - s
print(f'Status Code : {r.status_code}')
print(f'Request Took {run_time} ms')

Note : make a check_requests.py and paste the above code into it. save it. run it in the same python environment.
_Paste your output in this thread_

himanshudabas picture himanshudabas on 13 Oct 2020
  1. Linux version 4.19.0-10-amd64 ([email protected]) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.132-1 (2020-07-24)
  2. Python 3.7.3, pip 20.2.3
  3. system-wide installation
  4. no gui system
    5. 5. 
# python3 check_requests.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.7/dist-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/usr/local/lib/python3.7/dist-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.7/dist-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.7/dist-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.7/dist-packages/urllib3/connection.py", line 309, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.7/dist-packages/urllib3/connection.py", line 167, in _new_conn
    % (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x7f2c8297dda0>, 'Connection to twitter.com timed out. (connect timeout=25)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.7/dist-packages/urllib3/connectionpool.py", line 727, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/dist-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f2c8297dda0>, 'Connection to twitter.com timed out. (connect timeout=25)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "check_requests.py", line 9, in <module>
    r = _session.send(req, allow_redirects=True, timeout=_timeout)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 504, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f2c8297dda0>, 'Connection to twitter.com timed out. (connect timeout=25)'))
certrik picture certrik on 13 Oct 2020
  1. Linux version 5.4.0-48-generic (buildd@lcy01-amd64-010) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #52-Ubuntu SMP Thu Sep 10 10:58:49 UTC 2020
  2. Python 3.8.5, pip 20.0.2
  3. system-wide installation
  4. no gui system
    5.
# python3 check_requests.py
Status Code : 200
Request Took 0.3047313690185547 ms
certrik picture certrik on 13 Oct 2020

1.windows
2.python3.8.5,pip 20.1.1
3.system-wide installation
4.no gui system and browser loads the website correctly
5.
Traceback (most recent call last):
File "D:\anconda\lib\site-packagesurllib3\connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "D:\anconda\lib\site-packagesurllib3\connectionpool.py", line 976, in _validate_conn
conn.connect()
File "D:\anconda\lib\site-packagesurllib3\connection.py", line 361, in connect
self.sock = ssl_wrap_socket(
File "D:\anconda\lib\site-packagesurllib3\util\ssl_.py", line 377, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "D:\anconda\lib\ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "D:\anconda\lib\ssl.py", line 1040, in _create
self.do_handshake()
File "D:\anconda\lib\ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
socket.timeout: _ssl.c:1091: The handshake operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\anconda\lib\site-packagesrequests\adapters.py", line 439, in send
resp = conn.urlopen(
File "D:\anconda\lib\site-packagesurllib3\connectionpool.py", line 724, in urlopen
retries = retries.increment(
File "D:\anconda\lib\site-packagesurllib3\utilretry.py", line 403, in increment
raise six.reraise(type(error), error, _stacktrace)
File "D:\anconda\lib\site-packagesurllib3\packages\six.py", line 735, in reraise
raise value
File "D:\anconda\lib\site-packagesurllib3\connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "D:\anconda\lib\site-packagesurllib3\connectionpool.py", line 384, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
File "D:\anconda\lib\site-packagesurllib3\connectionpool.py", line 335, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='twitter.com', port=443): Read timed out. (read timeout=25)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:/Users/Ryan/Desktop/twint-master1/twint/1.py", line 9, in
r = _session.send(req, allow_redirects=True, timeout=_timeout)
File "D:\anconda\lib\site-packagesrequests\sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "D:\anconda\lib\site-packagesrequests\adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='twitter.com', port=443): Read timed out. (read timeout=25)

aaryanwang picture aaryanwang on 13 Oct 2020

@himanshudabas
Another info ... I am using this behind a proxy with environmental variables ($http[s]_proxy=http://proxy.domain.tld:8080) set. I tested the proxy setup. So that is not the problem.
A connection with the requests module is working fine, if not using requests.Session() to connect. If I use requests.Session() it seems that it doesn't care about the environmental variables.

certrik picture certrik on 13 Oct 2020
❤2

1.Mac Catalina
2.python3.6.10 pip 20.2.1
3.using conda virtual environment
4.the browser loads the website correctly.
5.still not working, here is the log:

Traceback (most recent call last):
File "/anaconda2/envs/py3/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/anaconda2/envs/py3/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/anaconda2/envs/py3/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/anaconda2/envs/py3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/anaconda2/envs/py3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "/anaconda2/envs/py3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn
conn.connect()
File "/anaconda2/envs/py3/lib/python3.6/site-packages/urllib3/connection.py", line 308, in connect
conn = self._new_conn()
File "/anaconda2/envs/py3/lib/python3.6/site-packages/urllib3/connection.py", line 167, in _new_conn
% (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (, 'Connection to twitter.com timed out. (connect timeout=25)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/anaconda2/envs/py3/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/anaconda2/envs/py3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 725, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/anaconda2/envs/py3/lib/python3.6/site-packages/urllib3/util/retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(, 'Connection to twitter.com timed out. (connect timeout=25)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "test1.py", line 9, in
r = _session.send(req, allow_redirects=True, timeout=_timeout)
File "/anaconda2/envs/py3/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/anaconda2/envs/py3/lib/python3.6/site-packages/requests/adapters.py", line 504, in send
raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(, 'Connection to twitter.com timed out. (connect timeout=25)'))

Gabriel0402 picture Gabriel0402 on 13 Oct 2020

@certrik @Gabriel0402 @aaryanwang

The script you ran was to check whether you could make get request to 'https://twitter.com'. From the results which you posted, I can say that there is some issue with your firewall. or your security group (in case you are using AWS/GCP). You are not being able to make the get request, so you get the ConnectTimeoutError.
@certrik your machine with Python 3.8.5 runs the check_requests.py fine, so i'd suggest you to take a look at the firewall settings of both the machines.

Also you said that you running _twint_ behind proxy, I don't see why that would cause any problem, but try disabling the proxy temporarily and see if it fixes the issue.

Another thing is both @Gabriel0402 & @aaryanwang are running this in conda virtual environment, so that's another thing i'll have to look into. i'll try to replicate this on my conda virtual environment.

himanshudabas picture himanshudabas on 13 Oct 2020

@himanshudabas
Another info ... I am using this behind a proxy with environmental variables ($http[s]_proxy=http://proxy.domain.tld:8080) set. I tested the proxy setup. So that is not the problem.
A connection with the requests module is working fine, if not using requests.Session() to connect. If I use requests.Session() it seems that it doesn't care about the environmental variables.

It definitely is the problem. If I alter your script to read ...

import requests
from time import time
s = time()

_timeout = 25
url = 'https://twitter.com'
_session = requests.Session()
_session.proxies = {
  "http": "http://proxy.domain.tld:8080",
  "https": "http://proxy.domain.tld:8080",
}
req = _session.prepare_request(requests.Request('GET', url))
r = _session.send(req, allow_redirects=True, timeout=_timeout)

run_time = time() - s
print(f'Status Code : {r.status_code}')
print(f'Request Took {run_time} ms')

... it works.
So either the requests module has to be changed to honor environmental variables when using Session() (which would be good I think) or a workaround could be that your API improvement reads config settings from twint regarding proxy (Proxy_host, Proxy_port, Proxy_type).

certrik picture certrik on 13 Oct 2020
🚀1 ❤1

@certrik I'm already making changes to this mechanism of fetching Guest Token to fix another issue with AWS server IP addresses. So i'll try to take a look into this too.
will update you once I'm done with the fix so you can test it on your machine.

Also it's weird that session doesn't respect the _environment variables_. I thought it did. Perhaps only requests respect _environment variables_ and session doesn't. I'll have to search that again I guess.

himanshudabas picture himanshudabas on 13 Oct 2020

@certrik
I just confirmed on my machine.
session doesn't respect _environment variable_, more specifically, it doesn't respect if proxy is set in _environment variables_.
So yes, we'll have to explicitly set read the proxy from _environment variables_ and create the session using that.

But that might pose a new problem. As you might already know that _twint_ has a functionality which offers to use proxy instead of your own IP. Currently I haven't set up proxies in token.py, which means soon I'll have to. So we'll have to be careful while setting up the proxy from _environment variables_ because if someone passes the proxy parameters to _twint_ then this new implementation might expose their proxy IP to twitter (which they might not want to use for _twint_, I mean it's possible that someone uses different types of proxies for different purposes)

himanshudabas picture himanshudabas on 13 Oct 2020

@himanshudabas Thanks for the reply. I tried to turned off firewall and got the same error. Also I've tried to use Postman to make get request to twitter.com, which works. So I'm not sure whether the conda virtual environment causes the issue. I'll try to not use the virtual environment and try to see whether it works or not.

Gabriel0402 picture Gabriel0402 on 13 Oct 2020

@Gabriel0402 @aaryanwang
can you try to install from this branch on your conda environment and see if it works.

@certrik can you thy the above branch too on your system which gives you ConnectTimeoutError?

I'm working on this branch to fix some other issues. Let me know what happens.

himanshudabas picture himanshudabas on 13 Oct 2020

i have same problem :

twint.run.Search(c)
WARNING:root:Error retrieving https://twitter.com/: ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(, 'Connection to twitter.com timed out. (connect timeout=10)'))")), retrying
WARNING:root:Error retrieving https://twitter.com/: ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(, 'Connection to twitter.com timed out. (connect timeout=10)'))")), retrying
WARNING:root:Error retrieving https://twitter.com/: ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(, 'Connection to twitter.com timed out. (connect timeout=10)'))")), retrying
WARNING:root:Error retrieving https://twitter.com/: ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(, 'Connection to twitter.com timed out. (connect timeout=10)'))")), retrying
WARNING:root:Error retrieving https://twitter.com/: ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(, 'Connection to twitter.com timed out. (connect timeout=10)'))")), retrying
ERROR:root:Error retrieving https://twitter.com/: ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(, 'Connection to twitter.com timed out. (connect timeout=10)'))"))
CRITICAL:root:6 requests to https://twitter.com failed, giving up.
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.9/site-packages/twint/run.py", line 427, in Search
run(config, callback)
File "/usr/local/lib/python3.9/site-packages/twint/run.py", line 319, in run
get_event_loop().run_until_complete(Twint(config).main(callback))
File "/usr/local/lib/python3.9/site-packages/twint/run.py", line 35, in __init__
self.token.refresh()
File "/usr/local/lib/python3.9/site-packages/twint/token.py", line 61, in refresh
res = self._request()
File "/usr/local/lib/python3.9/site-packages/twint/token.py", line 57, in _request
raise RefreshTokenException(msg)
twint.token.RefreshTokenException: 6 requests to https://twitter.com failed, giving up.

topliftarm picture topliftarm on 13 Oct 2020

@topliftarm can you provide the following info

also please follow these steps after you are done with the above.

himanshudabas picture himanshudabas on 13 Oct 2020

1 mac catalina 10.15.7
2 python 3.6.12 pip 20.2.3
3 conda virtual environment
4 can load the page
5 Traceback (most recent call last):
File "/Users/lijiajin/.local/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/Users/lijiajin/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/Users/lijiajin/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/lijiajin/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/Users/lijiajin/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "/Users/lijiajin/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
conn.connect()
File "/Users/lijiajin/.local/lib/python3.6/site-packages/urllib3/connection.py", line 309, in connect
conn = self._new_conn()
File "/Users/lijiajin/.local/lib/python3.6/site-packages/urllib3/connection.py", line 167, in _new_conn
% (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (, 'Connection to twitter.com timed out. (connect timeout=25)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/lijiajin/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/Users/lijiajin/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/Users/lijiajin/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(, 'Connection to twitter.com timed out. (connect timeout=25)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/lijiajin/PycharmProjects/zhai/issue.py", line 9, in
r = _session.send(req, allow_redirects=True, timeout=_timeout)
File "/Users/lijiajin/.local/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/Users/lijiajin/.local/lib/python3.6/site-packages/requests/adapters.py", line 504, in send
raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(, 'Connection to twitter.com timed out. (connect timeout=25)'))

Simngscat picture Simngscat on 13 Oct 2020

@Simngscat
can you please follow these steps. and let me know the result?

himanshudabas picture himanshudabas on 13 Oct 2020

i tried this: pip3 install --user --upgrade git+https://github.com/himanshudabas/twint.git@twint-fixes
and it still doesn't work

Simngscat picture Simngscat on 13 Oct 2020

@himanshudabas I tried the branch and here's the log:
Traceback (most recent call last):
File "/anaconda2/envs/py3/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/anaconda2/envs/py3/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/anaconda2/envs/py3/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/anaconda2/envs/py3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/anaconda2/envs/py3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "/anaconda2/envs/py3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn
conn.connect()
File "/anaconda2/envs/py3/lib/python3.6/site-packages/urllib3/connection.py", line 308, in connect
conn = self._new_conn()
File "/anaconda2/envs/py3/lib/python3.6/site-packages/urllib3/connection.py", line 167, in _new_conn
% (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (, 'Connection to twitter.com timed out. (connect timeout=25)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/anaconda2/envs/py3/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/anaconda2/envs/py3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 725, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/anaconda2/envs/py3/lib/python3.6/site-packages/urllib3/util/retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(, 'Connection to twitter.com timed out. (connect timeout=25)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "test1.py", line 9, in
r = _session.send(req, allow_redirects=True, timeout=_timeout)
File "/anaconda2/envs/py3/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/anaconda2/envs/py3/lib/python3.6/site-packages/requests/adapters.py", line 504, in send
raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(, 'Connection to twitter.com timed out. (connect timeout=25)'))

Gabriel0402 picture Gabriel0402 on 14 Oct 2020

@himanshudabas Also when I tried twint -u realdonaldtrump here's the log:
CRITICAL:root:twint.get:User:Cannot connect to host api.twitter.com:443 ssl:True [Connect call failed ('66.220.149.18', 443)]
Traceback (most recent call last):
File "/anaconda2/envs/py3/lib/python3.6/site-packages/aiohttp/connector.py", line 936, in _wrap_create_connection
return await self._loop.create_connection(args, *kwargs) # type: ignore # noqa
File "/anaconda2/envs/py3/lib/python3.6/asyncio/base_events.py", line 798, in create_connection
raise exceptions[0]
File "/anaconda2/envs/py3/lib/python3.6/asyncio/base_events.py", line 785, in create_connection
yield from self.sock_connect(sock, address)
File "/anaconda2/envs/py3/lib/python3.6/asyncio/selector_events.py", line 439, in sock_connect
return (yield from fut)
File "/anaconda2/envs/py3/lib/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb
raise OSError(err, 'Connect call failed %s' % (address,))
TimeoutError: [Errno 60] Connect call failed ('66.220.149.18', 443)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/anaconda2/envs/py3/bin/twint", line 11, in
load_entry_point('twint==2.1.21', 'console_scripts', 'twint')()
File "/.local/lib/python3.6/site-packages/twint/cli.py", line 344, in run_as_command
main()
File "/.local/lib/python3.6/site-packages/twint/cli.py", line 335, in main
run.Search(c)
File "/.local/lib/python3.6/site-packages/twint/run.py", line 419, in Search
run(config, callback)
File "/.local/lib/python3.6/site-packages/twint/run.py", line 315, in run
get_event_loop().run_until_complete(Twint(config).main(callback))
File "/anaconda2/envs/py3/lib/python3.6/asyncio/base_events.py", line 488, in run_until_complete
return future.result()
File "/.local/lib/python3.6/site-packages/twint/run.py", line 235, in main
await task
File "/.local/lib/python3.6/site-packages/twint/run.py", line 253, in run
self.config.Guest_token, True)
File "/.local/lib/python3.6/site-packages/twint/get.py", line 219, in User
response = await Request(_url, headers=_headers)
File "/.local/lib/python3.6/site-packages/twint/get.py", line 161, in Request
return await Response(session, _url, params)
File "/.local/lib/python3.6/site-packages/twint/get.py", line 167, in Response
async with session.get(_url, ssl=True, params=params, proxy=httpproxy) as response:
File "/anaconda2/envs/py3/lib/python3.6/site-packages/aiohttp/client.py", line 1012, in __aenter__
self._resp = await self._coro
File "/anaconda2/envs/py3/lib/python3.6/site-packages/aiohttp/client.py", line 483, in _request
timeout=real_timeout
File "/anaconda2/envs/py3/lib/python3.6/site-packages/aiohttp/connector.py", line 523, in connect
proto = await self._create_connection(req, traces, timeout)
File "/anaconda2/envs/py3/lib/python3.6/site-packages/aiohttp/connector.py", line 859, in _create_connection
req, traces, timeout)
File "/anaconda2/envs/py3/lib/python3.6/site-packages/aiohttp/connector.py", line 1004, in _create_direct_connection
raise last_exc
File "/anaconda2/envs/py3/lib/python3.6/site-packages/aiohttp/connector.py", line 986, in _create_direct_connection
req=req, client_error=client_error)
File "/anaconda2/envs/py3/lib/python3.6/site-packages/aiohttp/connector.py", line 943, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host api.twitter.com:443 ssl:True [Connect call failed ('66.220.149.18', 443)]

I checked I cannot ping to 66.220.149.18. However I'm able to use api.twitter.com. So I'm wondering whether this IP address cause some issues?

Gabriel0402 picture Gabriel0402 on 14 Oct 2020

Thanks @himanshudabas,I turned off firewall and installed from this branch on my conda environment,but got the same error with @Gabriel0402

aaryanwang picture aaryanwang on 14 Oct 2020

@Gabriel0402 @aaryanwang
can you try to install from this branch on your conda environment and see if it works.

@certrik can you thy the above branch too on your system which gives you ConnectTimeoutError?

I'm working on this branch to fix some other issues. Let me know what happens.

@himanshudabas Did pip3 install --user --upgrade git+https://github.com/himanshudabas/twint.git@twint-fixes and it seems it did fix my problems. Thank you.

certrik picture certrik on 14 Oct 2020

@Gabriel0402
Hi, Thank You for providing the debug info.

This one seems new to me.

I checked I cannot ping to 66.220.149.18. However I'm able to use api.twitter.com. So I'm wondering whether this IP address cause some issues?

It also tells me that you somehow managed to get the initial Token even though previously you were unable to get past that step.

Oh, I think I just figured something out. Please run the following 2 commands and tell me the exact output you see in your terminal
_Note : these are to be run on my latest twint-fixes branch :_

  1. twint -u realDonalTrump
  2. twint -u realDonaldTrump --tor-guest

Yesterday I blocked my incoming and outgoing traffic and was getting NameResolutionError (or something similar, but it was related to name resolution) which tells me that my script could not communicate to the DNS server at all. which is expected as you might expect.
In your case though (second case), it seems you are able to communicate to your DNS server but eventually your traffic gets blocked.
Seems to me that you are behind a Administrative proxy.

_I have not been able to replicate the exact error as yours. So I can't debug this on my machine as of now._

Another thing I noticed is that people with this issue are using _anaconda_ and not _system-wide_ or _Python virtual environment_. So if you could try these following steps it might help to narrow down the issue :

  1. Try using a VPN on your system and then run the script.
  2. Try using a different Internet connection, perhaps try using your mobile internet connection as a _Hotspot_.
  3. Try to run the script on either _system-wide_ or _Python virtual env_. Although I'd suggest your to first try this on _Python virtual environment_ then _system-wide_.
  4. If you have access to a different machine other than MAC, try on that too, it'll help to rule out that this is not related to MAC only. (as a lot of people are using MAC)

Please bear with me. I know this is a lot of work, but this would help me fix this.
Thank you for helping me figure this out until now.

himanshudabas picture himanshudabas on 14 Oct 2020

@Gabriel0402 @aaryanwang
can you try to install from this branch on your conda environment and see if it works.
@certrik can you thy the above branch too on your system which gives you ConnectTimeoutError?
I'm working on this branch to fix some other issues. Let me know what happens.

@himanshudabas Did pip3 install --user --upgrade git+https://github.com/himanshudabas/twint.git@twint-fixes and it seems it did fix my problems. Thank you.

Yes, I simply read from environment variables and set the proxy server to what was in there. In your case Sole reason for the error was your proxy in environment variables. But I don't think people with similar problem above are using any proxy on their system, although I suspect there might be some Administrative proxy which is to be blamed in their case.

himanshudabas picture himanshudabas on 14 Oct 2020

@himanshudabas Thanks for your patience to debug this. Here's the log:

  1. twint -u realDonalTrump
    CRITICAL:root:twint.get:User:Cannot connect to host api.twitter.com:443 ssl:True [Connect call failed ('104.244.46.208', 443)]
    Traceback (most recent call last):
    File "/anaconda2/envs/py3/lib/python3.6/site-packages/aiohttp/connector.py", line 936, in _wrap_create_connection
    return await self._loop.create_connection(args, *kwargs) # type: ignore # noqa
    File "/anaconda2/envs/py3/lib/python3.6/asyncio/base_events.py", line 798, in create_connection
    raise exceptions[0]
    File "/anaconda2/envs/py3/lib/python3.6/asyncio/base_events.py", line 785, in create_connection
    yield from self.sock_connect(sock, address)
    File "/anaconda2/envs/py3/lib/python3.6/asyncio/selector_events.py", line 439, in sock_connect
    return (yield from fut)
    File "/anaconda2/envs/py3/lib/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb
    raise OSError(err, 'Connect call failed %s' % (address,))
    TimeoutError: [Errno 60] Connect call failed ('104.244.46.208', 443)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/anaconda2/envs/py3/bin/twint", line 11, in
load_entry_point('twint==2.1.21', 'console_scripts', 'twint')()
File "/.local/lib/python3.6/site-packages/twint/cli.py", line 344, in run_as_command
main()
File "/.local/lib/python3.6/site-packages/twint/cli.py", line 335, in main
run.Search(c)
File "/.local/lib/python3.6/site-packages/twint/run.py", line 419, in Search
run(config, callback)
File "/.local/lib/python3.6/site-packages/twint/run.py", line 315, in run
get_event_loop().run_until_complete(Twint(config).main(callback))
File "/anaconda2/envs/py3/lib/python3.6/asyncio/base_events.py", line 488, in run_until_complete
return future.result()
File "/.local/lib/python3.6/site-packages/twint/run.py", line 235, in main
await task
File "/.local/lib/python3.6/site-packages/twint/run.py", line 253, in run
self.config.Guest_token, True)
File "/.local/lib/python3.6/site-packages/twint/get.py", line 219, in User
response = await Request(_url, headers=_headers)
File "/.local/lib/python3.6/site-packages/twint/get.py", line 161, in Request
return await Response(session, _url, params)
File "/.local/lib/python3.6/site-packages/twint/get.py", line 167, in Response
async with session.get(_url, ssl=True, params=params, proxy=httpproxy) as response:
File "/anaconda2/envs/py3/lib/python3.6/site-packages/aiohttp/client.py", line 1012, in __aenter__
self._resp = await self._coro
File "/anaconda2/envs/py3/lib/python3.6/site-packages/aiohttp/client.py", line 483, in _request
timeout=real_timeout
File "/anaconda2/envs/py3/lib/python3.6/site-packages/aiohttp/connector.py", line 523, in connect
proto = await self._create_connection(req, traces, timeout)
File "/anaconda2/envs/py3/lib/python3.6/site-packages/aiohttp/connector.py", line 859, in _create_connection
req, traces, timeout)
File "/anaconda2/envs/py3/lib/python3.6/site-packages/aiohttp/connector.py", line 1004, in _create_direct_connection
raise last_exc
File "/anaconda2/envs/py3/lib/python3.6/site-packages/aiohttp/connector.py", line 986, in _create_direct_connection
req=req, client_error=client_error)
File "/anaconda2/envs/py3/lib/python3.6/site-packages/aiohttp/connector.py", line 943, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host api.twitter.com:443 ssl:True [Connect call failed ('104.244.46.208', 443)]

  1. twint -u realDonaldTrump --tor-guest
    [TOR SESSION] Creating new TOR Session. Please give it a couple of seconds...
    ERROR:torpy.cell_socket:timed out
    ERROR:root:[ignored]
    Traceback (most recent call last):
    File "/.local/lib/python3.6/site-packages/torpy/cell_socket.py", line 63, in connect
    self._socket.connect((self._router.ip, self._router.or_port))
    File "/anaconda2/envs/py3/lib/python3.6/ssl.py", line 1109, in connect
    self._real_connect(addr, False)
    File "/anaconda2/envs/py3/lib/python3.6/ssl.py", line 1096, in _real_connect
    socket.connect(self, addr)
    socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/.local/lib/python3.6/site-packages/torpy/utils.py", line 78, in newfn
return func(args, *kwargs)
File "/.local/lib/python3.6/site-packages/torpy/consesus.py", line 163, in renew
raw_string = self._authorities.download_consensus(prev_hash)
File "/.local/lib/python3.6/site-packages/torpy/consesus.py", line 105, in download_consensus
with TorGuard(authority) as guard:
File "/.local/lib/python3.6/site-packages/torpy/guard.py", line 65, in __init__
self.__tor_socket.connect()
File "/.local/lib/python3.6/site-packages/torpy/cell_socket.py", line 69, in connect
raise TorSocketConnectError(e)
torpy.cell_socket.TorSocketConnectError: timed out
WARNING:torpy.utils:Retry with another authority...
ERROR:torpy.cell_socket:timed out
ERROR:root:[ignored]
Traceback (most recent call last):
File "/.local/lib/python3.6/site-packages/torpy/cell_socket.py", line 63, in connect
self._socket.connect((self._router.ip, self._router.or_port))
File "/anaconda2/envs/py3/lib/python3.6/ssl.py", line 1109, in connect
self._real_connect(addr, False)
File "/anaconda2/envs/py3/lib/python3.6/ssl.py", line 1096, in _real_connect
socket.connect(self, addr)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/.local/lib/python3.6/site-packages/torpy/utils.py", line 78, in newfn
return func(args, *kwargs)
File "/.local/lib/python3.6/site-packages/torpy/consesus.py", line 163, in renew
raw_string = self._authorities.download_consensus(prev_hash)
File "/.local/lib/python3.6/site-packages/torpy/consesus.py", line 105, in download_consensus
with TorGuard(authority) as guard:
File "/.local/lib/python3.6/site-packages/torpy/guard.py", line 65, in __init__
self.__tor_socket.connect()
File "/.local/lib/python3.6/site-packages/torpy/cell_socket.py", line 69, in connect
raise TorSocketConnectError(e)
torpy.cell_socket.TorSocketConnectError: timed out
WARNING:torpy.utils:Retry with another authority...
ERROR:torpy.cell_socket:timed out
ERROR:root:[ignored]
Traceback (most recent call last):
File "/.local/lib/python3.6/site-packages/torpy/cell_socket.py", line 63, in connect
self._socket.connect((self._router.ip, self._router.or_port))
File "/anaconda2/envs/py3/lib/python3.6/ssl.py", line 1109, in connect
self._real_connect(addr, False)
File "/anaconda2/envs/py3/lib/python3.6/ssl.py", line 1096, in _real_connect
socket.connect(self, addr)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/.local/lib/python3.6/site-packages/torpy/utils.py", line 78, in newfn
return func(args, *kwargs)
File "/.local/lib/python3.6/site-packages/torpy/consesus.py", line 163, in renew
raw_string = self._authorities.download_consensus(prev_hash)
File "/.local/lib/python3.6/site-packages/torpy/consesus.py", line 105, in download_consensus
with TorGuard(authority) as guard:
File "/.local/lib/python3.6/site-packages/torpy/guard.py", line 65, in __init__
self.__tor_socket.connect()
File "/.local/lib/python3.6/site-packages/torpy/cell_socket.py", line 69, in connect
raise TorSocketConnectError(e)
torpy.cell_socket.TorSocketConnectError: timed out
WARNING:torpy.utils:Retry with another authority...
Traceback (most recent call last):
File "/.local/lib/python3.6/site-packages/torpy/cell_socket.py", line 63, in connect
self._socket.connect((self._router.ip, self._router.or_port))
File "/anaconda2/envs/py3/lib/python3.6/ssl.py", line 1109, in connect
self._real_connect(addr, False)
File "/anaconda2/envs/py3/lib/python3.6/ssl.py", line 1096, in _real_connect
socket.connect(self, addr)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/anaconda2/envs/py3/bin/twint", line 11, in
load_entry_point('twint==2.1.21', 'console_scripts', 'twint')()
File "/.local/lib/python3.6/site-packages/twint/cli.py", line 344, in run_as_command
main()
File "/.local/lib/python3.6/site-packages/twint/cli.py", line 335, in main
run.Search(c)
File "/.local/lib/python3.6/site-packages/twint/run.py", line 419, in Search
run(config, callback)
File "/.local/lib/python3.6/site-packages/twint/run.py", line 315, in run
get_event_loop().run_until_complete(Twint(config).main(callback))
File "/.local/lib/python3.6/site-packages/twint/run.py", line 36, in __init__
self.token.refresh()
File "/.local/lib/python3.6/site-packages/twint/token.py", line 92, in refresh
if not self._request():
File "/.local/lib/python3.6/site-packages/twint/token.py", line 55, in _request
with self._session as f:
File "/anaconda2/envs/py3/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/.local/lib/python3.6/site-packages/torpy/http/requests.py", line 60, in tor_requests_session
with TorRequests(hops_count, headers, auth_data) as tr:
File "/.local/lib/python3.6/site-packages/torpy/http/requests.py", line 35, in __enter__
self._tor = TorClient(auth_data=self._auth_data)
File "/.local/lib/python3.6/site-packages/torpy/client.py", line 37, in __init__
self._consensus = consensus or TorConsensus()
File "/.local/lib/python3.6/site-packages/torpy/consesus.py", line 146, in __init__
self.renew()
File "/.local/lib/python3.6/site-packages/torpy/utils.py", line 78, in newfn
return func(args, *kwargs)
File "/.local/lib/python3.6/site-packages/torpy/consesus.py", line 163, in renew
raw_string = self._authorities.download_consensus(prev_hash)
File "/.local/lib/python3.6/site-packages/torpy/consesus.py", line 105, in download_consensus
with TorGuard(authority) as guard:
File "/.local/lib/python3.6/site-packages/torpy/guard.py", line 65, in __init__
self.__tor_socket.connect()
File "/.local/lib/python3.6/site-packages/torpy/cell_socket.py", line 69, in connect
raise TorSocketConnectError(e)
torpy.cell_socket.TorSocketConnectError: timed out

I have a clean Ubuntu machine and I'll try to install twint system wide to help to narrow down the issue.

Gabriel0402 picture Gabriel0402 on 14 Oct 2020

Thanks for the debug info.
It seems even Tor session is not being created.
It would have been much easier to pin down the problem if debug logging was working for _twint_. Right now it doesn't seem to work.

Let me know what happens when you try to follow these steps.
_Also these steps are not co-related, these are independent of each other. So it's not required for you to follow them in this sequence._

  1. Try using a VPN on your system and then run the script.
  2. Try using a different Internet connection, perhaps try using your mobile internet connection as a Hotspot.
  3. Try to run the script on either _system-wide_ or _Python virtual env_. Although I'd suggest your to first try this on Python virtual environment then _system-wide_.
  4. If you have access to a different machine other than _MAC_, try on that too, it'll help to rule out that this is related to MAC only. (as a lot of people experiencing this are using _MAC_)
himanshudabas picture himanshudabas on 14 Oct 2020

@himanshudabas I made some interesting experiments, and it works with the fix branch.
I installed twint system-wide on Ubuntu 18.04, with python 3.8.3. And I'm not using conda virtual environment.

  1. I use shadowsocks on the system and it works.
  2. I use a different internet connection and it works.
  3. In the same system, I use conda virtual env and it does not work.
  4. It works on Ubuntu.

I think the conda virtual env might cause the issue. Hope it helps. This is really a great project!

Gabriel0402 picture Gabriel0402 on 15 Oct 2020
👍1

@Gabriel0402
Thank you very much for providing the details. So currently we can safely say that anaconda virtal env is at fault here.
I'll try to pin down what exactly in conda is the reason for this issue.

If you can still do a couple more things, I'll be really grateful for that.
Do these on your MAC Catalina (which has conda).
Run the following commands in your _MAC terminal_

  1. Use this link to find your .condarc file and open it. See if there is something like HTTP or HTTPS in it. (to check if you have proxy setup in your conda environment) Also note that there might be multiple .condarc files depending upon how many _virtual envs_ you have on conda.
  2. printenv in your MAC terminal (to check if you have proxy setup in your system)
himanshudabas picture himanshudabas on 15 Oct 2020

@himanshudabas Hi, I checked and here's the results:

  1. there is no http or https in it.
  2. Here is the log
    TERM_SESSION_ID=w0t0p0:B38F3BBA-2C5F-408C-850B-071A07B98D7E
    SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.TBImVWouTm/Listeners
    LC_TERMINAL_VERSION=3.3.12
    COLORFGBG=15;0
    ITERM_PROFILE=Default
    XPC_FLAGS=0x0
    LANG=en_GB.UTF-8
    PWD=/Users/test
    SHELL=/bin/zsh
    TERM_PROGRAM_VERSION=3.3.12
    TERM_PROGRAM=iTerm.app
    PATH=/Users/test/anaconda2/bin:/Users/test/anaconda2/condabin:/Users/test/Documents/gutil/google-cloud-sdk/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin:/usr/local/mysql/bin:/opt/apache-maven/bin
    LC_TERMINAL=iTerm2
    COLORTERM=truecolor
    TERM=xterm-256color
    HOME=/Users/test
    TMPDIR=/var/folders/mt/hmbsd5j165jbf63cpjmjpwhh0000gn/T/
    USER=test
    XPC_SERVICE_NAME=0
    LOGNAME=test
    ITERM_SESSION_ID=w0t0p0:B38F3BBA-2C5F-408C-850B-071A07B98D7E
    __CF_USER_TEXT_ENCODING=0x0:0:0
    SHLVL=1
    OLDPWD=/Users/test
    ZSH=/Users/test/.oh-my-zsh
    PAGER=less
    LESS=-R
    LC_CTYPE=en_GB.UTF-8
    LSCOLORS=Gxfxcxdxbxegedabagacad
    PYTHONPATH=/Users/test/Documents/xgboost/python-package
    CONDA_EXE=/Users/test/anaconda2/bin/conda
    _CE_M=
    _CE_CONDA=
    CONDA_PYTHON_EXE=/Users/test/anaconda2/bin/python
    CONDA_SHLVL=1
    CONDA_PREFIX=/Users/test/anaconda2
    CONDA_DEFAULT_ENV=base
    CONDA_PROMPT_MODIFIER=(base)
    CONDA_BACKUP_HOST=x86_64-apple-darwin13.4.0
    HOST=x86_64-apple-darwin13.4.0
    AR=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-ar
    AS=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-as
    CHECKSYMS=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-checksyms
    CODESIGN_ALLOCATE=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-codesign_allocate
    INDR=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-indr
    INSTALL_NAME_TOOL=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-install_name_tool
    LIBTOOL=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-libtool
    LIPO=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-lipo
    NM=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-nm
    NMEDIT=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-nmedit
    OTOOL=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-otool
    PAGESTUFF=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-pagestuff
    RANLIB=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-ranlib
    REDO_PREBINDING=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-redo_prebinding
    SEG_ADDR_TABLE=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-seg_addr_table
    SEG_HACK=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-seg_hack
    SEGEDIT=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-segedit
    SIZE=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-size
    STRINGS=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-strings
    STRIP=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-strip
    LD=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-ld
    CLANG=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-clang
    CC=x86_64-apple-darwin13.4.0-clang
    CPPFLAGS=-D_FORTIFY_SOURCE=2 -mmacosx-version-min=10.9
    CFLAGS=-march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe
    LDFLAGS=-Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs
    LDFLAGS_LD=-pie -headerpad_max_install_names -dead_strip_dylibs
    DEBUG_CFLAGS=-march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -Og -g -Wall -Wextra
    _CONDA_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_x86_64_apple_darwin13_4_0
    CONDA_BUILD_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
    CLANGXX=/Users/test/anaconda2/bin/x86_64-apple-darwin13.4.0-clang++
    CXX=x86_64-apple-darwin13.4.0-clang++
    CXXFLAGS=-march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -std=c++14 -fmessage-length=0
    DEBUG_CXXFLAGS=-march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -std=c++14 -fmessage-length=0 -Og -g -Wall -Wextra
    _=/usr/bin/printenv
Gabriel0402 picture Gabriel0402 on 16 Oct 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

anderl80 picture anderl80  路  3Comments
pushshift picture pushshift  路  3Comments
wagerpascal picture wagerpascal  路  4Comments
PatricioAlvarado picture PatricioAlvarado  路  4Comments
sillyfatcat picture sillyfatcat  路  4Comments