Twint: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[jeekajoo]

Command Ran

twint -u jeekajoo

Description of Issue

Traceback (most recent call last):
  File "/mnt/data/twint/twint-venv/bin/twint", line 11, in <module>
    load_entry_point('twint', 'console_scripts', 'twint')()
  File "/mnt/data/twint/twint-venv/src/twint/twint/cli.py", line 269, in run_as_command
    main()
  File "/mnt/data/twint/twint-venv/src/twint/twint/cli.py", line 261, in main
    run.Search(c)
  File "/mnt/data/twint/twint-venv/src/twint/twint/run.py", line 271, in Search
    run(config, callback)
  File "/mnt/data/twint/twint-venv/src/twint/twint/run.py", line 200, in run
    get_event_loop().run_until_complete(Twint(config).main(callback))
  File "/root/bin/python3.6/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "/mnt/data/twint/twint-venv/src/twint/twint/run.py", line 141, in main
    await task
  File "/mnt/data/twint/twint-venv/src/twint/twint/run.py", line 144, in run
    self.user_agent = await get.RandomUserAgent()
  File "/mnt/data/twint/twint-venv/src/twint/twint/get.py", line 142, in RandomUserAgent
    browsers = loads(r)['browsers']
  File "/root/bin/python3.6/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/root/bin/python3.6/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/root/bin/python3.6/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Environment Details

Debian 9 / Python 3.6.8 / Twint master

[gfelitti]

I'm having the same issue on a Mac OS 10.14.5 // Python 3.6 and Ubuntu 18.10 // Python 3.7.2

[x00x0x00x]

Yeah, same thing
Hope this isn't related to my cashtag pull request
Things were operating fine upon testing

Update: not related, older version also acting up
What library is messing about?

[dwu12]

Same on Mac OS 10.14.4 , Python 3.7.2!

[12mta]

Same issue on arch linux.
python: 3.7.3, twint master

[x00x0x00x]

This is the issue:
https://fake-useragent.herokuapp.com/browsers/0.1.8

[12mta]

So im new here. we pull user agents from a website? can we use a list?

[x00x0x00x]

https://github.com/twintproject/twint/pull/427

[jformaldehydem]

Same issue, I was in the middle of a process as well.

My guess would be that Twitter has changed something. I'm new to programming so perhaps someone more talented than myself can make suggested changes?

[x00x0x00x]

There's a working fix located here: #427
Otherwise wait for the pull request to be completed and update afterwards

[jformaldehydem]

Yes, thank you! I manually made the changes and it works just beautifully!

Next I need to work out how to use diff files to streamline the process :-P

[yuuiki]

Hi, I change the file as #427 describe, but it still have the same error:
"json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)"

on Windows 7 python: 3.7

could you give me some suggestion ? thanks a a lot

[tomming]

@yuuiki What changes have you made? Did you update get.py according to #427?

Don't forget to pip3 install fake-useragent before you run.

[pielco11]

The fake-useragent server is unreachable, pushing fix right now

[pielco11]

Merging #427 , better than my workaround

Sorry for troubles guys

[jeekajoo]

Thank you :cupid:

[louisgregg]

@pielco11 thanks for pushing the fix so quick. However I still had to run two commands to update twint:
pip3 install --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint
and
pip3 install fake-useragent
Perhaps that's the expected behavior and I see that fake-useragent is included in the requirements.txt file so it's probably not a big issue.

[pielco11]

@louisgregg because a new requirement has been added, fake-useragent is already in requirements.txt

Hope that when people will come here will read the comments

I also pinned this issue to give users a direct link to the solution

But yeah, quite strange that it's not automatically installed as other requirements

[pielco11]

@louisgregg fixed now, had to update setup.py as well. Now everyone will get fake-useragent installed automatically

[yuuiki]

Thank you! it works now

[badele]

HI,

I have Archlinux with virtualenv.

it seems that I always have the error, even after updating.
 of twint. It worked last week

my requirements.txt

aiodns
beautifulsoup4
cchardet
elasticsearch
pysocks
pandas>=0.23.0
aiohttp_socks
schedule
geopy
fake-useragent
unidecode

-e git+https://github.com/twintproject/twint.git@origin/master#egg=twint

My steps

rm -rf .virtualenv
virtualenv --no-site-packages -p ${PYTHON} .virtualenv
${PIP} install -r requirements.txt

Here are the error messages I have

TWINT_DEBUG=debug .virtualenv/bin/python retrieve.py
Get tweets for 39298212/jesuislibre for 2019-05-17
[+] Inserting into Database: datas/drongo.db
Traceback (most recent call last):
  File "/usr/lib64/python3.7/urllib/request.py", line 1317, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/usr/lib64/python3.7/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.7/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.7/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.7/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/usr/lib64/python3.7/http/client.py", line 956, in send
    self.connect()
  File "/usr/lib64/python3.7/http/client.py", line 928, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/usr/lib64/python3.7/socket.py", line 727, in create_connection
    raise err
  File "/usr/lib64/python3.7/socket.py", line 716, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/badele/private/projects/drongo/.virtualenv/lib/python3.7/site-packages/fake_useragent/utils.py", line 67, in get
    context=context,
  File "/usr/lib64/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib64/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/usr/lib64/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.7/urllib/request.py", line 1345, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/lib64/python3.7/urllib/request.py", line 1319, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error timed out>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/badele/private/projects/drongo/.virtualenv/lib/python3.7/site-packages/fake_useragent/utils.py", line 166, in load
    verify_ssl=verify_ssl,
  File "/home/badele/private/projects/drongo/.virtualenv/lib/python3.7/site-packages/fake_useragent/utils.py", line 122, in get_browser_versions
    verify_ssl=verify_ssl,
  File "/home/badele/private/projects/drongo/.virtualenv/lib/python3.7/site-packages/fake_useragent/utils.py", line 84, in get
    raise FakeUserAgentError('Maximum amount of retries reached')
fake_useragent.errors.FakeUserAgentError: Maximum amount of retries reached

[pielco11]

It seems that they are having some issues with the backend

I'll add a list of known user-agents

[sdtblck]

Hi,
I'm still getting the fake_useragent error even after updating, and trying all the fixes suggested here. I'm at a loss - running python 3.7.4 on a MacBook Pro 2013, macos Mojave 10.14.5
Any help would be appreciated. Here's the error message:

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/envs/bin/python /Users/sidblack/PycharmProjects/HelloWorld/hkairport.py
WARNING:fake_useragent:Error occurred during loading data. Trying to use cache server https://fake-useragent.herokuapp.com/browsers/0.1.11
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1317, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1244, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1290, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1239, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1414, in connect
server_hostname=server_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 423, in wrap_socket
session=session
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 870, in _create
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1139, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/fake_useragent/utils.py", line 67, in get
context=context,
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1360, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1319, in do_open
raise URLError(err)
urllib.error.URLError:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/fake_useragent/utils.py", line 154, in load
for item in get_browsers(verify_ssl=verify_ssl):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/fake_useragent/utils.py", line 97, in get_browsers
html = get(settings.BROWSERS_STATS_PAGE, verify_ssl=verify_ssl)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/fake_useragent/utils.py", line 84, in get
raise FakeUserAgentError('Maximum amount of retries reached')
fake_useragent.errors.FakeUserAgentError: Maximum amount of retries reached

Process finished with exit code 0

[pielco11]

@sdtblck may you update the package? The issue has been fixed

[sdtblck]

@pielco11 The package is fully updated, and I also tried manually implementing the fix above with no luck. I actually get a result from twint, which is strange, but i'm not sure if it's complete

[pielco11]

@sdtblck I guess the issue is not related to Twint itself. I don't get error messages, anyway if Twint can't fetch user-agents (Maximum amount of retries reached error message), it'll use a random one from a hardcoded list. So Twint will return tweets, one way or another